How to Reference Azure Storage Files from Cloud Shell

Recently I used Azure Cloud Shell for the first time. This is a quick post to show how I referenced the file share in Azure Storage to communicate with Cloud Shell.

What is Cloud Shell?

Cloud Shell is a lightweight way to run scripts using either Bash or PowerShell. You can run scripts in a browser using the Azure portal or shell.azure.com, with the Azure mobile app, or using the VS Code Azure Account extension. If you have seen the "Try it now" links in Azure documentation pages, that will direct you to use Cloud Shell.

CloudShell_AzurePortal.jpg

The rest of this post focuses on using PowerShell with Cloud Shell.

Finding the File Service Info to Use with Cloud Shell

When you create a Cloud Shell account, you are prompted to also create an Azure Storage account. This gives you a mounted file share in the Azure Files service which is available for all of your Cloud Shell sessions.

You can use Get-CloudDrive to find the information related to the drive for your Cloud Shell account:

Get-CloudDrive.jpg
 

Example of Exporting a File to Azure Files Using Cloud Shell

Using the mount point information we got from Get-CloudDrive shown above, we now have what we need for how to reference the Azure file share that is associated to my Cloud Shell account. For an example, I’m going to export a PBIX file from the Power BI Service (discussed in my previous blog post).

First we need to log in to the Power BI Service. In Cloud Shell, authentication to another service like Power BI is a little different than how we see it within a local client tool - it utilizes a login page with a code provided by Cloud Shell:

Connect-PowerBIServiceAccount.jpg

Now that we are authenticated, we can execute our Export-PowerBIReport cmdlet:

Export-PowerBIReport_CloudShell.jpg

Note in the above PowerShell cmdlet, I referenced the mount point for the Azure file share associated with my Cloud Shell account as:

'/home/melissa/clouddrive/PowerBIExportFiles/ExportTest_V2workspace.pbix'
AzureFiles.jpg

Notice in the above example I used a folder called ‘PowerBIExportFiles’ — note that any folder(s) need to already exist before you can export a file to it. It won’t auto-create for you; if the folder doesn’t exist you’ll get an error that says “Could not find a part of the path.” Using a folder is optional though.

You Might Also Like…

Controlling Data Access in Azure for Administrators and Owners

Getting Started with Azure

PowerShell for Assigning and Querying Tags in Azure

Lesson Learned - Keep PowerShell Modules Consistent and Up To Date

PowerShellLogo.jpg

This is a quick post to share something that happened on a project recently. We began to experience some intermittent issues with Azure Data Factory (V1) and it was ultimately related to an out-of-date AzureRM PowerShell module. What does ADF have to do with the AzureRM PowerShell module you ask? For this project, the nightly loads are triggered by a signal file which indicates the source data is ready. Therefore, we have a PowerShell script that controls the whole end-to-end process (which overrides the ADF V1 built-in scheduling). The PowerShell script looks for the signal file and then proceeds to resume the Azure Data Factory pipelines (via the Resume-AzureRmDataFactoryPipeline cmdlet). The resume causes each pipeline to execute immediately. When all pipelines are finished, they all get suspended again (via the Suspend-AzureRmDataFactoryPipeline cmdlet) until the next execution of the data load process. This PowerShell process runs on a virtual machine.

In production recently, we started seeing cases of ADF pipelines that wouldn't resume properly which resulted in some data not getting loaded properly. It was inconsistent and intermittent. With the exact same PowerShell script, we couldn't reproduce the issue in UAT. The issues were only occurring in production, but not with precise regularity. 

My colleague on the project, Terry Crist, did some investigation and found that the AzureRM module installed on UAT was much newer than what was installed on production. Sure enough, once the AzureRM module was updated in production everything began to run reliably again.

So, this served as a good reminder to ensure that (a) UAT and production environments should be running on the same PowerShell module version, and (b) preferably the latest version should be installed when possible. In environments which don't have a full-time DBA looking after this sort of thing it's good for developers to know to watch out for these types of issues too. 

You Might Also Like...

Find Pipelines Currently Running in Azure Data Factory with PowerShell

PowerShell for Assigning and Querying Tags in Azure

Assigning Data Permissions for Azure Data Lake Store (Part 3)

Update Jan 6, 2019: The previously posted PowerShell script had some breaking changes, so both scripts below (one for groups & one for users) have been updated to work with Windows PowerShell version 5.1.

This is part 3 in a short series on Azure Data Lake permissions. 

Part 1 - Granting Permissions in Azure Data Lake
Part 2 - Assigning Resource Management Permissions for Azure Data Lake Store
Part 3 - Assigning Data Permissions for Azure Data Lake Store {you are here}

In this section, we're covering the "data permissions" for Azure Data Lake Store (ADLS). The ACL (access control list) grants permissions to to create, read, and/or modify files and folders stored in the ADLS service. Uploading and downloading data falls in this category of ACLs. If you come from the Unix or Linux world, the POSIX-style ACLs will be a familiar concept. 

There are two types of ACLs: Access ACLs and Default ACLs.

An Access ACL is the read/write/execute permissions specified for a folder or file. Every single folder or file has its security explicitly defined -- so that means the ADLS security model is not an 'inheritance' model. That is an important concept to remember.

A Default ACL is like a 'template' setting at a folder level (the concept of a default doesn't apply at the file level). Any new child item placed in that folder will automatically obtain that default security setting. The default ACLs are absolutely critical, given that data permissions aren't an inheritance model. You want to avoid a situation where a user has permission to read a folder, but is unable to see any of the files within the folder -- that situation will happen if a new file gets added to a folder which has an access ACL set at the folder level, but not a default ACL to apply to new child objects.

Tips for Assigning Data Permissions for ADLS

Organize your data lake folders and files so that folder-level security is one of the main considerations, and so that security is easier to manage.

Access to raw data is typically highly limited. This is partially due to lack of usability, and partially to ingest data as quickly as possible. Because every single file in ADLS has security properties specified, that is one of several reasons why a large number of very tiny files in ADLS is generally discouraged.

Typically in a data lake, the majority of users need only read+execute rights to consume the data. However, you may also have an area like a 'manual drop zone' or an 'analytics sandbox' where select users have write permissions to create, modify & delete folders and files. Generally speaking, write permissions in a data lake are minimal.

Be selective with granting permissions at the root level ("/"). It does minimize maintenance if you define an access ACL + default ACL at the root level, but only if you feel confident that is adequate security.

Try to use Azure Active Directory (AAD) groups whenever you can to grant access, rather than individual user accounts. This is a consistent best practice for managing security across many types of systems. This will reduce maintenance, and reduce the risk of inaccurate or out of date user permissions.

Currently the maximum # of ACLs that can be assigned to a file or folder is 32. This is another big reason to use AAD groups for managing access, rather than individual users.

Try to assign access at the folder level whenever you can. Although ADLS doesn't have a true inheritance model, you can set a 'default' entry which will handle new child items. 

Be aware that changing a default ACL on a folder doesn't automatically propagate to change the default ACL on any existing child folders. So, managing changes to existing data needs to be done carefully. Although it can seem like the default ACLs act like an inheritance model in some respects, it definitely is not.

Grant access to an AAD application (aka service principal identity or SPI) for automated operations, such as data loads. For service principals, you often can assign just the data permissions (the ACL) and not any permissions to the ADLS service (the RBAC). Check Part 4 for more discussion about service principals.

You almost always want to assign read + execute permissions together. The execute permissions allow a user to traverse the folder structure to where a file resides, which is needed in conjunction with the read (or write) permissions for the actual file.

The portal interface makes it easy to apply permissions to existing child-level folders and files. It's is a little harder to specify via scripting methods since your script will need to explicitly be set up to handle recursive operations. Therefore, try to assign relevant permissions as early as possible in your design/development/operationalization phase. 

When you set permissions on existing data, it can take a little while if you are asking it to recursively traverse the folders and files to set permissions for every object. This is another reason to try to set permissions at the AAD group level, rather than via individual users. 

The PowerShell cmdlets to manage ADLS changed in January 2018. See this post: Breaking changes to Azure Data Lake Store cmdlets

Defining ADLS Data Permissions in the Azure Portal

In my example, I want to assign read + execute permissions for the StandardizedData folder, but not for the RawData folder. In the portal, I open Data Explorer, navigate to the applicable folder which sets the "scope" for the permissions, then the Access button:

ADLS_ACL_Portal_1.jpg

Click the Add button to select a user or a group. Notice the permissions are read/write/execute. You can have the new permission entry add to all existing child folders & files (which you typically want to do). The last radio button is really important - this lets you set it as both an access entry *and* the default entry.

ADLS_ACL_Portal_2.jpg

Important! When using the web interface as shown above, you need to leave the blade open while it assigns permissions. If you navigate away and close it before it completes, the process will get interrupted.

Defining ADLS Data Permissions via PowerShell Script

The technique shown above in the portal is convenient for quick changes, for learning, or for "one-off" scenarios. However, in an enterprise solution, or a production environment, it's a better practice to handle permissions via a script so you can do things such as:

  • Promote changes through different environments

  • Pass off scripts to an administrator to run in production

  • Include permission settings in source control

The two scripts below (option 1 is for groups; option 2 is for users) are split into 4 sub-steps. You might not need all these steps (for instance, if you want to specify permissions at the root level rather than at a subfolder level which is done in the example below).

Important! Definitely make sure to assign the default entries as well as the access entries or you’ll run into problems (the default entries are part of step 4 below). If you were to accidentally set an access entry for a folder without any sort of default entry, new files which are added to the folder won’t have permissions (thus making the files ‘invisible’ to users).

Option 1 - Permissions at the Group Level (recommended)

#This script sets ADLS data-level permissions for one or more AAD groups.
#It is a basic ad hoc script (without error handling, etc).
#Script last tested with PowerShell version 5.1 on Jan 6, 2019.
#-----------------------------------------
#Input Area
#-----------------------------------------
$subscriptionName = 'Microsoft Azure Sponsorship'
$dataLakeStoreName =  'bankingadlsdev.azuredatalakestore.net'
$groupNameArray = @('DataReaderGroup','DataWriterGroup')
$adlsPermissionType = 'Group'
$adlsTraversePermission = 'ReadExecute'
$adlsPathPermission = 'All' #All = Read,Write,Execute
$adlsRoot = '/'
$adlsPath1 = '/ATMMachineData'
$adlsPath2 = '/ATMMachineData/StandardizedData'
#-----------------------------------------
#Manual login into Azure
#-----------------------------------------
#Login-AzureRmAccount -SubscriptionName $subscriptionName
#-----------------------------------------
#Step 1: Set root permissions to traverse
#-----------------------------------------
Write-Host '[*] Begin setting the root permissions to traverse (execute only).'
ForEach ($groupName in $groupNameArray)
    {
        $groupId = Get-AzureRmADGroup -DisplayName $groupName
        Set-AzureRmDataLakeStoreItemAclEntry  `
                -AccountName $dataLakeStoreName `
                -Path $adlsRoot `
                -AceType $adlsPermissionType `
                -Permissions $adlsTraversePermission `
                -Id $groupId.Id    
        Write-Host "[*] Complete setting root permissions for $groupName."
    }
#-----------------------------------------
#Step 2: Set parent folder permissions to traverse
#-----------------------------------------
Write-Host '[*] Begin setting parent folder permission to traverse (execute only).'
ForEach ($groupName in $groupNameArray)
    {
        $groupId = Get-AzureRmADGroup -DisplayName $groupName
        Set-AzureRmDataLakeStoreItemAclEntry  `
                -AccountName $dataLakeStoreName `
                -Path $adlsPath1 `
                -AceType $adlsPermissionType `
                -Permissions $adlsTraversePermission `
                -Id $groupId.Id    
        Write-Host "[*] Complete setting parent folder permissions for $groupName."
    }
#-----------------------------------------
#Step 3: Set ACLs recursively
#-----------------------------------------
Write-Host '[*] Begin setting the ACL (All permissions set recursively on all child items).'
ForEach ($groupName in $groupNameArray)
    {
        $groupId = Get-AzureRmADGroup -DisplayName $groupName    
        Set-AzureRmDataLakeStoreItemAclEntry  `
                -AccountName $dataLakeStoreName `
                -Path $adlsPath2 `
                -AceType $adlsPermissionType `
                -Permissions $adlsPathPermission `
                -Id $groupId.Id `
                -Recurse `
                -Concurrency 128
        Write-Host "[*] Complete setting ACLs for $groupName."
    }
#-----------------------------------------
#Step 4: Set default entries recursively
#-----------------------------------------
Write-Host '[*] Begin setting the default entry so any new items under this path will obtain the access.'   
ForEach ($groupName in $groupNameArray)
    {
        $groupId = Get-AzureRmADGroup -DisplayName $groupName  
        Set-AzureRmDataLakeStoreItemAclEntry  `
                -AccountName $dataLakeStoreName `
                -Path $adlsPath2 `
                -AceType $adlsPermissionType `
                -Permissions $adlsPathPermission `
                -Id $groupId.Id -Default `
                -Recurse `
                -Concurrency 128
        Write-Host "[*] Complete setting default entries for $groupName."
    }

Option 2 - Permissions at the User Level

#This script sets ADLS data-level permissions for one or more AAD users.
#It is a basic ad hoc script (without error handling, etc).
#Script last tested with PowerShell version 5.1 on Jan 6, 2019.
#-----------------------------------------
#Input Area
#-----------------------------------------
$subscriptionName = 'Microsoft Azure Sponsorship'
$dataLakeStoreName =  'bankingadlsdev.azuredatalakestore.net'
$userNameArray = @('WayneWriter@sqlchick.com','RayReader@sqlchick.com')
$adlsPermissionType = 'User'
$adlsTraversePermission = 'ReadExecute'
$adlsPathPermission = 'All' #All = Read,Write,Execute
$adlsRoot = '/'
$adlsPath1 = '/ATMMachineData'
$adlsPath2 = '/ATMMachineData/StandardizedData'
#-----------------------------------------
#Manual login into Azure
#-----------------------------------------
Login-AzureRmAccount -SubscriptionName $subscriptionName
#-----------------------------------------
#Set the data / POSIX/ ACL permissions
#-----------------------------------------
Write-Host '[*] Begin setting the root permissions to traverse (execute only).'
ForEach ($userName in $userNameArray)
    {
        $userId = Get-AzureRmADUser -UPN $userName 
        Set-AzureRmDataLakeStoreItemAclEntry  `
             -AccountName $dataLakeStoreName `
             -Path $adlsRoot `
             -AceType $adlsPermissionType `
             -Permissions $adlsTraversePermission `
             -Id $userId.Id   
        Write-Host "[*] Complete setting root permissions for $userName."
    }
Write-Host '[*] Begin setting the parent folder permission to traverse (execute only).'
ForEach ($userName in $userNameArray)
    {
        $userId = Get-AzureRmADUser -UPN $userName 
        Set-AzureRmDataLakeStoreItemAclEntry  `
             -AccountName $dataLakeStoreName `
             -Path $adlsPath1 `
             -AceType $adlsPermissionType `
             -Permissions $adlsTraversePermission `
             -Id $userId.Id    
        Write-Host "[*] Complete setting root permissions for $userName."             
    }
Write-Host '[*] Begin setting the ACL (All permissions set recursively on all child items).'
ForEach ($userName in $userNameArray)
    {
        $userId = Get-AzureRmADUser -UPN $userName
        Set-AzureRmDataLakeStoreItemAclEntry  `
             -AccountName $dataLakeStoreName `
             -Path $adlsPath2 `
             -AceType $adlsPermissionType `
             -Permissions $adlsPathPermission `
             -Id $userId.Id `
             -Recurse `
             -Concurrency 128
        Write-Host "[*] Complete setting root permissions for $userName."
    }
Write-Host '[*] Set the default entry so any new items under this path will obtain the access.'
ForEach ($userName in $userNameArray)
    {    
        $userId = Get-AzureRmADUser -UPN $userName
        Set-AzureRmDataLakeStoreItemAclEntry  `
             -AccountName $dataLakeStoreName `
             -Path $adlsPath2 `
             -AceType $adlsPermissionType `
             -Permissions $adlsPathPermission `
             -Id $userId.Id -Default `
             -Recurse `
             -Concurrency 128
        Write-Host "[*] Complete setting root permissions for $userName."
    }

Finding More Information

PowerShell Cmdlets for Azure Data Lake Store

Breaking Changes to Data Lake Store Cmdlets

Access Control in Azure Data Lake Store <--Definitely take time to read this

Secure Data in Azure Data Lake Store

Assigning Resource Management Permissions for Azure Data Lake Store (Part 2)

This is part 2 in a short series on Azure Data Lake permissions. 

Part 1 - Granting Permissions in Azure Data Lake
Part 2 - Assigning Resource Management Permissions for Azure Data Lake Store {you are here}
Part 3 - Assigning Data Permissions for Azure Data Lake Store

In this section, we're covering the "service permissions" for the purpose of managing Azure Data Lake Store (ADLS). Granting a role on the resource allows someone to view or manage the configuration and settings for that particular Azure service (i.e., although we're talking about ADLS, this post is applicable to Azure services in general). RBAC, or role-based access control, includes the familiar built-in Azure roles such as reader, contributor, or owner (you can create custom roles as well).

Tips for Assigning Roles for the ADLS Service

Setting permissions for the service + the data stored in ADLS is always two separate processes, with one exception: when you define an owner for the ADLS service in Azure, that owner is automatically granted 'superuser' (full) access to manage the ADLS resource in Azure *AND* full access to the data. Any other RBAC role other than owner needs the data access specifically assigned via ACLs. This is a good thing because not all system administrators need to see the data, and not all data access users/groups/service principals need access to the service itself. This type of separation is true for certain other services too, such as Azure SQL Database.

Try to use groups whenever you can to grant access, rather than individual accounts. This is a consistent best practice for managing security across many types of systems.

If you are using resource groups in Azure the way they are intended to be used, you may be able to define service permissions at the resource group level rather than at the individual resource level (although the example shown is here is setting RBAC for ADLS specifically). Managing permissions at the resource group level reduces maintenance, assuming your resource group isn't too broadly defined.

If you need to allow a user or group to be able to peruse data using the Data Explorer, then they'll need reader permissions to the ADLS Azure service. Basically, if any access to ADLS in the Azure portal is needed, or if managing the ADLS service (such as through ARM or PowerShell or the portal) is needed, then the appropriate RBAC entry is necessary. 

Typically, automated processes which do need access to the data (discussed in Part 3), don't need any access to the ADLS resource itself. In Part 4 we'll talk a bit about using service principals. I've found that frequently a service principal needs data access (ACLs), but not any RBAC access to the service.

The RBAC functionality is consistent across Azure services. When roles are updated for an Azure resource, it is recorded in the Activity Log:

ADLS_RBAC_ActivityLog.jpg

Defining RBAC Permissions in the Azure Portal

Setting up permissions can be done in the portal in the Access control (IAM) pane. (By the way, the IAM acronym stands for Identity and Access Management.)

ADLS_RBAC_Portal.jpg

Defining RBAC Permissions via PowerShell Script

The technique shown above in the portal is convenient for quick changes, for learning, or for "one-off" scenarios. However, in an enterprise solution, and for production environments, it's a better practice to handle permissions via a script so you can do things such as:

  • Promote changes through different environments
  • Pass off scripts to an administrator to run in production
  • Include permission settings in source control

Setting Permissions for a Group

In the following PowerShell script, we are assigning contributor permissions to an AAD group:

ADLS_RBAC_PowerShell.jpg

Here's a copy/paste friendly script from the above screenshot - for a group:

#-----------------------------------------

#Input Area
$subscriptionName = 'YourSubscriptionName'
$resourceGroupName = 'YourResourceGroupName'
$resourceName = 'YourResourceName'
$groupName = 'YourAADGroupName'
$userRole = 'Contributor'

#-----------------------------------------

#Manual login into Azure
Login-AzureRmAccount -SubscriptionName $subscriptionName

#-----------------------------------------

$resourceId = Get-AzureRmResource `
    -ResourceGroupName $resourceGroupName `
    -ResourceName $resourceName 
$groupId = Get-AzureRMADGroup `
    -SearchString $groupName

New-AzureRmRoleAssignment `
    -ObjectId $groupId.Id `
    -RoleDefinitionName $userRole `
    -Scope $resourceId.ResourceId 

Setting Permissions for a User

This next script is nearly the same, but this time we are assigning read+execute permissions to a user instead of a group (which should be the exception not the rule):

ADLS_RBAC_PowerShell_User.jpg
 

And, here's the copy/paste friendly version of the above screenshot - for a user:

#-----------------------------------------

#Input Area
$subscriptionName = 'YourSubscriptionName'
$resourceGroupName = 'YourResourceGroupName'
$resourceName = 'YourResourceName'
$userName = 'YourUserNameInEmailFormat'
$userRole = 'Contributor'

#-----------------------------------------

#Manual login into Azure
#Login-AzureRmAccount -SubscriptionName $subscriptionName

#-----------------------------------------

$resourceId = Get-AzureRmResource `
    -ResourceGroupName $resourceGroupName `
    -ResourceName $resourceName 
$userId = Get-AzureRmADUser `
    -UPN $userName 

New-AzureRmRoleAssignment `
    -ObjectId $userId.Id `
    -RoleDefinitionName $userRole `
    -Scope $resourceId.ResourceId 

The online examples for the New-AzureRmRoleAssignment cmdlet enumerates the IDs or GUIDs, which makes things clear for learning but isn't ideal for operationalized scripts. Therefore, the purpose for $resourceId and $groupId above is to do the work of looking up the GUIDs so you don't have to do that manually.

Personally, I like using PowerShell instead of ARM (Azure Resource Manager) templates for certain things, such as permissions, but you do have additional options beyond what I've discussed here based on what you're most comfortable with.

Finding More Information

Get Started with Role-Based Access Control in Azure