This is a quick post to share something that happened on a project recently. We began to experience some intermittent issues with Azure Data Factory (V1) and it was ultimately related to an out-of-date AzureRM PowerShell module. What does ADF have to do with the AzureRM PowerShell module you ask? For this project, the nightly loads are triggered by a signal file which indicates the source data is ready. Therefore, we have a PowerShell script that controls the whole end-to-end process (which overrides the ADF V1 built-in scheduling). The PowerShell script looks for the signal file and then proceeds to resume the Azure Data Factory pipelines (via the Resume-AzureRmDataFactoryPipeline cmdlet). The resume causes each pipeline to execute immediately. When all pipelines are finished, they all get suspended again (via the Suspend-AzureRmDataFactoryPipeline cmdlet) until the next execution of the data load process. This PowerShell process runs on a virtual machine.
In production recently, we started seeing cases of ADF pipelines that wouldn't resume properly which resulted in some data not getting loaded properly. It was inconsistent and intermittent. With the exact same PowerShell script, we couldn't reproduce the issue in UAT. The issues were only occurring in production, but not with precise regularity.
My colleague on the project, Terry Crist, did some investigation and found that the AzureRM module installed on UAT was much newer than what was installed on production. Sure enough, once the AzureRM module was updated in production everything began to run reliably again.
So, this served as a good reminder to ensure that (a) UAT and production environments should be running on the same PowerShell module version, and (b) preferably the latest version should be installed when possible. In environments which don't have a full-time DBA looking after this sort of thing it's good for developers to know to watch out for these types of issues too.
You Might Also Like...
Find Pipelines Currently Running in Azure Data Factory with PowerShell