Synapse Analytics Workspace CI/CD

In this article we’ll review how to implement continuous integration and continuous delivery on Azure Synapse Analytics Workspace using Azure DevOps. We’ll be covering how to setup source control, and how to build & deploy ARM templates.

Setup source control

Setting the source control for Synapse Analytics is absolutely the same as Data Factory. Skip this step if you’re already familiar with it.

Login to Azure DevOps open your project and navigate to Project settings > Repositories > Create

Leave the repository type to “Git”, type a name for the repository that makes sense for you and click “Create”.

Now login to Azure Synapse studio and navigate to Manage > Git configuration > Configure. Select Azure DevOps Git and your Azure Active Directory and click Continue.

Then you’ll have to select the DevOps organization, project name and repository name. For collaboration branch leave it as “main” and for publish branch as “workspace_publish” (or you can name it different, if you want). You can also import existing resources to the repository in case you already have objects you want to import. Click “Apply” when ready, it’ll take few seconds to setup.

And now you have source control enabled for your Synapse Analytics.

Setup Continuous Integration

Go to Pipelines > Pipelines > and click “Create”. (We can also create a pipeline folder if we wish to separate the services to be in different folders)

Select “Azure Repos Git”.

Select “synapse_analytics”

Select “Starer pipeline”.

And finally type the following code into the workspace_publish branch.

    - workspace_publish

   vmImage: "windows-latest"

- checkout: self
  path: src

- checkout: git://IvoTalksTech/IvoTalksTech

- task: AzurePowerShell@5
  displayName: 'Override Synapse Parameters'
    azureSubscription: 'ivo-devops-sp'
    ScriptType: 'FilePath'
    ScriptPath: 'IvoTalksTech/devops_scripts/asa_override_params_all.ps1'
    azurePowerShellVersion: 'LatestVersion'
    workingDirectory: '$(Pipeline.Workspace)\src'

- task: CopyFiles@2  
    SourceFolder: '$(Pipeline.Workspace)\src'
    Contents: '**/*Workspace.json'
    TargetFolder: '$(Build.ArtifactStagingDirectory)\src'
    CleanTargetFolder: true
    OverWrite: true
    flattenFolders: true
  displayName: 'Extract ARM templates'

- publish: $(build.ArtifactStagingDirectory)
  artifact: drop
  displayName: 'Publish artifacts'

- task: UniversalPackages@0
  displayName: 'Universal publish'
    command: 'publish'
    publishDirectory: '$(Build.ArtifactStagingDirectory)'
    feedsToUsePublish: 'internal'
    vstsFeedPublish: 'XXXXX/XXXXX'
    vstsFeedPackagePublish: 'drop-asa-prod'
    versionOption: 'patch'
    packagePublishDescription: 'ver'
  • The trigger step will start the build pipeline every time the code is published from the Synapse Analytics Workspace.
  • For pool we’re using a regular Windows OS for the Virtual Machine.
  • For checkout we’re getting the branch that is currently in use and a second repository where the DevOps scripts are located. (This will come clear in the next step)
  • The PowerShell task is to run a script that overrides the ARM template parameters (more on this later).
  • The Copy task is to extract the ARM template in the Artifact directory.
  • The next task is to publish the artifact in the pipeline.
  • The last task is to publish the artifact in the artifacts feed. Please edit “vstsFeedPublish” to your artifact feed. If you don’t know the link you can click on “Settings” and select it from the dropdown menu. If you don’t have artifact feed, you can create one by going to Artifacts > Create Feed.

Now for the PowerShell task, you can either add it inline or store it in different repo. For easier maintenance we’re storing it in different repo, where the script is overriding the ARM template parameters (i.e. values that need to be changed in accord to the target environment before we deploy). There are several approaches here, one is to use regular expressions, so all values would be changed:

(Get-Content TemplateParametersForWorkspace.json) | Foreach-Object {
    $_ -replace 'ivo-asa-dev', 'ivo-asa-prod' 
    } | Set-Content P_TemplateParametersForWorkspace.json

Other is to specifically edit each value in the template:

$json=Get-Content -Raw "TemplateParametersForWorkspace.json" | ConvertFrom-Json
$json.parameters.workspaceName.value = "ivo-asa-prod"
$json | ConvertTo-Json | Set-Content "P_TemplateParametersForWorkspace.json"

Or you can also use the combination of the two. The last approach is to not to override the parameters in the build at all, but to edit them in the release pipeline, but I don’t recommend that because it’ll require you to edit the release pipeline each time a new ARM template parameter value needs to be overridden, which is suboptimal from maintenance perspective. But feel free to pick whatever approach works for you the best.

From authorization point the “ivo-devops-sp” is a service connection to a service principal with Contributor RBAC.

You can also rename the build pipeline, because by default the name is the same as the repository.

Setup Continuous Delivery

Go to Pipelines > Releases > New release pipeline:

Select “empty job”:

Change the Pipeline name and Stage name:

For artifact choose the build pipeline we’ve just created:

Enable continuous deployment trigger for the artifact from the build pipeline. This will activate the deployment each time the build pipeline completes.

If needed, we can also add pre-deployment conditions like manual approval step. We can add either specific persons or a whole team.

Now click on the Stage’s job and search for “Synapse workspace deployment” addon from the Microsoft’s marketplace:

This will refer you to the marketplace

And then the Marketplace will ask for which DevOps organization you want to install the plugin:

Finally add the Synapse workspace deployment task to your agent job, where:

  1. Template would be your ARM template: (i.e. $(System.DefaultWorkingDirectory)/_synapse_workspace/drop/src/TemplateForWorkspace.json)
  2. Template parameters would be the ARM template parameters: (i.e. $(System.DefaultWorkingDirectory)/_synapse_workspace/drop/src/TemplateForWorkspace.json)
  3. Synapse workspace connection type: Your service principal with Contributor RBAC to the resource group/subscription.
  4. Synapse workspace resource group: The resource group where your Synapse Analytics is located.
  5. Synapse workspace name: The name of your Synapse workspace.

Finally go to your target Synapse, open the studio, the go to Manage > Access Control > Add. Select workspace scope, Synapse administrator role, the service principal we’re deploying with as user and click apply. This will give permissions to the app registration to deploy the synapse workspace artifacts.

CI/CD test run

Let’s start by creating new branch. You can login to Azure DevOps. (Go to your project’s repo, click on “New branch” add name and click “create”) or you can do the same via Azure Synapse studio.

Then for the test, let’s go to the branch we’ve just created and add some objects there, like a SQL script and commit the code.

Create and complete a new pull request from the feature branch to master:

And publish the changes from Synapse Studio:

This will trigger the build pipeline:

You can check the pipeline progress by clicking on it.

Clicking on the published artifact we can see all the generated files.

Clicking on the job we can see all steps. We can clearly see that everything completed successfully and fast.

On the artifact feed we are also storing a link to the pipeline.

After the build pipeline completes (the continuous integration part), the release pipeline starts (the continuous delivery part). We’ve added manual pre-deployment condition to approve.

Clicking on the pipeline logs will show the whole deployment process and we can see that everything passed successfully.

Going to Azure Synapse studio we can also see that the new object is there.

Also, by default we have mail notification for the pipelines:

Enjoy the CI/CD setup!

Stay Awesome,

Leave a Reply

Your email address will not be published.