Categories
Azure

Automatic creation of directory structure in Data Lake Gen2 container

In this article we’ll review how we can automatically create directory structure in Data Lake Gen2 container utilizing Azure DevOps Repo and Release pipeline with custom PowerShell script. We’ll create new repo for Data Lake, add the PowerShell script in different repo where we store the DevOps scripts (or use inline), and create release pipeline […]

Categories
Azure

Synapse Analytics Workspace CI/CD

In this article we’ll review how to implement continuous integration and continuous delivery on Azure Synapse Analytics Workspace using Azure DevOps. We’ll be covering how to setup source control, and how to build & deploy ARM templates. Setup source control Setting the source control for Synapse Analytics is absolutely the same as Data Factory. Skip […]

Categories
Azure

Data Factory CI/CD

In this article we’ll review how to implement continuous integration and continuous delivery on Azure Data Factory using Azure DevOps. We’ll be covering how to setup source control, and how to build & deploy ARM templates, including the new auto-validating and auto-publishing capabilities that Microsoft just introduced. We’ll start with source control setup, then we’ll […]

Categories
Azure

Databricks CI/CD

In this article we’ll review how to implement continuous integration and continuous delivery on Azure Databricks using Azure DevOps. We’ll be covering how to build and deploy Databricks notebooks, interactive clusters and libraries for the interactive clusters. We’ll start by organizing the authorization via Service Principals and Databricks Personal Access Tokens, including setup of service […]

Categories
Azure

Track Data Factory scheduled triggers inside PowerBI Report

I’m assuming you already have:1. DevOps organization available with project’s service connection setup.(i.e. permissions to view your Data Factory)2. PowerBI Desktop installed or license to use PowerBI online. Currently there is no way to clearly see all scheduled triggers in Data Factory. You can check them one by one, but if you have very big […]

Categories
Azure

Tracking Azure Quotas and Usage

I’m assuming you already have:1. Log Analytics.2. Automation Account (Hybrid Worker Group not required). On subscription level, currently there is no diagnostic settings that can be linked to Log Analytics to track quotas and usage. We’ll be creating PowerShell script into Automation Account Runbook to get the current quotas and usage and save them in […]

Categories
Azure

Scheduled scaling of PowerBI Embedded Gen1

I’m assuming you already have:1. PowerBI Embedded Gen12. Automation Account with Hybrid Worker Group and dedicated system user.Note: The example can be rework to work even without the hybrid worker group. The idea is to have PowerShell script that can change the capacity of the PowerBI, which can be authenticated with specific system user that […]

Categories
Azure

Auto-provision Databricks AD groups that match specific RegEx

I’m assuming you already have:1. Configured Databricks with SCIM provissioning connector 2. Active Directory Plan eligable for these operations3. Automation Account from which you have specific system user you authenticate with. Also these auto-assignemnts of users and/or groups works for other Enterprise Applications or Application Registration as well. Go to Enterprise Applications, locate your configured […]