Categories
Azure

Backup Data Lake Gen2 containers with Data Factory

In this article we’ll review how to create a Data Factory pipeline, linked services, dataset and trigger in order to copy files between Data Lakes. The storage accounts can be setup to have disaster recovery and replication (and some storage accounts have soft-deletes), but this does not cover accidental deletes cause by human error. We’ll […]

Categories
Azure

Key Vault adoption for Data Factory and Databricks

In this article we’ll review how to create a Key Vault and setup access policies so it can be used from Data Factory and Databricks. Then setup Data Factory linked service and Databricks secret scope to the Key Vault. Setup Key Vault access policies To get Key Vault secrets from any service, it needs to […]

Categories
Azure

Azure Active Directory authorization strategy utilizing nested groups

In this article we’ll review how we can create Azure Active Directory authorization strategy utilizing nested groups with assigned role-based access control (RBAC) and access control lists (ACL). What’s the difference between the two access types. How we can access resources and fine-grain the control with examples for Data Lake and Databricks. Role-Based Access Control […]

Categories
Azure

Automatic creation of directory structure in Data Lake Gen2 container

In this article we’ll review how we can automatically create directory structure in Data Lake Gen2 container utilizing Azure DevOps Repo and Release pipeline with custom PowerShell script. We’ll create new repo for Data Lake, add the PowerShell script in different repo where we store the DevOps scripts (or use inline), and create release pipeline […]

Categories
Azure

Synapse Analytics Workspace CI/CD

In this article we’ll review how to implement continuous integration and continuous delivery on Azure Synapse Analytics Workspace using Azure DevOps. We’ll be covering how to setup source control, and how to build & deploy ARM templates. Setup source control Setting the source control for Synapse Analytics is absolutely the same as Data Factory. Skip […]

Categories
Azure

Data Factory CI/CD

In this article we’ll review how to implement continuous integration and continuous delivery on Azure Data Factory using Azure DevOps. We’ll be covering how to setup source control, and how to build & deploy ARM templates, including the new auto-validating and auto-publishing capabilities that Microsoft just introduced. We’ll start with source control setup, then we’ll […]

Categories
Azure

Databricks CI/CD

In this article we’ll review how to implement continuous integration and continuous delivery on Azure Databricks using Azure DevOps. We’ll be covering how to build and deploy Databricks notebooks, interactive clusters and libraries for the interactive clusters. We’ll start by organizing the authorization via Service Principals and Databricks Personal Access Tokens, including setup of service […]

Categories
Azure

Track Data Factory scheduled triggers inside PowerBI Report

I’m assuming you already have:1. DevOps organization available with project’s service connection setup.(i.e. permissions to view your Data Factory)2. PowerBI Desktop installed or license to use PowerBI online. Currently there is no way to clearly see all scheduled triggers in Data Factory. You can check them one by one, but if you have very big […]

Categories
Azure

Tracking Azure Quotas and Usage

I’m assuming you already have:1. Log Analytics.2. Automation Account (Hybrid Worker Group not required). On subscription level, currently there is no diagnostic settings that can be linked to Log Analytics to track quotas and usage. We’ll be creating PowerShell script into Automation Account Runbook to get the current quotas and usage and save them in […]

Categories
Azure

Scheduled scaling of PowerBI Embedded Gen1

I’m assuming you already have:1. PowerBI Embedded Gen12. Automation Account with Hybrid Worker Group and dedicated system user.Note: The example can be rework to work even without the hybrid worker group. The idea is to have PowerShell script that can change the capacity of the PowerBI, which can be authenticated with specific system user that […]