Categories
Azure

Backup Data Lake Gen2 containers with Data Factory

In this article we’ll review how to create a Data Factory pipeline, linked services, dataset and trigger in order to copy files between Data Lakes. The storage accounts can be setup to have disaster recovery and replication (and some storage accounts have soft-deletes), but this does not cover accidental deletes cause by human error. We’ll […]

Categories
Azure

Key Vault adoption for Data Factory and Databricks

In this article we’ll review how to create a Key Vault and setup access policies so it can be used from Data Factory and Databricks. Then setup Data Factory linked service and Databricks secret scope to the Key Vault. Setup Key Vault access policies To get Key Vault secrets from any service, it needs to […]

Categories
Azure

Data Factory CI/CD

In this article we’ll review how to implement continuous integration and continuous delivery on Azure Data Factory using Azure DevOps. We’ll be covering how to setup source control, and how to build & deploy ARM templates, including the new auto-validating and auto-publishing capabilities that Microsoft just introduced. We’ll start with source control setup, then we’ll […]

Categories
Azure

Track Data Factory scheduled triggers inside PowerBI Report

I’m assuming you already have:1. DevOps organization available with project’s service connection setup.(i.e. permissions to view your Data Factory)2. PowerBI Desktop installed or license to use PowerBI online. Currently there is no way to clearly see all scheduled triggers in Data Factory. You can check them one by one, but if you have very big […]

Categories
Azure

Auto-provision Databricks active table access and configure consumption for PowerBI

I’m assuming you already have:1. Databricks with admin priviliges.2. Data Factory with contributing access.3. PowerBI Desktop installed. The idea is to update the access for Databricks tables for specific AD groups. If you’re interested in auto-provisioning Databricks AD Groups you can follow up this article: https://ivotalkstech.com/azure/auto-provision-databricks-ad-groups-that-match-specific-regex Go to your Databricks admin console and make sure […]