Author: Tomasz Kostyrka

Azure, Azure Data Factory, DevOps, Key Vault, Storage

Terraforming ADF: Credentials & Linked Services

Azure Data Factory is almost never created as an isolated resource. In nearly every project where we have used this service, it was provisioned alongside a Key Vault and a Storage Account. In the following article, I’ll describe the standard method for authenticating between these services using managed identities. As indicated by the title, I’ll […]

Azure, Azure Data Factory, DevOps, PowerShell

Terraforming ADF: Create & Configure Self-Hosted Integration Runtimes

Git and mandatory CI/CD have become widely accepted standards, and an increasing number of people are advocating for defining cloud resources using IaC tools. Many are coming to understand that quality, reliability, and scalability in modern cloud data projects are simply unattainable without embracing the DevOps culture and proper automation. The ClickOps resistance movement is […]

Databricks, Delta Lake, DevOps

Elevate your Databricks development workflow with SHALLOW CLONE

In this post, I would like to present how the SHALLOW CLONE functionality can be utilized to streamline your development process by accelerating the creation of dedicated development environments and enabling comprehensive data testing in your CI/CD pipeline. One more time, what exactly is SHALLOW CLONE? Let’s start with a quick recap of Delta Lake […]

DevOps

githooks and pre-commit framework in the daily work of a Data Engineer

Do you recall a situation when, at the beginning of the project, your team, after many days of intense discussions, finally established coding standards that everyone agreed to follow? You thoroughly discussed rules regarding capitalization, spacing, indentation, line lengths, etc. You nearly came to blows over whether the comma should be placed at the beginning […]