In this post, I would like to present how the SHALLOW CLONE functionality can be utilized to streamline your development process by accelerating the creation of dedicated development environments and enabling comprehensive data testing in your CI/CD pipeline. One more time, what exactly is SHALLOW CLONE? Let’s start with a quick recap of Delta Lake […]
Author: Tomasz Kostyrka
githooks and pre-commit framework in the daily work of a Data Engineer
Do you recall a situation when, at the beginning of the project, your team, after many days of intense discussions, finally established coding standards that everyone agreed to follow? You thoroughly discussed rules regarding capitalization, spacing, indentation, line lengths, etc. You nearly came to blows over whether the comma should be placed at the beginning […]
Azure Policy – an underrated component of a scalable data platform (part2)
In the first part of the article, we discussed the basics of the Azure Policy service and provided examples of using definitions with audit and deny effects, or in simpler terms, policies that block the creation of incorrect resources. In this part, we will focus on a slightly more advanced concept: remediation, which involves automatically […]
Azure Policy – an underrated component of a scalable data platform (part1)
Recently, we’ve been encountering an increasing number of projects where a complete data platform has to be designed and built almost from scratch. In such projects, in addition to the typical duties and tasks in the data area, such as data modeling, designing and implementing pipelines, or the final reporting layer, there is a growing […]