Collecting the history of query and command executions in Azure Databricks is essential for troubleshooting, optimizing performance, and ensuring robust security measures. This practice establishes an audit trail for swift issue resolution, aids in creating a usage baseline for objects, and contributes to maintaining a secure data processing environment. The historical data becomes a valuable […]
Latest Posts
Easy way to retrieve metadata from loaded files in Azure Databricks
As you continue adding more files to your table, it keeps growing larger. One day, while reviewing it, you notice some records in the middle that contain mistakes. To troubleshoot this in an easy way, it’s essential to have proper metadata about the files you have already loaded. How can you obtain this information? Another […]
Processing stream data with Microsoft Fabric Event Streams (part2)
In my earlier article (link), I initiated a discussion on the functionality of Microsoft Fabric Event Streams. I provided an overview of the tool and covered the steps for connecting to data sources. In this follow-up article, I will guide you through the process of data processing and demonstrate how to efficiently store it in […]
Incrementally loading files from SharePoint to Azure Data Lake using Data Factory
Introduction Companies store a lot of useful data in SharePoint, OneDrive (which is essentially SharePoint), and MS Teams (also based on SharePoint). If you want to enhance your data platform with information that is usually stored in various file formats (Excel, MS Access, CSV, and others), you’ll need to save it to a more user-friendly […]
Last comments