You’ve probably heard about a new mode for datasets/semantic models called Direct Lake. This mode combines the advantages of existing modes: the performance of Import mode and the lack of need to refresh or cache data from Direct Query. It works by directly accessing files from a one lake without requiring extra steps. This means […]
Latest Posts
Notebook orchestration in Microsoft Fabric made easy
When I meet with customers, we discuss how to efficiently organize notebook execution via orchestration mechanism. They often mention managing it manually using tools like Azure Data Factory. On the other hand, they sometimes express that Data Factory is too simplistic and they require other tools to address their needs. Similar questions arise regarding Microsoft […]
Retrieve execution query history in Databricks
Collecting the history of query and command executions in Azure Databricks is essential for troubleshooting, optimizing performance, and ensuring robust security measures. This practice establishes an audit trail for swift issue resolution, aids in creating a usage baseline for objects, and contributes to maintaining a secure data processing environment. The historical data becomes a valuable […]
Easy way to retrieve metadata from loaded files in Azure Databricks
As you continue adding more files to your table, it keeps growing larger. One day, while reviewing it, you notice some records in the middle that contain mistakes. To troubleshoot this in an easy way, it’s essential to have proper metadata about the files you have already loaded. How can you obtain this information? Another […]
Last comments