Module-5 – Azure Lakehouse

🎯 Objectives

• Secure & configure ADLS Gen2 storage.
• Use Azure Databricks & Unity Catalog for governance.
• Implement Bronze/Silver/Gold medallion pattern.

🗓️ Weekly Plan

• Week 17 – ADLS Gen2: hierarchical namespace, RBAC, firewall.
• Week 18 – Databricks: workspace, clusters, notebooks, jobs.
• Week 19 – Unity Catalog: metastore, schemas, permissions.
• Week 20 – Lakehouse best practices & folder layout.

🔑Key Concepts

ADLS Gen2 Features

• Hierarchical namespace, storage tiers, lifecycle policies.

Security & Networking

• RBAC vs POSIX ACLs, service principals, private endpoints.

Databricks Administration

• All-purpose vs job clusters, autoscaling, init scripts.

Unity Catalog Governance

• Centralized metastore, grant/revoke, lineage.

Medallion Architecture

• Bronze (raw), Silver (clean), Gold (aggregates).

🔨 Mini-Projects

• Secure ADLS: implement ACLs, private endpoint, firewall rules.
• DBX Ingest Job: schedule notebook to load into Bronze.
• Governance Demo: set up Unity Catalog roles & access.

📚 Resources

• ADLS Gen2 documentation
• Databricks Unity Catalog guide
• Microsoft Fabric Lakehouse whitepapers