Module-6 – ADF & Synapse

🎯 Objectives

• Build advanced ETL/ELT pipelines in Azure Data Factory.
• Tune Synapse serverless & dedicated SQL pools for performance & cost.
• Automate CI/CD for data workloads.

🗓️ Weekly Plan

• Week 21 – ADF fundamentals: pipelines, activities, parameters.
• Week 22 – Mapping Data Flows: transformations & expression builder.
• Week 23 – Synapse Analytics: serverless vs dedicated tuning.
• Week 24 – CI/CD: ARM templates, Git integration, releases.

🔑Key Concepts

ADF Core Concepts

• Linked services, datasets, pipelines, triggers.

Integration Runtimes

• Azure vs self-hosted IR, cost & performance trade-offs.

Mapping Data Flows

• Derived columns, aggregations, join behavior.

Synapse Analytics

• Serverless SQL pool vs dedicated SQL pool: MPP, DWUs, distributions.

CI/CD Practices

• ARM templates, Git branch strategies, release pipelines.

🔨 Mini-Projects

• End-to-End ADF: API → ADLS → Databricks → Synapse.
• Complex Data Flow: multi-step transformation & error handling.
• Pipeline CI: deploy via Azure DevOps or GitHub Actions.

📚 Resources

• Azure Data Factory documentation
• Azure Synapse performance tuning guide
• Azure DevOps Pipelines docs