Materialized Lake Views: Simplifying Your Medallion ETL

Description

Materialized lake views turn Spark SQL into auto-orchestrated smart tables in your Fabric lakehouse. In this session, learn how to replace fragile ETL with declarative medallion layers, understand refresh and lineage, and decide when materialized lake views beat notebooks, jobs, or warehouses.

Key Takeaways

• Understand where MLVs fit in your ETL
• See refresh + lineage in a practical Fabric workflow
• Leave with a concrete when to use MLVs decision framework
• Hard-coded dependency chains
• Declarative SQL-first transforms
• Manual retries and orchestration glue
• Platform-managed refresh behavior

My Notes

Action Items

• [ ]

Resources & Links

Slides

Materialized Lake Views
Simplifying Your Medallion ETL
Declarative, auto-orchestrated smart tables for Fabric Lakehouse
Justin Cunningham
Director, Analytics & Engineering
Session Outcome
• Understand where MLVs fit in your ETL
• See refresh + lineage in a practical Fabric workflow
• Leave with a concrete when to use MLVs decision framework
The Current Pain vs The Better Path
Why ETL Gets Fragile
What Modern Teams Need
• Hard-coded dependency chains
• Declarative SQL-first transforms
• Manual retries and orchestration glue
• Platform-managed refresh behavior
• Pipeline drift over time
• Transparent lineage and impact
• Slow incident triage
• Lower operational overhead
FOUNDATION
Medallion + Materialized Lake Views
Medallion Overview
MLVs can be used to declaratively formalize layer transitions
Layer
Goal
MLV Fit
Bronze
raw immutable history, minimal transformations
Silver
standardized and quality-controlled
Gold
business-ready curated outputs
What MLVs Change
• Define transformations as SQL, persist results as smart tables
• Let them manage refresh dependencies
• Use lineage to quickly evaluate downstream impact
• Reduce the number of notebooks/pipelines you have to maintain
Refresh + Lineage: Operational Impact
Refresh Gains
Lineage Gains
• More predictable update behavior
• Faster root-cause analysis
• Fewer custom scheduler dependencies
• Safer schema change planning
• Consistent data freshness posture
• Better governance visibility
• Improved resilience during failures
• Cleaner stakeholder communication
CHOICE
Decision Framework
When to Use MLVs vs Alternatives
MLVs are ideal when…
Prefer other options when…
• Transforms are SQL-centric
• Heavy procedural logic is needed
• Maintainability > orchestration control
• ML/non-SQL processing dominates
• Built-in dependency handling is wanted
• Custom runtime control is required
• Lineage transparency is important
• Warehouse semantics are mandatory
Tradeoffs
• MLVs simplify operations but reduce low-level runtime tuning flexibility
• Performance depends on query shape, partitioning, and workload patterns
• Operational simplicity often compounds value over time
• Start where orchestration pain is highest to maximize ROI
Demo Narrative
Step 1
Step 2
Step 3
Step 4
Bronze source + baseline pain
Silver/Gold MLV definitions
Refresh + lineage walkthrough
MLV comparison
Demo
30-Day Adoption Playbook
• Pick 1–2 brittle ETL flows as pilot candidates
• Refactor logic into declarative MLV statements
• Validate refresh and lineage expectations
• Standardize patterns and then scale
Key Takeaways
Materialized Lake Views: Simplifying Your Medallion ETL
• Materialized Lake Views can simplify medallion ETL significantly
• Refresh + lineage improve reliability, operability, and trust
• Use a decision framework — fit-for-purpose wins
• Pilot fast, prove value, then scale with standards
QA & Discussion
• Where is orchestration fragility hurting your team most today?
• What workloads are best first candidates for MLV adoption?
• What would successful implementation look like in your org?
Justin Cunningham
Thank you for attending this session!
Director, Analytics & Engineering
How was
the session?
Complete Session Surveys in
for your chance to WIN
PRIZES!
Sound off.
The mic is all yours.
Influence the product roadmap.
Join the Fabric User Panel
Join the SQL User Panel
Share your feedback directly with our
Fabric product group and researchers.
Influence our SQL roadmap and ensure
it meets your real-life needs
https://aka.ms/JoinFabricUserPanel
https://aka.ms/JoinSQLUserPanel