Description

Discover how AI transforms Fabric development! Learn our production-proven approach combining GitHub Copilot with Azure DevOps CI/CD for metadata-driven pipelines. We'll showcase automated notebook validation, security scanning, and AI-guided PR reviews—all without external dependencies. Real code, real results: from medallion architecture to zero-downtime deployments.

Key Takeaways

• Helped teams get their Fabric CI/CD working without losing their minds
• You work with Fabric (or thinking about it)
• You've copy-pasted notebooks between workspaces and hated it
• You know Git exists (maybe even used it once or twice)
• Someone said "DevOps" and you're here to figure out what that means
• You've played around in Fabric workspaces (notebooks, lakehouses)

My Notes

Action Items

• [ ]

Resources & Links

Slides

ATLANTA MARCH 16 20, 2026
AI-Powered CI/CD for Microsoft
Fabric: Building Enterprise-Grade
Pipelines with GitHub Copilot
Eugene Paden
CTO – Ray Business Technologies
About Me + About You
Who Am I?
• Helped teams get their Fabric CI/CD working without losing their minds
About You (probably)
• You work with Fabric (or thinking about it)
• You've copy-pasted notebooks between workspaces and hated it
• You know Git exists (maybe even used it once or twice)
• Someone said "DevOps" and you're here to figure out what that means
You'll Get the Most Out of This If
• You've played around in Fabric workspaces (notebooks, lakehouses)
• You've heard of CI/CD (even if you've never set it up)
• You're comfortable clicking around Azure DevOps or GitHub
Don't Panic If You Haven’t We'll explain the important stuff as we go. The demos will make it clear.
Not on Today's Menu (so you're not waiting for it):
• Fabric licensing decisions or capacity planning
• Deep-diving into Spark performance tuning
• Teaching you Python, Scala, or SQL from scratch
• Fixing your Git merge conflicts (sorry!)
What You'll Actually Walk Away With
The Good Stuff You Can Use Right Away
• Azure Pipelines CI/CD template
• Artifact validators you can drop into your project today
• Git branch strategy that won't make you cry at 2am
• Github Copilot Skills – Create PR, Review PR
• BONUS 1-hour free consultation we'll look at your setup and tell you
what's going to break
This is stuff you can implement Monday morning without asking for budget approval
How This Actually Works
What We're Building Today
• Real-world patterns from actual Fabric projects (not theory)
• 5-dimension quality framework (sounds fancy, saves your bacon)
• Let GitHub Copilot write your CI/CD
• Deploy without downtime (yes, really)
Fabric Already Gives You Some Good Stuff
• Git Integration: 40+ item types, built-in auto-sync
• Deployment Pipelines: Visual UI, item pairing
• REST API Suite: Full automation capabilities
We Add Enterprise CI/CD Patterns
• Pre-deployment validation (schema, security, quality gates)
• AI-assisted development (GitHub Copilot integration)
• Zero-downtime deployments (Git branch orchestration)
• Rollback procedures
Partnership Approach: Build on Microsoft's platform with proven automation patterns
Session Focus: Practical patterns you can implement immediately
The Pain We've All Felt
The Good News: "We've made all these mistakes so you don't have to. Seriously, we've broken
production in creative ways you haven't even thought of yet. Let's chat after and I'll tell you war stories."
Survey Says….
How We're Fixing This
AI-Assisted Development (eliminates the learning curve)

• Copilot generates CI/CD workflows from plain English
• No need to memorize Git commands or YAML syntax
• You focus on data, AI handles automation
  Data-Centric CI/CD Patterns (adapted for data workflows)
• Schema validation against DBML (catch breaking
  changes early)
• Security scanning (credentials, SQL injection)
• Git branch strategy for zero-downtime deployments
• Artifact-specific validators (notebooks, pipelines,
  transformations)
  "Data engineers stay productive with DevOps patterns without needing to become DevOps engineers."
  Quick Context What's Fabric Data Transformer?
  The Framework We're Using in These Demos
  FDT in 60 Seconds
  • Scala library for spark – reads YAML transformation configurations and executes them
  • YAML defines transformations – FDT runtime does the heavy lifting
  • 30 pre-built transformers All the usual suspects Filter, Join, Merge, Aggregate, Validate,
  Union, Pivot, Unpivot…
  • Data Quality Built-In Pre-write validation rules (34), post-write integrity checks (10),
  scheduled data audits (14)
  • Enterprise-grade Built-in validation, performance tuning, security stuff, and we have
  license tiers (gotta pay the bills)
  Why This Matters for CI/CD
  • YAML = Code Version control, review, validate transformations like code
  • Schema Validation CI pipeline validates YAML against DBML schemas
  • Security Scanning CI detects SQL injection in filter expressions
  • Data Quality Gates Pre-write validation rules catch issues before write, post-write
  checks confirm correctness
  • Consistent Patterns All transformations follow same structure
  For This Session We're focusing on the CI/CD automation, not the framework
  itself (that's a different talk)
  How It Works

YAML Configuration (What You Write)

source:
tableName: "bronze.TRIPS"
format: "delta"
filter: "status = 'completed'"
transformations:
type: RemoveDuplicates
columns: ["trip_id"]
type: Validate
mode: "failOnError"
rules:
ruleType: NotNull
name: "trip_id_not_null"
description: "Trip ID and pickup time must exist"
columnNames: ["trip_id", "pickup_datetime"]
severity: "ERROR"
destination:
tableName: "silver.TRIPS"
format: "delta"
writeMode: "overwrite"
Let AI Build Your Pipeline
AI Writes All Your CI/CD Code (Seriously)
What You Still Do Manually (in Fabric Workspace UI the point-and-click stuff)
• Create lakehouses (LH_Control, LH_Bronze, LH_Silver, LH_Gold)
• Create warehouse (WH_Control)
• Create notebooks (your actual data pipelines)
• Create pipelines (orchestration stuff)
• Configure variable library
• Set up Git integration
AI builds automation AROUND your manually-created Fabric stuff (it's not replacing your workspace, just
making it safer)
DEMO 1 – Setup CI and Validators
• Git Integration (8s)
• Setup Raw Data (34s)
• Copilot Instructions and Generate CI Pipeline (2:13)
• Create PR and PR Review Skills (1:29)
• Load into Bronze Layer, CI Run (1:27)
• Additional AI Artifact Generation and Validations
• dbml
• FDT Generate Models (2:48)
• yaml
• FDT Bronze to Silver Transformations (1:00)
• FDT Silver to Gold Transformations (1:00)
• FDT Execution and AI Log Analysis – (1:40)
Branch Policies and Git Integration
Load Raw Data
Copilot Instructions and CI Pipeline
Create PR and PR Review Skills
Load Bronze Layer, CI Run
DBML/YAML AI Artifacts and Validations
Transformation Execution and AI Log Analysis
What The Validators Actually Check
Real Examples of Issues We Catch
DEMO 2 – Generate and Execute CD
• Create and configure CD Pipeline (1:14)
• Execute CD Pipeline (1:50)
CD Pipeline Definition
CD Pipeline Execution
The Git Branch Magic How This Actually Works
Behind the Scenes: Fabric + Git Integration
Why This Matters (Architecture Trade-offs)

• Complete isolation No cross-environment
  contamination
• Instant rollback Change Git branch pointer
• Zero downtime Atomic updates (all resources or
  none)
• Audit trail Git history tracks all changes
• Workspace duplication 3x storage (Dev, QA, Prod)
• Branch proliferation Old releases archived monthly
• Capacity planning Each workspace needs separate
  capacity (Dev/QA can share)
  Putting It All Together - The Complete Pipeline
  The 5-Dimension Quality Framework (Your New Checklist)
  The Stuff You Need to Remember
  If You Forget Everything Else, Remember These
  • Validate in CI, not production - catch errors before they break stuff
  • Let Copilot write your CI/CD - 5x faster, better quality
  • Deploy with Git branches - zero downtime, instant rollback
  • Security from day one - no creds in code, validate everything
  • Start small, build iteratively - don't try to do everything at once
  • Use YAML, not code - for transformations (easier to maintain, version, and review)
  Sound off.
  The mic is all yours.
  Influence the product roadmap.
  Join the Fabric User Panel
  Join the SQL User Panel
  Share your feedback directly with our
  Fabric product group and researchers.
  Influence our SQL roadmap and ensure
  it meets your real-life needs
  https://aka.ms/JoinFabricUserPanel
  https://aka.ms/JoinSQLUserPanel