Managing deployments across multiple workspaces per environment in Microsoft Fabric can be a challenge. Based on real projects, this session shows how to use Fabric Deployment Pipelines with proven patterns, tips, and best practices for handling cross workspace dependencies through autobinding and variable library strategies, supported by practical demos that highlight the most important concepts.
Key Takeaways
Autobinding & Variable Library review
Running Fabric deployment pipelines from Azure DevOps
Limitations & considerations
When doing this in one single
Ingests data from source systems
Central raw data landing zone
Supplies data to domain workspaces
My Notes
Action Items
[ ]
Resources & Links
Slides
Agenda • Introductions • Problem overview • Example architecture • Autobinding & Variable Library review • Running Fabric deployment pipelines from Azure DevOps • Demo • Limitations & considerations Atte Sukari Data enthusiast from Finland Senior Data Engineer at Norrin M.Sc. in Economics & Business Favorite tools & technologies: Azure • Fabric • Databricks • Data platforms linkedin.com/in/attesukari Aleksi Partanen Microsoft MVP & YouTuber Data Architect & Team Lead at Norrin Founder at CertiAce (certiace.com) Founder at Fabric Forge (skool.com/fabricforge) Aleksi Partanen All my links! https://linktr.ee/aleksipartanen Problem in a nutshell • When doing this in one single workspace thins are quite straight forward and deploying from one environment to another is typically simple Development Test Production Development Test Production • When having multiple workspaces per environment things start to get way more complex High-level example architecture Extract Workspace • Ingests data from source systems • Central raw data landing zone • Supplies data to domain workspaces Microsoft Fabric Core Data Platform Common Workspace ws_common_ Sales Workspace ws_sales_ Extract Workspace ws_extract_ Source Systems Finance Workspace ws_finance_ MDM Workspace ws_mdm_ Domain Workspaces • Domain-specific transformations and modeling • Consume data from Extract • Deliver curated data to business Common Workspace • Central orchestration layer • Coordinates cross-workspace pipelines • Controls execution across domains MDM Workspace • Produces master data products • Owned by a separate team with their own CI/CD practices & setups • External dependency to the core data platform Detailed example architecture Microsoft Fabric Core Data Platform Common Workspace ws_extract_ Extract Workspace ws_extract_ Deployment Pipelines Lakehouses • Main datastore used in the platform • Lakehouse Shortcuts are used to share data across workspace Pipelines • Main orchestration tool • Use for many data extractions as well Domain Workspace ws_ Notebooks • Data transformation and data modelling Power BI Reports • Data is mainly used for Power BI reporting via semantic models Source Systems MDM Workspace ws_mdm Azure DevOps CI/CD • Repos in Azure DevOps • Deployment done via Fabric Deployment Pipelines Challenges with deployments Microsoft Fabric Core Data Platform Multiple cross-workspace dependencies • Shortcuts and pipeline invokes • Central orchestration layer • Multiple deployment pipelines needed Common Workspace ws_common_ Sales Workspace ws_sales_ Extract Workspace ws_extract_ Source Systems Finance Workspace ws_finance_ MDM Workspace ws_mdm_ Deployment order matters • Upstream first, then consumers • Partial deployment risks • Fails if dependencies missing Dependencies to MDM workspaces • Outside of core data platform CI/CD processes & practices Fabric uses internal ids rather than names • Different IDs per item per environment • Many things can’t be resolved using names Autobinding Microsoft Fabric DEV TEST PROD • Autobinding takes care of shortcuts across workspaces when deploying • Fabric knows that there is a relationship between these workspace items • If the relationship is not there => deployments brakes DEV TEST PROD When not to use autobinding Microsoft Fabric DEV TEST PROD You might have a case where you use shortcut and pipelines to production in every environment • In this case you want all of your shortcuts and pipelines to point to production workspace • DEV TEST PROD Variable Library Multiple variable libraries Generic environment variables vl_environment_variables Variable library for generic environment variables External connections vl_external_connections Variable library for managing external connections and dependencies to external systems Workspace references vl_<workspace_without_env> Variable library for (some) cross-workspace item references. One variable library per referenced workspace Workspace dependency handling Microsoft Fabric MDM DEV ws_mdm_dev Finance DEV ws_finance_dev MDM TEST ws_mdm_test Finance TEST ws_finance_test MDM PROD ws_mdm_prod Finance PROD ws_finance_prod • Variable libraries can be used to handle cross-workspace dependencies • Especially helpful when having dependencies that you don’t control and deploy • Can be also handy in feature branch workflows vl_ws_mdm Variable Libraries or Autobinding Autobinding is simple • Rely on autobinding in simple scenarios and when you are in control of the both workspaces and using Fabric deployment pipelines No need to manage overhead • Keeps the setup simpler since managing a lot of variables in variable library can become annoying For more complex scenarios • Use variable libraries for external connections Cross-environment references • Consider using variable libraries for shortcuts if you have multiple workspaces per environment and there is a need to do cross-environment references Running deployments from Azure DevOps Deployment steps Azure DevOps
Developer runs the deployment pipeline in Azure DevOps Pipeline
Deployment pipeline acquires Microsoft Fabric access token using service connections with SPN identity 2 Get Microsoft Fabric Access Token Developer Run Fabric Deployment Pipeline
Deployment runs Fabric deployment triggers the Fabric deployment pipeline via Fabric API using the token
Deployment pipeline deployment does the deployment from one environment to another Microsoft Fabric Fabric Deployment Pipeline Development Stage Extract DEV ws_extract_dev Test Stage Extract TEST ws_extract_test Production Stage Extract PROD ws_extract_prod
DevOps deployment pipeline polls the Fabric API for deployment status Why run deployments from Azure DevOps? • Avoid personal user ownership in downstream environments • Ensure consistent release identity (Service Principal) Deploying multiple workspaces Single release pipeline • One DevOps pipeline triggers all Fabric deployments • Centralized promotion process Controlled order • Extract → Domains → Common • Upstream artifacts exist before downstream deploy Fail-safe execution • Each step waits for success • Prevents broken bindings and partial environments Microsoft Fabric Azure DevOps Pipeline Run Extract Deployment Pipeline Extract Deployment Pipeline Run Finance Deployment Pipeline Finance Deployment Pipeline Run Common Deployment Pipeline Common Deployment Pipeline DEMO DEMO TIME! Limitations & considerations Key limitations Other things to consider • Deployments are not selective • Dependencies • Which can make hot fixes problematic during testing cycles • Allow deployment via Fabric UI in some cases • SPN as supported item owner • Nowadays supported in most cases but some items might still have problems with SPN as the item owner • Try to have dependencies only to one direction • Otherwise, deployment can become very challenging • Nontechnical aspects • Like deployment cycles • Communication • Pull Requests reviews • Why not use Fabric CI/CD Python library? • Only until recently it became supported by Microsoft, and this setup was had been done before that and that point decision was made to use the “official” deployment methods Questions? Connect with us! linkedin.com/in/attesukari https://linktr.ee/aleksipartanen Sound off. The mic is all yours. Influence the product roadmap. Join the Fabric User Panel Join the SQL User Panel Share your feedback directly with our Fabric product group and researchers. Influence our SQL roadmap and ensure it meets your real-life needs https://aka.ms/JoinFabricUserPanel https://aka.ms/JoinSQLUserPanel