Fabric Development: Repos, Collaboration and Deployments

Description

Are you a Fabric developer and struggling with using good development practices?
Implementing practices such as source code control, branching strategies, code reviews (PRs), and automated deployments to independent environments (ie. workspaces) can transform how you work with Fabric.
We'll show how to implement them and work around the gotchas that will bite you in your Fabric projects.

Key Takeaways

My Notes

Action Items

Slides

Fabric Development
Repos, Collaboration and Deployments
/mikediehlsqlbi
Mike Diehl
Mike.Diehl@Improving.com
Director of Data Engineering and Business Intelligence
https://github.com/xhead/
CANADA
VANCOUVER
CALGARY
WINNIPEG
OTTAWA
TORONTO
UNITED STATES
MINNEAPOLIS
OMAHA
CHICAGO (2)
COLUMBUS
CLEVELAND
ATLANTA
DALLAS
AUSTIN
HOUSTON
MEXICO
GUADALAJARA
AGUASCALIENTES
CENTRAL AMERICA
GUATEMALA
SOUTH AMERICA
SANTIAGO
BUENOS AIRES
INDIA
PUNE
Our Goals for Source Control in Fabric
• Allow developers to:
• work on the same project at the same time
• experience fewer pull request conflicts
• Maintain the ability to:
• Keep every workspace item in a repo
• Do code reviews during a pull request
• Have CI builds and releases (source code validation)
• Rollback to previous releases
• Always know what each environment contains
• History of changes:
• For the project
• For Releases
Our Rules for Source Control in Fabric
• The primary DEV workspace is always and only associated to the main branch
• Always work in a feature branch and workspace
• Pull requests approved by one other team member
• DEV workspace must be in a valid state
• Only reference valid workspace items
• Pull requests must leave references intact
• Pull requests and deployments
• validated by a CI build
• Initial deployment to a CI environment
Source Control & Workspaces
• Create a new repository in Azure DevOps
• Consider adding VisualStudio .gitignore if
you will use it to manage branch updates
and conflict resolution
Source Control
& Workspaces
• Open Workspace settings and
connect to Azure DevOps
• Can use your EntraID or a preconfigured connection
• Pre-configured connection is in
preview.
Source Control & Workspaces
• After connecting to Azure DevOps open the Source control
pane
• Open the Git account details
• Select your Git repository and branch
• We avoid changing our primary DEV workspace to any
other branch
Branching, Pull Requests & Code Reviews
• Open the source control pane
• Under current branch, select the Branch out to workspace
option
• Checkout new branch:
• Will create a new repo branch and update the current workspace
• Switch branch:
• Swaps the workspace to an existing branch
• All items are refreshed, uncommitted changes may be lost
Branching, Pull Requests & Code Reviews
• From the Fabric UI you can create a brand new branch
in the repo
• Will be based on the currently selected workspace
branch
• Create a new workspace, or
• Good for new developers or if you want to start from scratch
• Re-use an existing workspace
• Handy to retain existing security setup / existing data
Branching, Pull Requests & Code Reviews
• The workspace actively tracks changes
• Source control pane offers options to commit or undo individual
items
• Commit if you want to create a history step in your branch / repo
• Otherwise not required until you are ready
• Note: Lakehouse tables and files are not included in your changes
Branching, Pull Requests & Code Reviews
• Pull requests must be done through:
• Azure DevOps in the browser
• Another tool such as Visual Studio (Code)
• PRs offer the opportunity to:
• do code reviews
• do CI builds
• Squash commits for cleaner source history
Branching, Pull Requests & Code Reviews
• After a successful pull request, workspaces are left intact
including the branch name
• Alternate method for re-using workspace:
• Create a new branch in Azure DevOps
• Open workspace settings
• Associate to newly created branch
• For Large Repos, Highly Recommended
Branching, Pull Requests & Code Reviews
• Its not all roses and sunshine with branches in Fabric…
• Fabric UI does not support ‘updating your branch’
• Suggestion 1… Azure DevOps w/Browser
•Pull request ‘main’ into your feature branch
•Supports conflict resolution (sort of)
• Suggestion 2 … Visual Studio (Code)
[my preference]
•Visual Studio has great git support
•Visual Studio Code is free
Deployments to Other Environments
• DevOps Release vs Deployment Pipelines
• DevOps releases are:
• Flexible - Can include other artifacts such as SQL Database Projects
• Consistent – only items in repo can be released to the other environments
• CI builds offer extra opportunities:
• Workspace validation (we validate some ID values…)
• Perform unit tests
• Can be executed against any branch (i.e. during PRs)
• DevOps has more granular permissions:
• Approval gates by environment
• Do not need to be a Fabric Administrator
• Rollbacks are easier and more predictable
Deployments to Other Environments
• Fabric CI-CD
https://microsoft.github.io/fabric-cicd/latest/
• Python library that supports most Fabric items
• Flexible authentication options (Default, CLI, AZ PowerShell,
SPN)
• Yaml based deployment configurations (optional)
• Yaml based parameter file for environment-specific values
• Supports multiple deployment scenarios
Deployments to Other Environments
• Microsoft suggested Option 2 - Git- based deployments using Build environments
Deployments to Other Environments
• Git- based deployment using Feature Workspaces and Release Pipelines
Gotchas & Workspace Management
• Renaming objects in Fabric
• Only the Display Name is updated
• Repository files and folders retain original name
• You can safely rename them directly in your branch
• Be careful with Default Lakehouse references
• Branched workspaces will reference original Dev Workspace, Lakehouse
• Can dynamically connect Lakehouse to items
• Notebooks can use the %%configure cell magic command
• Adopt a standard pattern and enforce during pull-requests
Gotchas & Workspace Management
• Lakehouse Tables and Files are not repo items
• Use SQL Endpoint to deploy SQL Database projects
• Lakehouse Files require custom deployment steps
(Fabric Rest API, python DataLakeServiceClient class, etc.)
• Workspaces do not automatically sync with repos
• Opening workspaces offers a chance to update
• Deploy DEV as a release stage
•Release using SPN
• App Registration or Managed Identity
• As individual may cause issues
DISCUSSION
Sound off.
The mic is all yours.
Influence the product roadmap.
Join the Fabric User Panel
Join the SQL User Panel
Share your feedback directly with our
Fabric product group and researchers.
Influence our SQL roadmap and ensure
it meets your real-life needs
https://aka.ms/JoinFabricUserPanel
https://aka.ms/JoinSQLUserPanel