Fabric Sprawl - Techniques to Optimize Your Fabric Capacity

Description

Learn how to address a sprawling Fabric environment with ungoverned items across multiple workspaces and capacities. Extend FUAM to gain valuable insights that drive decision-making and backlog prioritization. Design the right balance between autonomy and security in your Fabric environment to deliver an optimal experience for both end users and developers.

Key Takeaways

• FUAM & its extensions
• Approach and strategies
• An open-source accelerator in
• Data collection of various Fabric
• Stores in Fabric Lakehouse.
• Easily installed and extendable.
• Add Personal Workspace Data

My Notes

Action Items

• [ ]

Resources & Links

Slides

Fabric Sprawl
Curtis Stallings
Senior Architect, Baker Tilly US
GitHub - Curts0
Marcus Radue
Senior Architect, Baker Tilly US
https://mrdata.pro
All code shown today
is available on GitHub!
Session Topics
•What is Fabric Sprawl?
•What are the symptoms?
•What are the benefits of
controlling Fabric Sprawl?
•How do you solve for Fabric
Sprawl?
Know Remove Improve
• Know
• FUAM & its extensions
• Remove
• Approach and strategies
• Improve
•
•
•
•
Artifact consolidation
Security cleanup
CICD standardization
Monitor
What is Fabric Sprawl?
The uncontrolled growth of Fabric artifacts.
What does Fabric Sprawl look like in your Org?
Some look
like this…
Or this…
Or this…
Or this.
Sprawl Symptoms
Duplication of data assets
Accuracy issues
Capacity peaking
It could be worse! Better to have
sprawl inside of Fabric vs. outside…
Security violations
Redundant build efforts
Benefits of Sprawl Removal
Reduce Fabric Spend
Repurpose Fabric
Compute
Resource
Optimization
Streamline Data
Governance
Fabric Sprawl Resolution
Know
Remove
Improve
Know
FUAM - Fabric Unified Admin Monitoring
• An open-source accelerator in
Microsoft's GitHub repository.
• Completely Fabric.
• Data collection of various Fabric
APIs.
• Stores in Fabric Lakehouse.
• Easily installed and extendable.
Different Ways to Extend FUAM
• Add Personal Workspace Data
• Include Semantic Model Metadata
• Perform Lineage Analysis on Fabric
Items
See FUAM documentation on
updating.
Modifications to Existing FUAM
Pipelines & Tenant Setting Changes
Semantic Model
Metadata Example:

• Read FUAM files from
  API collection into
  Dataframes
• Write as delta tables
• Build direct lake
  semantic model
• View details of all
  semantic models
  across your entire
  tenant.
  Lineage Analysis
• For each fabric item
  type locate the parent
  and child connections
  in FUAM data.
• Conform data into
  matching Dataframe
  schema.
• Let Networkx Python
  package do the hard
  work and find all the
  possible paths and
  connections.
• Build semantic model
  to look at data.
• Perform analysis.
  Remove
  Why is Remove second?
  Removal Strategies

1. Objects identified and added to the
   backlog
   • Backlog prioritized
2. Users notified (Email, Teams, Office
   Hours)
3. Access removed
   • Monitor feedback
4. Archive objects
   • Semantic Link Labs script
5. Delete objects
   Semantic Link Labs
   combined with data
   collected by FUAM allows
   for easy automated
   archiving.
   Improve
   Fabric Sprawl Prevention
   Artifact
   Consolidation
   Security
   Cleanup
   CI/CD
   Implementation
   Monitor
   Artifact Consolidation
   Collaborate with Data Governance
   Utilize Fabric Tags for archival process
   Promote and Certify content through Fabric Endorsements
   Domain Based Governance
   • Identify data governance roles
   • Executive Sponsor, Domain Owners, Data Owners,
   Data Stewards
   • Select a data governance platform
   • Microsoft Purview
   • Scan, collect, curate, and monitor data assets
   Endorsements
   • Leverage Fabric endorsement labels to make
   data trust visible and actionable
   • Business domain self-service data assets are
   promoted
   • IT created data assets are certified
   • “Golden” record data assets are labeled master
   data
   • Align endorsement with data governance
   processes
   Security Group Cleanup
   Fabric Permission Levels
   Workspace Permissions
   • Evaluate Fabric permissions for coarse-grain
   workspace access
   •
   •
   •
   •
   • Evaluate item permissions for coarse-grain item
   access
   • Evaluate OneLake RBAC permissions for finegrain folder access
   Admin
   Contributor
   Member
   Viewer
   Item Permissions
   (Permission options vary per item)
   •
   •
   •
   •
   •
   •
   • Consider trusted workspace access and private
   endpoints for increased network security
   Semantic model
   Warehouse
   Data Factory
   Lakehouse
   Data science
   Real-Time Intelligence
   OneLake Permissions
   (Data Access Roles)
   • RBAC model
   Fabric as a Product
   Tighter Controls
   Push down content
   Reporting platforms
   Certified
   Push up content
   VS Code + Git Integration
   Department
   Knowledge Management
   Information sharing
   Promoted
   Team
   Business developed and maintained
   Enterprise
   More creators, fewer consumers
   Permanent record
   Subject to compliance
   Change Management
   Fewer creators, more consumers
   IT – developed and maintained
   Fabric Platform
   Individual
   Personal / private views
   Data science exploration
   Looser Controls
   CICD Implementation
   Enterprise IT
   Citizen Developer
   Semantic Model CICD Example
   Audit & Monitor
   • Fabric Unified Admin Monitoring (FUAM)
   • Easy install
   • Extend functionality
   • Analyze and create actionable insights
   • Fabric Workspace Monitoring
   • High-cost entry point (Fabric CUs)
   • Low development time
   • Capture high granularity logging events
   • Log Analytics
   • Low-cost entry point
   • Higher development time
   • Capture high granularity logging events
   Azure Log Analytics
   “How can I keep track of the overall performance of all these visuals when so many users are building out their own content?”
   Azure Log Analytics
   “I want to remove this column, is it being used?”
   Azure Log Analytics
   “Did I refresh a blank partition? How long are my partitions taking to refresh?”
   Azure Log Analytics
   “My processing is slow; how does my encoding look?”
   Azure Log Analytics
   “I got 99 problems, and a dupe key error is definitely one. Can I see the history of my processing errors?”
   Resources
   • FUAM: fabric-toolbox/monitoring/fabric-unified-admin-monitoring at main · microsoft/fabric-toolbox
   • Code resources: GitHub - Curts0
   Sound off.
   The mic is all yours.
   Influence the product roadmap.
   Join the Fabric User Panel
   Join the SQL User Panel
   Share your feedback directly with our
   Fabric product group and researchers.
   Influence our SQL roadmap and ensure
   it meets your real-life needs
   https://aka.ms/JoinFabricUserPanel
   https://aka.ms/JoinSQLUserPanel