Own Your Workflow: Build Notebooks Locally & Deploy Them to Fabric

Description

Break free from browser IDE in Fabric Python and PySpark Notebooks! Learn to install Python and Spark, create your own local lakehouse, and develop Python and PySpark Notebooks in Visual Studio, VSCode, or whatever IDE you're used to. We'll start with a clean machine and download, install and configure everything we need. We will create a feature-rich notebook, and wrap up by deploying to Fabric.

Key Takeaways

• •Use Fabric for scale, governance, scheduling
• Common pitfalls & how to avoid them
• Spark/Delta version mismatch
• Verify the compatibility matrix in Delta docs
• Start Spark with Delta extensions enabled
• If using manual Spark binaries, set JAVA_HOME, SPARK_HOME, and (sometimes) HADOOP_HOME for winutils
• Prefer pip install pyspark for simplicity

My Notes

Action Items

• [ ]

Resources & Links

Slides

Own Your Workflow: Build
Notebooks Locally & Deploy
Them to Fabric
• Andy Parkerson
Data Engineer
MMR Group
andyparkerson@gmail.com
linkedin.com/in/andyparkerson
Microsoft Fabric
•Power BI
•Jupyter Notebooks
•Pipelines
•OneLake storage
•Delta Lake
•Warehouse
•Eventhouse
Fabric Notebooks
•Connect to Data
•Code
•Python
•PySpark
•Spark SQL
•Markdown
•Visualizations
Why Jupyter
Notebooks?
•Julia
•Python
•R
Ju – Pyt – eR
Why local-first?
•Faster edit–run loop
•Familiar tooling
•Lighter capacity usage
•Use Fabric for scale, governance, scheduling
What we’ll build
•Local VS Code + PySpark + Delta
•A small medallion-style ETL notebook
•Deploy to Fabric and run it there
What we’ll need
•Python 3.12+
•Java 17
•PySpark 3.5.1
•delta-spark 3.1.0
•VS Code + Fabric extension (GA)
•Access to a Fabric workspace + capacity
•OneLake File Explorer for easy file moves on Windows
Package Installation

winget install --id=Python.Python.3.12 -e
$env:PYSPARK_HADOOP_VERSION="3"
pip install “pyspark3.5.1”
pip install “delta-spark3.1.0”
Package Installation - Spark
•https://spark.apache.org/downloads.html
•Spark release 3.5.8
•Download Pre-built for Apache Hadoop 3.3 and later
•Unzip into “C:\Users\your_name\spark\spark_3.5”
Package Installation - Hadoop
•https://github.com/robguilarr/spark-winutils-3.3.1
•Hadoop 3.4.0
•Clone repo or download
•Select files in hadoop-3.4.0-win10-x64/bin directory
•Copy into “C:\Users\your_name\spark\Hadoop\bin”
•Copy hadoop.dll into “C:\Windows\system32” folder (?)
Package Installation - Environment
setx JAVA_HOME “CK\Program Files\Exclipse Adoptium\jpk-17”
setx SPARK_HOME “C:\Users\your_name\spark\spark-3.5.8-bin-hadoop3\”
setx HADOOP_HOME “C:\Users\your_name\spark\Hadoop\”
Package Installation - VSCode
•https://code.visualstudio.com/download
•https://marketplace.visualstudio.com/items?itemName=f
abric.vscode-fabric
•Microsoft Fabric Extension
•https://marketplace.visualstudio.com/items?itemName=S
ynapseVSCode.vscode-synapse-remote
•Fabric Data Engineering Extension
•https://marketplace.visualstudio.com/items?itemName=
ms-toolsai.jupyter
•Jupyter notebook Extension
Demo
Demo: Create a local “lakehouse” with Delta
Demo continued 2/3
Demo
continued
3/3
Develop in VS Code (Fabric Extension)
•Explore Fabric workspaces
•Clone Git-enabled workspaces
•Edit item definitions
•Manage items
•Extension is Gorgeously Awesome (GA).
Demo 2
Deploy to Fabric — three paths
•A. Git integration (Azure DevOps & GitHub supported)
•Commit locally → sync in Fabric
•Notebooks stored as source files (e.g., notebook-content.py)
•B. VS Code extension
•Create/update items (including notebooks) directly via item
definitions
•C. REST API
•Create/update notebooks
•Run on demand (service principal supported)
CI/CD options
•Git Integration (Azure DevOps, GitHub) + Deployment
Pipelines
•Treat Git as the source of truth
•Promote Dev → Test → Prod.
•Community accelerator (fabric-toolbox) for Git-based
deployments and automation patterns.
•REST APIs or VS Code extension item definitions for
scripted deployments
Common pitfalls & how to avoid them
• Spark/Delta version mismatch
• Verify the compatibility matrix in Delta docs
• Start Spark with Delta extensions enabled
• Windows specifics
• If using manual Spark binaries, set JAVA_HOME, SPARK_HOME, and (sometimes) HADOOP_HOME for winutils
• Prefer pip install pyspark for simplicity
• Notebook type selection
• Python vs PySpark
• Use Python for in-memory/smaller data (starter pool ~seconds)
• Use PySpark for large/parallel workloads
• OneLake File Explorer is preview
• Be aware of sync behavior, naming limitations, and update prompts; latest installer on Microsoft Download Center
• Environment management
• Put libraries in a Fabric Environment and attach to the notebook
• version in Git
• Publish after update.
Own Your Workflow: Build
Notebooks Locally & Deploy
Them to Fabric
• Andy Parkerson
Data Engineer
MMR Group
andyparkerson@gmail.com
linkedin.com/in/andyparkerson
Sound off.
The mic is all yours.
Influence the product roadmap.
Join the Fabric User Panel
Join the SQL User Panel
Share your feedback directly with our
Fabric product group and researchers.
Influence our SQL roadmap and ensure
it meets your real-life needs
https://aka.ms/JoinFabricUserPanel
https://aka.ms/JoinSQLUserPanel
How was
the session?
Complete Session Surveys in
for your chance to WIN
PRIZES!