How I Cut Our Microsoft Fabric CU Usage From 70% to 30%

Our F32 capacity was sitting at ~70% utilization and creeping toward throttling. A few months later it runs at ~30% — same data, same business, roughly $50K a year less in infrastructure cost. Nothing we did was exotic. Here are the three changes that actually moved the number, in the order I'd do them again.

1. Stop using Spark for small tables

This is the biggest one, and almost nobody talks about it.

In Fabric, a PySpark notebook doesn't just run your code — it spins up a Spark session with a driver and executors, and that whole cluster bills CUs the entire time it's alive. For a table with 50 million rows, fine. That's what Spark is for. But if you look honestly at your pipelines, most tables are small. Dimension tables. API responses. Config data. Tables with 10,000 rows getting processed by a distributed compute engine designed for billions.

A pure Python notebook in Fabric runs on 2 vCores. A default Spark session uses many times that — for the same tiny merge.

The fix: for small and medium tables, replace PySpark with plain Python using delta-rs (the deltalake package). It reads and writes real Delta tables — same format, same lakehouse, no Spark session at all.

# Pure Python notebook — no Spark session, tiny CU footprint
from deltalake import DeltaTable, write_deltalake
import polars as pl

df = pl.read_parquet("/lakehouse/default/Files/staging/customers.parquet")

dt = DeltaTable("abfss://.../Tables/silver_customers")
(
    dt.merge(
        source=df.to_arrow(),
        predicate="target.customer_id = source.customer_id",
        source_alias="source",
        target_alias="target",
    )
    .when_matched_update_all()
    .when_not_matched_insert_all()
    .execute()
)

The pattern that works well in production: route by size. One decision at the top of the pipeline — if the table is under a threshold (we use row count, but file size works too), it goes to the Python notebook; if it's over, it goes to Spark. You keep Spark for the workloads that deserve it, and everything else gets 10x cheaper overnight.

DuckDB and Polars cover the transformation side in pure Python, and honestly, for anything under a few million rows they're often faster than Spark too, because you're not paying cluster startup time.

2. Kill your full loads

The second-biggest CU burner is reprocessing data that hasn't changed.

A full load re-reads the entire source and rewrites the entire table, every run. That was defensible when the table had 100K rows. Two years later it has 40 million, the pipeline takes 50 minutes, and 99% of that work is copying yesterday's data on top of yesterday's data.

Incremental loading means: only pull rows that changed since the last run. The simplest version is a watermark — store the max timestamp (or ID) you've loaded, and next run, ask the source only for rows past it.

-- What the pipeline asks the source, each run
SELECT *
FROM sales.orders
WHERE modified_date > '2026-07-15 02:00:00'  -- last watermark

Then merge those rows into the target instead of overwriting it. Save the new max timestamp. Done.

Real-world notes from doing this on 20+ sources:

• Not every source has a reliable modified column. Check it actually updates on every change before you trust it. If it doesn't, look for CDC on the source side, or fall back to a hash comparison on a key subset.
• Handle late-arriving data by overlapping the window slightly (e.g., watermark minus 1 hour) and letting the merge deduplicate. Cheap insurance.
• Keep one full-load path you can trigger manually for backfills and recovery. Incremental for every day, full load for the day something breaks.

One warehouse-sized full load converted to incremental can drop that pipeline's CU consumption by 90%+. Multiply across your worst five pipelines and the capacity chart visibly changes.

3. The real recommendation: find out what's actually burning CUs first

Here's the honest one, and it's the step I'd now do before the other two.

When capacity is high, everyone has a theory about why. The theories are usually wrong. Install the Microsoft Fabric Capacity Metrics app and look at the actual top consumers over 14 days. When we did this, the results were not what anyone guessed — a handful of items were responsible for the majority of consumption, and some of them were things nobody was even using anymore.

What you'll typically find at the top:

1. Dataflows Gen2 doing transformations. Dataflows are convenient and expensive. Almost anything a dataflow does, a notebook does for a fraction of the CUs. Migrating your heaviest dataflows to notebooks is usually the fastest single win in Fabric.
2. Overlapping schedules. Six pipelines all firing at 6:00 AM, hammering the capacity into its burndown, when they could be staggered — or when three of them feed a report that's only opened on Mondays. Ask "who consumes this, and how often?" before asking "how do I make it cheaper?" The cheapest pipeline is the one that stops running hourly for a daily report.
3. Semantic model refreshes importing data that's already sitting in the lakehouse. If the data is in OneLake, Direct Lake mode skips the import refresh entirely.
4. Interactive queries from one dashboard someone built with auto-refresh on.

Measure first. Then apply fix #1 and #2 to the things at the top of your list, not the things a blog post (including this one) guesses at.

The short version

If your Fabric capacity is running hot: install Capacity Metrics and find your real top consumers, move small-table workloads from Spark to pure Python with delta-rs, convert your biggest full loads to watermark-based incremental, and migrate heavy Dataflows Gen2 to notebooks. That sequence took us from 70% to 30% on the same capacity — no SKU upgrade, no dropped workloads.

One honest caveat: don't expect to stay at 30–40% forever. If your company is growing, your data is growing — more sources, more rows, more reports — and consumption will climb back up over time no matter how clean your pipelines are. That's normal, and it's not failure. The point of this work isn't to hit a number once; it's to make sure that when you eventually do upgrade your SKU, it's because the business genuinely grew — not because Spark was processing 10,000-row tables and full loads were copying the same data every night.

Murad Nabizade

Data Engineer in Dallas, TX — Microsoft Fabric and Azure pipelines, data quality, and monitoring. I write about production Fabric and build free tools for data & QA engineers.