As a Databricks Gold Elite SI and ISV Partner, Blueprint must ensure that customers get the most value out of their Databricks investment. Blueprint’s Lakehouse Optimizer (LHO) is built to operate within the Databricks ecosystem and follow the Partner Well Architected Framework for measuring and attributing customer consumption. That gives us a trusted foundation for understanding which usage belongs to which customer. But attribution is only the starting point. Because LHO pricing is tied to Databricks consumption, we also have to determine how that usage translates into what a customer actually pays. That means being precise about what consumption we measure, how we account for changes in a customer’s environment, and exactly what a number like “cost” represents before it ever becomes part of an invoice.
The band I’m in records most of our rehearsals. Ask how loud the guitar was during any given practice, and you can get three different answers: the level coming from the amp, the level captured by the mixing board, and the level heard in the room. All three numbers can be accurate, but they are measuring the same guitar at different points in the system.
A Databricks environment works the same way. The same workload can show up differently depending on whether you are looking at compute usage, system-reported consumption, or the usage ultimately attributed to a customer or business unit. Every number can be correct. The problem starts when you treat them as interchangeable without knowing exactly what each one measured.
As a Databricks Gold Elite Partner and Connected ISV, we follow the Databricks Partner Well Architected Framework to attribute LHO usage to the right customer. That gives us a clear and trusted starting point. But because LHO pricing is based on customer Databricks consumption, attribution is only the beginning. We still have to determine exactly what consumption is included, what it costs, when a billing period is complete, and how every resulting charge can be verified.
That is the problem we built the LHO billing pipeline to solve.
But attribution is not billing. Knowing who generated the consumption does not automatically tell you what that consumption should cost or what belongs on the customer’s invoice. Just like a signal moving through a sound system, the number only means something when you know where and how it was measured.
Because LHO pricing is tied to Databricks consumption, getting that distinction right is our responsibility. We had to define what we measure, what we mean by cost, how consumption translates into LHO pricing, and how every number can be traced back to the underlying data.
That space between attributing consumption and producing an accurate invoice is where the LHO billing pipeline lives.
The short version
What sits between attribution and an invoice
Attribution answers a question Databricks needs answered: which partner drove this workload. Billing answers a different one, and that one is entirely yours.
Say you’ve got a month of attributed consumption. You still have to decide what that number actually covers, when a period is complete enough to bill, what happens to that number after you’ve invoiced from it, how a correction gets recorded, and how anyone downstream would know if the whole thing had quietly gone wrong. None of that is in the framework. But all of it matters the first time a customer questions a line item.
What we bill for
Blueprint Lakehouse Optimizer (LHO) monitors how a customer consumes Databricks inside their own environment. It deploys per environment, so one customer can run several: a US deployment and an EU one, or a dozen across business units. Each deployment writes usage reports to a Unity Catalog external volume. They’re tabular files, a row per period and a column per metric. We bill a contractual percentage of the consumption those reports carry.
That last clause is where the work is. LHO observes the spend directly, which is one of its value propositions: the meter sits inside the customer’s environment, next to the workloads that generate the cost. But billing doesn’t read that observation live. It reads a report LHO exports, one per deployment, on a schedule. Everything that follows lives in that handoff.
The rest works about how you’d expect. A month closes once it’s complete, and we lock one consumption number per customer for it. Operations invoices the contractual percentage; sales reads that same number to see how an account is trending into a renewal. The close ends with a human review against the previous billed amount.
Why the number is hard to compute
If you price on consumption, you probably inherit some version of this list. Ours has all five.
The billed unit isn’t the reporting unit. You invoice a customer. The product reports a deployment. So every invoice is a rollup, and every rollup encodes an assumption about identity that nothing in the data checks for you.
A cost column doesn’t tell you what it covers. It can be scoped to the deployment that wrote the file, or to the whole account that deployment belongs to. Both are defensible product decisions. Only one of them survives being summed.
Identity drifts upstream and nothing announces it. Environments get renamed. Tokens get retired. A deployment with no name of its own reports under a fallback identifier. And a license roster and a billing book can disagree in both directions at once.
Complete and current pull against each other. A source that’s complete is late. A source that’s current is approximate. Actuals and forecasts can show up in the same file, in the same shape, separated by a column most people never look at.
The report is a moving target, even when you own it. A reporting schema under active development changes shape, which is entirely reasonable behavior for the product and completely fatal to anything that maps columns by position. We build LHO, and this still caught us.
None of this is exotic. It’s what it’s like to bill from an export rather than from the observation itself, and that holds whether or not you wrote the thing producing the export. We did write ours, and we still needed every control below.
If you don’t have a meter in the customer’s environment
Most Connected ISVs don’t, and it’s worth being straight about that, because it changes where your number comes from without changing much about what you then have to do with it.
Without something watching spend inside the customer’s environment, your source is the customer’s own system tables, reached through whatever access they grant you. system.billing.usage carries a record per workload execution, with workspace_id, sku_name, custom_tags, usage_metadata and billing_origin_product on the row. That’s a real source, and plenty of partners bill from it.
It’s also the same problem one layer down:
Scope. workspace_id sits on the row, so the account-versus-deployment question doesn’t arrive stamped into a column the way ours does. It arrives as a choice about how you group. Getting it wrong costs you the same way.
Complete or current. Records are typically available within 12 hours, and longer for new workspaces. A month isn’t final the moment the calendar says it is.
Corrections. The availability guarantee covers ORIGINAL records. Retractions and restatements arrive later, which means the number you read in week one isn’t necessarily the number that stands. That is the whole argument for locking a billed figure instead of recomputing it.
Attribution. Separating your product’s usage from everything else sharing a SKU is what the billing_origin_product and product_features fields exist for, alongside custom_tags. It’s the same question the User-Agent requirement answers from the other end.
So the decisions below apply either way. We read a report LHO exports; you may read system tables. Both are a picture of an observation somebody else made earlier, and both drift.
Decision one: what does that number actually cover?
Start here. Getting this one wrong is the mistake that quietly multiplies an invoice instead of breaking it, and a multiplied invoice reaches your customer looking completely reasonable.
The obvious reading of a cost column is that it’s scoped to the deployment that wrote it, which makes the customer figure a sum across that customer’s deployments. Data people call that the column’s grain. Nothing in the data tells you which one you’ve got. Where the figure is scoped to the account instead, every deployment report carries the same number, and summing multiplies the invoice by the deployment count.
The tell is sitting in the data if you go looking: identical cost across deployments whose usage columns are nowhere near each other. Two deployments running genuinely different workloads don’t produce costs that agree to the cent month after month. Cost that agrees while usage doesn’t means you’re looking at one figure written twice, and you can check it without asking anyone upstream what the column is scoped to.
Now the fix, which is the part worth taking. A declaration on its own would be useless, because nobody notices when the scope changes. So the seed declares the scope per customer, effective-dated, since what a figure covers is a property of the feed at a point in time and not a fixed property of the customer. The pipeline works out the scope independently from each month’s own figures. Any customer whose detected scope contradicts its declaration gets refused instead of billed, and the ledger records the rollup method on every row, so each number says how it was computed.
Two things make that guard worth the effort. It fails loudly. And it keeps working when the answer changes underneath it.
Decision two: complete or current
We read two sources and neither one satisfies billing on its own, which forces a choice you’ll probably meet in some form.
| Monthly report | Weekly report | |
|---|---|---|
| Complete at close | Yes, and it distinguishes actuals from forecast | Approximate, within about a percent |
| Available | Only after the month ends | Within the week |
| Role we gave it | Billing source | Cross-check, never writes a billing number |
The monthly report matched the figure we’d been billing from to the cent on every closed month, so that’s what we bill from. And because a reporting schema under development moves, we map columns by header name rather than by position, which is the difference between absorbing a change and silently misreading one.
Rather than keep having the same argument about which feed to bill from, we split the roles for good. One source gets to write a billing number. The other gets to disagree with it, which turns out to be the more useful job.
Decision three: lock the number, don’t recompute it
Once you’ve invoiced from a number, re-deriving it later lets the answer change without anyone deciding it should. A locked figure is a fact, not a query result.
So the ledger is append-only. When we lock a number, the row records the source filenames and their sha256 hashes, which makes a billed figure explainable down to the bytes it came from. Corrections append instead of updating: a corrected number is a new row pointing at the row it supersedes, and the original stays exactly as written. Reads go through a view that resolves those chains, so billing never triggers a live recompute.
It’s the least interesting decision in the build. It’s also the one most likely to matter in a customer conversation eighteen months from now.
Making the quiet conditions loud
A billing pipeline’s real job is to tell you the truth about whether it ran. The transforms here are an unpivot and a rollup, so we spent the design budget on alarms instead.
| Condition | What the pipeline does |
|---|---|
| No files matched | Fails before it builds the graph, so a zero-file run can never report success |
| The column set changed | Auto Loader captures the unexpected fields in _rescued_data and a pipeline expectation flags them |
| The month isn’t finished | Stays forecast, judged against the report’s own run timestamp, so it can’t reach a locked number |
| A file won’t tie to a customer | Lands in a holding table we call quarantine, so every row in the model has an owner |
| Detected scope contradicts the declaration | Refuses to bill that customer for that month |
Feed freshness is a queryable fact computed from file modification times alone, so the signal never depends on the thing it’s watching.
Alerting carries its own conflict, though. A check that always alerts is a check everyone learns to ignore, and an ignored check is how a quiet condition stays quiet. So the health check fails on three things only: schema drift, a forecast reaching somewhere it shouldn’t, and a stale billing feed for an active customer. Everything else it reports for triage.
How do you prove a billing system is right?
You run it as a controlled experiment. Both pipelines compute the same customer’s month at the same time, everything held constant except one variable, and you publish the result at both settings. We could only do that because the system we were replacing was still running. You can’t prove parity against a pipeline you’ve already torn down.
Our control was April 2026, the last fully closed month where both sources carried a complete picture. Under those conditions the two should be indistinguishable, and they were. Every customer running a single deployment reconciled to the cent.
Our variable was source completeness at close. What if the live feed only carried part of a month while the monthly report still resolved the whole calendar? May 2026 handed us exactly that for one operator. We moved one input and left everything else alone.
| Condition | Prior path | New pipeline | Result |
|---|---|---|---|
| Control: April 2026, both sources complete | 100.00 | 100.00 | Match to the cent, 0.0000% |
| Variable moved: May 2026, live feed partial at close | 29.05 | 100.00 | 70.9% of the month missing from the live feed |
Indexed to the new pipeline’s figure for each month.
Because the control proves the two paths agree to the cent, that second row tells you exactly one thing. The gap isn’t arithmetic and it isn’t a judgment call. It’s what it costs to read a source that’s still moving at close.
Then we replicated it a second way. The weekly report rebuilds the same month by another route, spreading each week’s cost across the days it covers, and lands within 0.8 percent on every closed month of 2026. A second method that only ever agreed with the first wouldn’t be worth running.
What parity doesn’t prove
This one took us longer to get to than any of the engineering above, and it’s the part I’d most want you to take away.
A parity proof can’t catch a scope error on a single-deployment customer. Sum one row, take one copy: same operation, same answer. So the customers that reconcile to the cent aren’t telling you anything about what your cost column covers. They can’t. The proof tests your arithmetic, not what the source actually means.
Which makes agreement a weaker result than it feels like at the time. It settles the months where both systems read the source the same way, and says nothing about the months where they don’t. That’s why we kept testing after parity passed, using questions the build team didn’t write. That practice is written up separately in The what-if drills we run on our billing pipeline.
Answering consumption questions without an engineer
The other half of the problem is that a consumption number generates questions. Customer-facing teams ask them, every one becomes an errand for somebody in engineering, and those errands add days to invoicing and push revenue recognition out with them.
So instead of another dashboard, we put a Genie agent over the governed model. Operations asks what we locked for a customer last month. Sales asks how an account is trending before a renewal. Both resolve against the same governed model, so neither team has to work out who to go ask.
Billing answers can’t be allowed to drift. A question about a locked number routes through the ledger view by way of a governed SQL function, so the agent returns the number we locked rather than summing facts into something close enough. Month-over-month change is a Unity Catalog function, not something the agent improvises.
Observability got easier too. “Which feeds are stale?” is a question now, not a widget somebody maintains. And the metadata does real work here: every table and column carries a comment we wrote when we designed the schema, ahead of any pipeline code, and those Unity Catalog comments are what the agent reads at query time.
We imported every scorer rather than reimplementing it locally where it could drift. The Databricks Genie Workbench IQ Scanner scored the agent Trusted at 12 of 12, and an in-tenant harness using a large language model (LLM) as the judge scored 100 percent on the benchmark question bank.
Then reading the agent’s sample questions caught something no structural check would. One of them, “what did we bill this customer,” returned the customer’s Databricks consumption. Which implies a Blueprint invoice, when the invoice is a percentage of that number. So we took the word “billed” out of the agent entirely. No query changed. If you price on consumption, that distinction isn’t nitpicking; it’s the difference between a number your customer recognizes and one they dispute.
What’s next
Three things on the list. Two are about making the system admit what it doesn’t know, and the third is about never calling the drill list finished.
Teach the agent to say “we don’t have that.” Some questions have no answer in our data, and a confident wrong answer is worse than a refusal. Not-available needs to be a first-class response rather than something a model improvises around.
Give every billed figure a second opinion. The weekly cross-check rebuilds a month by an independent route, and we want that available for every customer and running at every close, rather than on request.
Keep adding drills. The scenario list isn’t finished, and it isn’t supposed to be. The next two we’re writing are what happens when a contract changes mid-period, and what the model should do when two customers merge and their histories have to combine without double-counting the overlap.
What transferred
The harder check is whether your data model can answer consumption questions in natural language without inventing numbers, because that’s where a customer-facing team is going to meet it. Blueprint runs a no-cost, two-week Databricks Genie Readiness Assessment that scores your Unity Catalog metadata and relationships, then hands back a prioritized remediation roadmap. It’s the same assessment that graded this build, run out of Blueprint’s Databricks Center of Excellence.
Appendix
Source documentation
Databricks Partner Well-Architected Framework
The reference architectures and integration requirements Databricks publishes for its technology partners; the framework this post argues stops one step short of billing.
Telemetry and attribution for Connected ISV partners
The User-Agent requirement and its validation path, which is where attribution ends and the billing decisions in this post begin.
system.billing.usage, its attribution columns, and the 12-hour availability guarantee that covers ORIGINAL records but not corrections; the alternative source for partners without a meter in the customer’s environment.
The declarative framework the ingestion layer is built on.
Manage data quality with pipeline expectations
The expectation syntax behind the schema-drift and forecast alarms in the alarms table.
Incremental file processing, and the mechanism behind the zero-file and column-set alarms.
Auto Loader schema inference and evolution
The _rescued_data column that catches fields a moving reporting schema adds.
Governed storage for the non-tabular report files each deployment writes.
Genie One for business users and the Genie agents data teams configure to power its answers; the interface operations and sales use against the governed model.
Declarative Automation Bundles
The deployment unit that makes every resource in the pipeline reproducible from source.
We deliver intelligence that matters.
Gary Nakanelua is Managing Director of Innovation & Product at Blueprint and a Databricks MVP. He’s the author of Experiment or Expire, and publishes what he learns working with customers as a running series of Databricks experiments, each one a notebook you can run yourself on Free Edition.
