# The Token Bill Is an Architecture Problem

> At July's costs for our question enrichment pipeline, aligning our question bank to multiple state standards frameworks was projected at tens of thousands of dollars. The full run cost a few hundred, and quality held. None of the savings came from cheaper token prices alone. They came from the same moves that fix a warehouse bill.

By Cody Bakken · 2026-10-15 · https://codybakken.com/writing/token-bill-is-an-architecture-problem/

Yesterday I predicted that the
[token bill is coming](/writing/work-before-the-code/) for every team using AI,
and that CTOs will start asking what the spend bought. This week CIO ran a piece by
Prasanna Arikala,
[Token efficiency: Building AI that spends intelligence wisely](https://www.cio.com/article/4235350/token-efficiency-building-ai-that-spends-intelligence-wisely.html),
which makes a point I agree with: most AI waste comes from the architecture around the
model. His examples: frontier models doing routine classification, context carried
forward long after it stopped mattering, and retry loops with no definition of
success.

We just lived through that, so here's what it looked like with real numbers.

## The project

Our question enrichment pipeline aligns assessment questions to state standards
frameworks: for every question and every framework, which standards does it actually
assess? It's a classification
problem with a lot of judgment in it, and it's a good fit for LLMs. It's also a
multiplication problem: questions times frameworks. Any work you do inside that
product gets paid for once per pair, so the cost of a sloppy step grows with every
framework you add.

In July, the pipeline's costs were climbing fast. When we projected those costs across
the full question bank and all the frameworks we needed, the estimate came out in the
tens of thousands of dollars.

The full run, when we did it in September, cost a few hundred dollars. Quality held
or improved on every change we measured.

## Where the money was going

When we dug in, very little of the July spend was the model doing useful thinking.
Most of it was the pipeline paying for the same work more than once.

- **Re-runs.** One approval launched several jobs over the same work queue at once.
  They all timed out, and we paid a few hundred dollars for zero completed work. On
  another day, about 89% of the spend went to judging the same items again across
  concurrent jobs.
- **The same reasoning in multiple stages.** One stage worked out what each question
  was asking: its depth, its grade band, a summary. A later classifier re-derived the
  same fields on every item. We were paying for that understanding twice.
- **Per-framework rework.** Each question was embedded again for every framework,
  so a question aligned to a dozen frameworks got embedded roughly two dozen times
  when once would do.
- **Expensive defaults.** One capable model for every role, online pricing for work
  that didn't need to be real-time, and unbounded thinking tokens billed at the
  output rate.

A better model price list wouldn't have fixed any of these. They're the same
mistakes that make a warehouse bill balloon: recomputing what you could have stored, running the same job twice, and
paying premium rates for work that could wait.

## What we changed

### Model it once, reuse it everywhere

The biggest structural change was pulling work out of the multiplication. Most of
what the model needs to know about a question has nothing to do with any framework:
what it asks, how deep the thinking goes, what grade band it fits. So we separated
*understanding a question* from *aligning it to a framework*. Understanding runs once per question, and the result is
stored, keyed on the question's content, with no dependence on any framework.

Aligning to a new framework then starts from the stored understanding instead of
re-reading the question. Each question is embedded once and reused. In the September
run, understanding was about half the total cost, paid once, and each additional
framework was a small increment on top of it.

This is the same move I made when we re-architected our BigQuery reporting layer with
Dataform: model the expensive thing once, in one place, and let everything downstream
read from it.

### Never pay for the same work twice

- A claims-based work queue, so an item can't be judged by two jobs at once.
- Idempotent runs with checkpoints, so a failure resumes instead of starting over.
- Every stored result records a fingerprint of its inputs. If a question hasn't
  changed, it makes zero LLM calls. If only a later stage changed, only that stage
  re-runs.
- A cost estimate before a run is approved, and an automatic stop if spend starts
  climbing faster than expected.

### Match the price to the job

- **Batch pricing.** Almost none of this work needed an answer in seconds. Moving it to
  batch jobs cut the price in half.
- **A cheaper model for the routine judging.** The classification and judging steps
  moved to a much cheaper model. We tested it before calling it done: quality was as
  good or arguably better. It rejected a classification slightly more often, and when
  we checked, those rejections were mostly right.
- **A cap on thinking tokens.** Capping how long the model could reason on the
  alignment step cut that step's cost by about 70%. The mappings got more
  complete, too: the number of skills it missed dropped from 20 to 7 in our test.

The expensive reasoning still happens. It just happens once, where judgment is
actually needed.

These levers aren't free. Batch results don't come back identical to online results
for the same prompts, so we tested the move to batch the same way we'd test a model change. Every lever in this list got a quality check before we kept it.

## What I'd do differently

**Build the cost measurement first.** In July we had to reconstruct what we'd spent by
hand from token logs, after the money was gone. Two accounting bugs only showed up
once we ran real jobs: thinking tokens weren't being counted, and batch spend was
being recorded as zero. You can't manage a bill you can only see after the fact.

**Make the cost change and the quality check travel together.** Every lever here
could have quietly made the output worse. We wrote a model-swap procedure so no
change ships on cost alone. Before trusting any A/B result, we also measured how much
the model disagreed with itself on identical runs, because a model that isn't
perfectly repeatable will show you "differences" that are only noise.

## The takeaway

Token prices will keep falling, and that will help. But almost all of the gap between
our July projection and our September bill came from architecture: what we computed
once, what we refused to compute twice, and which jobs we sent to which price tier.

When the token bill comes due, the teams with an answer will be the ones who treated
model calls the way good data teams treat warehouse queries: something you design,
measure, and stop paying for twice.