# The Work That Makes AI Coding Agents Pay Off

> A study of 700+ software companies found AI coding agents produced more code without more finished software. Our move from an embedded BI tool to an in-product open-source stack went the other way: pages under a second, 3–4x the daily page views, and a lower cost to run. The difference was the work done before an agent wrote a line.

By Cody Bakken · 2026-10-14 · https://codybakken.com/writing/work-before-the-code/

A study making the rounds this month put numbers on something a lot of engineering
leaders have felt. As
[Ars Technica reported](https://arstechnica.com/ai/2026/10/ai-coding-agents-generate-more-code-but-not-more-software/),
Harvard researchers Fiona Chen and James Stratton looked at work data from more than 700
software companies and found that after teams adopted AI coding agents, lines of
code went up 30%, commits 20%, and pull requests 23%. The rate at which finished
features shipped didn't change in any statistically significant way.

The authors trace the gap to code review. After agents arrived, the time from pull
request to merge rose 49%, the share of pull requests sent back for changes nearly
doubled, and reviewers left 35% more comments. In the authors' words, the gains in the coding phase
were "absorbed by downstream constraints in the production process."

I don't doubt the review finding. When you can produce code faster than people can
read it, review becomes the constraint. But I read the result as a planning problem
as much as a review problem. When building gets cheap, the expensive mistakes are
building the wrong thing, or building the right thing without a plan and paying for
it in review. Review is where a missing decision finally gets noticed, and it's the
most expensive place to notice it.

My team writes most of its code with AI agents, and we move much faster because of
it. Here's a project where that speed turned into value, and the work that made the
difference.

## The project

Our customer-facing reporting started on an embedded BI tool, and it served our
initial use case well. I'd recommend that path to most teams starting out: a mature
BI tool gets you credible reporting in front of customers quickly, and you learn what
they actually use.

What we learned, over time, was where the embedded setup stopped fitting what
customers needed. So we moved the reporting onto an open-source stack built directly
into the product. Agents wrote most of the code. Without them, the same project would
have needed more people and a lot more time.

The results:

- **Pages load in under a second.** Some dashboards had been loading slower than our
  users needed.
- **Daily page views are roughly 3–4x** what they were before the move.
- **It costs less to run,** with no BI licensing for the embedded reports.

Three pieces of work, all done before the agents started, account for those results.

## 1. The user research already existed

I designed the original dashboards, and that work started with research into who
used them and what decisions they were trying to make. That research didn't expire
when we changed platforms. Most of the design work for the new version was
translating what we already knew into the new stack, so the experience carried over.

That matters more than it sounds. A migration that makes users relearn their reports
spends the goodwill the new features are supposed to earn. Because the new platform
kept what already worked, the switch didn't feel jarring to customers, and they could
put the new features to use right away.

## 2. The backlog chose the framework

We had a backlog of feature requests that were hard to meet with an embedded BI tool.
Two examples:

- **Sorting and filtering large tables.** Customers wanted to work with tables
  directly: sort a column, filter a few values, find the rows that matter. Grid
  libraries like AG Grid are built for exactly that.
- **Sharing a link that keeps its filters.** A user narrows a report to what they
  care about and wants to send that exact view to a colleague. That's hard when the
  reporting sits inside another application and users can't reach the BI server
  directly, so a shared link can't carry the filter context with it.

There were many others, but the pattern was the same. Each request was a job a
customer was trying to get done, and the backlog gave us a concrete list of jobs the
new platform had to do well. We picked the stack against that list. The cost
savings were real, and they came along with a product that did more for customers.

## 3. The architecture came before the agents

Before anyone wrote code, I planned the structure of the codebase and the
application, down to the folder structure, and pointed the agents at that plan.

This is the part I'd stress for any team building with agents. Agents are good at
filling in a structure you've defined. Left to invent one, they produce a lot of code
that doesn't hang together, and someone has to review all of it. That's the study's
bottleneck showing up one step earlier. A clear structure gives every piece of
generated code an obvious place to live and an obvious standard to be reviewed
against, so review can focus on whether the code does its job.

## The old discipline still applies

None of this is new. The data science life cycle starts with business understanding
for a reason: you identify the value and the job to be done before you build
anything. Software teams have their own versions of the same step.

What agents change is the cost of skipping it. When writing code was slow, the build
itself put a brake on bad ideas, because there was only so much you could build in a
quarter. Agents remove that brake. You can now build the wrong thing at full speed,
and the bill shows up as review load, maintenance, and a product nobody asked for.

For an analytics team, that puts the most leverage upstream of the code:

- **Vet requests** against a real use case. A request with no decision or user
  behind it doesn't get built because it's now cheap to build.
- **Identify the jobs to be done** and write them down, so the build has something
  to be checked against.
- **Prioritize** the jobs that move what the business cares about. I wrote about
  keeping that prioritization honest in
  [A Data Team Without a Roadmap Is a Ticket Queue](/writing/data-team-roadmap/).
- **Settle the architecture** before the agents start, so their output has a place
  to go.

We didn't build this platform because we could. We built it because we had a use
case, a list of customer jobs it had to do, and a plan for the code. With those in
place, the speed from the agents went into the product.

## The token bill is coming

Here's my prediction. Right now most companies treat AI coding spend as an
experiment. That won't last. CTOs will start putting the brakes on token spend
wherever it isn't moving revenue or cutting operating costs, and every team using
agents will be asked what the spend bought.

The teams that did the upfront work will have an answer ready: the use case, the
customer jobs, and the measures that moved. For us that's load time, daily usage,
and cost to run. Lines of code never come up.