Writing
The Work That Makes AI Coding Agents Pay Off
A study making the rounds this month put numbers on something a lot of engineering leaders have felt. As Ars Technica reported, Harvard researchers Fiona Chen and James Stratton looked at work data from more than 700 software companies and found that after teams adopted AI coding agents, lines of code went up 30%, commits 20%, and pull requests 23%. The rate at which finished features shipped didn’t change in any statistically significant way.
The authors trace the gap to code review. After agents arrived, the time from pull request to merge rose 49%, the share of pull requests sent back for changes nearly doubled, and reviewers left 35% more comments. In the authors’ words, the gains in the coding phase were “absorbed by downstream constraints in the production process.”
I don’t doubt the review finding. When you can produce code faster than people can read it, review becomes the constraint. But I read the result as a planning problem as much as a review problem. When building gets cheap, the expensive mistakes are building the wrong thing, or building the right thing without a plan and paying for it in review. Review is where a missing decision finally gets noticed, and it’s the most expensive place to notice it.
My team writes most of its code with AI agents, and we move much faster because of it. Here’s a project where that speed turned into value, and the work that made the difference.
The project
Our customer-facing reporting started on an embedded BI tool, and it served our initial use case well. I’d recommend that path to most teams starting out: a mature BI tool gets you credible reporting in front of customers quickly, and you learn what they actually use.
What we learned, over time, was where the embedded setup stopped fitting what customers needed. So we moved the reporting onto an open-source stack built directly into the product. Agents wrote most of the code. Without them, the same project would have needed more people and a lot more time.
The results:
- Pages load in under a second. Some dashboards had been loading slower than our users needed.
- Daily page views are roughly 3–4x what they were before the move.
- It costs less to run, with no BI licensing for the embedded reports.
Three pieces of work, all done before the agents started, account for those results.
1. The user research already existed
I designed the original dashboards, and that work started with research into who used them and what decisions they were trying to make. That research didn’t expire when we changed platforms. Most of the design work for the new version was translating what we already knew into the new stack, so the experience carried over.
That matters more than it sounds. A migration that makes users relearn their reports spends the goodwill the new features are supposed to earn. Because the new platform kept what already worked, the switch didn’t feel jarring to customers, and they could put the new features to use right away.
2. The backlog chose the framework
We had a backlog of feature requests that were hard to meet with an embedded BI tool. Two examples:
- Sorting and filtering large tables. Customers wanted to work with tables directly: sort a column, filter a few values, find the rows that matter. Grid libraries like AG Grid are built for exactly that.
- Sharing a link that keeps its filters. A user narrows a report to what they care about and wants to send that exact view to a colleague. That’s hard when the reporting sits inside another application and users can’t reach the BI server directly, so a shared link can’t carry the filter context with it.
There were many others, but the pattern was the same. Each request was a job a customer was trying to get done, and the backlog gave us a concrete list of jobs the new platform had to do well. We picked the stack against that list. The cost savings were real, and they came along with a product that did more for customers.
3. The architecture came before the agents
Before anyone wrote code, I planned the structure of the codebase and the application, down to the folder structure, and pointed the agents at that plan.
This is the part I’d stress for any team building with agents. Agents are good at filling in a structure you’ve defined. Left to invent one, they produce a lot of code that doesn’t hang together, and someone has to review all of it. That’s the study’s bottleneck showing up one step earlier. A clear structure gives every piece of generated code an obvious place to live and an obvious standard to be reviewed against, so review can focus on whether the code does its job.
The old discipline still applies
None of this is new. The data science life cycle starts with business understanding for a reason: you identify the value and the job to be done before you build anything. Software teams have their own versions of the same step.
What agents change is the cost of skipping it. When writing code was slow, the build itself put a brake on bad ideas, because there was only so much you could build in a quarter. Agents remove that brake. You can now build the wrong thing at full speed, and the bill shows up as review load, maintenance, and a product nobody asked for.
For an analytics team, that puts the most leverage upstream of the code:
- Vet requests against a real use case. A request with no decision or user behind it doesn’t get built because it’s now cheap to build.
- Identify the jobs to be done and write them down, so the build has something to be checked against.
- Prioritize the jobs that move what the business cares about. I wrote about keeping that prioritization honest in A Data Team Without a Roadmap Is a Ticket Queue.
- Settle the architecture before the agents start, so their output has a place to go.
We didn’t build this platform because we could. We built it because we had a use case, a list of customer jobs it had to do, and a plan for the code. With those in place, the speed from the agents went into the product.
The token bill is coming
Here’s my prediction. Right now most companies treat AI coding spend as an experiment. That won’t last. CTOs will start putting the brakes on token spend wherever it isn’t moving revenue or cutting operating costs, and every team using agents will be asked what the spend bought.
The teams that did the upfront work will have an answer ready: the use case, the customer jobs, and the measures that moved. For us that’s load time, daily usage, and cost to run. Lines of code never come up.