Writing
Nobody Asked for a Chatbot
This week’s dbt Roundup, Open Data Infrastructure in the age of AI agents, makes the case that companies should build their own AI analysts on open data infrastructure instead of buying one off the shelf. Tristan Handy’s version of the argument is the one I’d underline: most of the work is already done if your data is in one place, modeled well, and documented. The MCP servers and the model choice in the middle are, in the newsletter’s words, not that big.
I agree with that, and I’d go further: data modeling matters more with agents in the picture. Where I’d add something is the last step, which the episode mostly skips: how the findings actually reach the people who use them.
The default picture of an AI analyst is a chat box where anyone in the company types a question and gets an answer. I’ve run analytics for a while now, and I’ve logged several hundred requests. Not one of them asked for that.
Three kinds of users, none of them asking for chat
When I sort the people who use our analytics by what they actually want, I get three groups.
Most users want the answer waiting for them. They have the same questions every week or every quarter. They don’t want to phrase a question, judge whether the answer is complete, and decide if they trust it. That’s analyst work, and a chat box hands it to them.
Analysts want time to go deeper. This is where agents have changed my team the most, and I’ll come back to it.
Power users want access closer to what my team has. Our CFO does their own analysis, and I don’t get in the way of it. What power users get is a negotiation over permissions, data access, and tools. In my experience they don’t want a chatbot either. When they want an agent, they want their own.
That leaves chat squeezed in the middle. Most users don’t want to ask questions, and the users who do want something stronger. I think of chat as an interface looking for a user: it’s built for someone who wants to explore but doesn’t need real access, and I haven’t met many of those people.
Finding the repeatable questions
Every request my team gets goes into HubSpot ticketing. That started as a way to manage a queue, but the log turned out to be the useful part. When you can look across several hundred requests, the repeats are obvious.
The clearest one was some version of “how does product usage relate to outcomes?” It came from different people with different wording, but it was the same analysis each time. That’s the signal I look for: a question that keeps coming back is one that should already be answered before anyone asks. So we built it once, properly, and put it where people could get to it without a ticket.
The request log tells me what people know to ask for. Usage monitoring in our analytics tells me what they actually use once it exists. I need both, because those two lists are not the same.
Agents let analysts go deeper
The article’s best argument for agents is the long tail. Handy points out that human analysts handle the expected questions well, and agents can take on the open-ended exploration nobody has time for. He cites Ramp’s research agent growing to roughly 50 times the data team’s question volume in six months.
Worth being straight about: Ramp’s agent was a chat interface in Slack, open to anyone. That’s the strongest case against what I’m arguing here. My read is that 50x the questions tells you the demand was real. It doesn’t tell you how many of those answers were right, or who checked them before someone acted. In my setup, that checking is the analyst’s job.
I agree, but I put the agent in a different seat. In my setup, the analyst is the one asking the long-tail questions.
Our ICP work is a good example. Describing the ideal customer profile is easy. Testing which traits actually line up with wins and losses is slow, because every trait has to be checked against real deal outcomes. With agents doing the legwork, my team could go much deeper on which traits held up against actual win/loss data, instead of stopping after the first few cuts.
The difference from chat is who stands between the model and the decision. An analyst knows when a number looks wrong, which join double-counts, and which trait is really a proxy for deal size. The end user gets the finding after someone with that judgment has checked it.
Power users bring their own agents
Our marketing leader is a good picture of where power users are going. He has his own agent, his own instructions, and his own flows, and he uses them to drive campaigns. Today that runs on exports plus MCP connections to Google Drive and HubSpot.
What he connects to is curated analysis my team has done, often analysis he asked for. He never touches raw data. Our ESSA Level III study is the clearest case. My team did the research. He turned its findings into emails and web pages through his own agent.
That changes how I read one of the article’s warnings. The guests argue against wiring agents straight to SaaS tools, because each system organizes data around its own roadmap and MCP is just an access protocol. Our marketing leader does wire his agent straight to Drive and HubSpot, and it works. It works because what’s sitting in Drive is analysis we already modeled, checked, and wrote up. The context problem got solved upstream.
So I’d put the rule a little differently: don’t point agents at data nobody has curated. Curated views exist because analyzing data takes more than having access to it, and a confident wrong number in a campaign costs more than waiting a day for the right one.
How I know it’s working
The evidence is mostly usage. Our partner success managers tell me they’re in the internal analytics every day, and the thing they mention is the depth and how fast they get answers. Customers say the same about our analytics platform: the depth and how easy it is to use. Usage of the new analytics is up three to four times over last year.
We got there without a chat interface. What drove it was answering the repeat questions in advance, letting analysts go deeper, and giving power users access they could build on.
The part nobody asks for
There’s a tension in everything above. I’ve said my users never asked for chat. I’d also say the most valuable thing a data team does is find what nobody asked for. The Ramp story is exactly that: demand nobody saw until the tool existed.
Here’s how I resolve it. The request log and the usage data show you demand people know about. Value nobody requested comes from someone with experience looking at the data, keeping up with new techniques, and seeing what users couldn’t picture. That takes regular research into what’s newly possible, and it’s a big part of my job.
Adding item response theory (IRT) analysis to our assessment reporting is my best example. Nobody filed a ticket for it, and nobody would have, because you have to know IRT exists before you can ask for it. It’s a psychometric method that estimates how difficult each question is and how well it separates stronger students from weaker ones, so reporting can say something about the questions themselves, beyond percent correct. That idea came from keeping up with assessment research. It never showed up in the request queue.
A chat box hands that discovery to the user, who is the person least equipped to do it. An agent in an analyst’s hands multiplies it.
What’s next
The direction I’m heading is the opposite of chat. Instead of waiting for people to come ask, findings should go to them: push alerts, findings sent by email, and findings that trigger a workflow directly. If a finding matters to someone’s job, they shouldn’t have to come looking for it.
That’s still the data team’s job. The modeled data has to be right, and the analysis has to be checked. The only thing that changes is the delivery.