Project · Flagship

Agentic Analytics

Ask a question about a business in plain English and get back an answer that is verified against the data and cited — not a plausible-sounding guess. Wiring an agent to a database is the easy part. The work is knowing when the answer is wrong.

The problem

Most text-to-SQL demos stop at "the model wrote some SQL and said some words." The question that decides whether a data team can put this in front of stakeholders is a different one: are the words true? A number that reads well and isn't backed by the query is worse than no answer at all, because someone acts on it. So this project treats verification as the main event, and ships an eval that measures it.

How it works

The agent plans a set of queries, writes and runs read-only SQL against the warehouse, and narrates a result. Then an independent judge — a separate call with a fresh context — checks that every figure in the answer traces back to a row the agent actually retrieved. If a claim isn't grounded, the judge's specific objections go back to the agent for one repair pass. Only a grounded answer ships.

Built on real, public data

The warehouse is the Olist Brazilian e-commerce dataset — a real marketplace of ~100k orders across orders, items, payments, reviews, customers, and sellers — loaded into DuckDB. Nothing about the method is specific to that data; the schema is small enough to hand to the model directly, so the interesting surface is the verification layer and the eval, not a retrieval-augmentation stack.

Verification is the point

The eval is where the design lives. It runs a benchmark of 26 questions — counts, joins, ratios, windowed ranks, and deliberate traps that a naive query gets wrong (counting orders as customers, double-counting revenue across a fan-out join, forgetting a status filter). Every question has a hand-written gold SQL, so scoring is objective, and two things are measured separately because they fail for different reasons:

• Headline accuracy — did the answer state the right number?
• Groundedness — did every claim trace to a result the judge could see?

An answer can have the right headline and still smuggle in an unsupported figure. Splitting the two is what makes the verifier's contribution legible.

What the numbers say

To see the verification layer actually earn its keep, I ran the same benchmark with two different SQL-writing agents while holding the judge fixed. With a strong agent (Opus 4.8) the layer is insurance that rarely fires — it caught one ungrounded side-claim and otherwise had nothing to correct. Swap in a weaker, cheaper agent (Haiku 4.5) and it becomes load-bearing: it caught every wrong answer and recovered full accuracy through the repair pass. In both conditions, zero wrong answers reached the user.

The honest read: on a small, clean warehouse with the schema in the prompt, a strong model is good enough that verification is mostly a groundedness gate. Its value scales with how hard the task is for the agent — which is exactly why the cheap-agent-plus-strong-judge pairing landing at the same 100% is the interesting result. It's a real cost lever, not just a safety net. (One caveat I'll state plainly: this run had a single outright-wrong headline, so the sturdier signal is the groundedness gap, not the catch rate. A dirtier or larger dataset would stress it harder — that's the next experiment.)

Try it

The source, the benchmark, and the eval harness are on GitHub: codybakken/agentic-analytics. It runs on a generated sample warehouse out of the box, so you can clone it and ask it a question without downloading anything.

Coming soon: a short recorded walkthrough — a question, the plan, the SQL, the verdict, the cited answer, and a trap the verifier catches — plus a gated live demo for anyone who wants to run it hands-on.

Read more

The method behind the verification layer is written up in the writing — including how I decompose agent accuracy and why grounded, defensible measurement is the part that actually holds up.

← All projects