Why your data agent gets metrics wrong

October 2026 · 6 min read

Give Claude or Cursor read access to your warehouse and it will happily answer "what was revenue last month?" The SQL will run. The number will look plausible. And there's a good chance it's wrong.

Not because the model can't write SQL. Modern models write excellent SQL. It's wrong because the answer depends on decisions that live in people's heads, not in the schema.

The failures are almost never syntax

When we look at logs of agents answering real business questions, the wrong answers cluster into a handful of types:

Metric definitions. "Revenue" excludes refunds, test orders and internal accounts. The schema has an amount column and nothing else.
Entity definitions. A "customer" is an account with a paid invoice, not every row in accounts. An "active user" means 2+ sessions in 28 days, not 1.
The wrong table. events_raw has bot traffic and duplicates. Everyone on the team knows to use events_clean. The agent doesn't.
Calendar and unit conventions. Fiscal year starts in February. Amounts are in cents. Timestamps are UTC but the business reports in Pacific time.
Join fan-out. Joining orders to line items before summing order totals silently multiplies revenue.

None of these is visible from information_schema. Every one of them is something a human analyst learned by being corrected once.

Why a semantic layer alone doesn't fix it

The textbook answer is "build a semantic layer": dbt MetricFlow, Cube, LookML. They're great, and if you have one, your agent should use it. But for most small and mid-size teams there are three problems:

It's never complete. You model the 20 metrics on the exec dashboard. People ask about the other 200.
It goes stale. The YAML gets written once, during a big modelling push, and drifts from how the business actually talks.
Nothing measures it. You have no idea whether your agent is right 60% or 95% of the time, so you can't tell whether all that YAML helped.

The fix: capture corrections where they happen

The highest-signal moment in the whole system is when a human says "no, that's wrong, revenue should exclude refunds". Right now that knowledge evaporates: it fixes one chat, on one laptop, for one person.

A better loop:

Before writing SQL, the agent looks up the team's definitions relevant to the question.
When a human corrects it, the agent records the correction as a proposed rule.
A human approves it (one click), and from then on every agent on the team gets it right.
Every question is logged as confirmed or corrected, so you get an accuracy number per week and can see it move.

This is cheap to run because the agent does the work: it already knows when it's been corrected. You just need somewhere shared to put it, and a way to serve it back.

Doing it yourself

You can get 80% of this with no tooling at all:

Keep a DATA_DEFINITIONS.md in a shared repo and reference it from CLAUDE.md / Cursor rules.
Add a rule: "When the user corrects a metric or filter, propose an edit to DATA_DEFINITIONS.md."
Once a week, review the proposed edits.
Keep a spreadsheet of questions and whether the first answer was right.

It works, and it's a good way to discover how many of your agent's errors are definitional (in our experience: most of them).

Or use Contextloop

We built Contextloop to do exactly this loop without the spreadsheet. It's an MCP server your agent calls: get_context before writing SQL, report_outcome when you confirm or correct an answer. Corrections arrive in a review queue, approved definitions are served to every agent on the team, and the dashboard shows weekly accuracy. It never touches your warehouse. Your agent keeps using its own database tools. You can export everything to Markdown, dbt MetricFlow or Cube YAML at any time.

It's free to start and takes about two minutes to connect to Claude Code or Cursor.

Try Contextloop free

Questions, or want us to run a free accuracy pilot on 50 of your team's real questions? Email delphine.taylor@kivamail.com.