Your agent evals are grading the wrong thing

Why the thing you can't measure about your agent is the thing that matters most.

Grading the scratch work instead of the answer

Picture a teacher who grades a math test by reading only the scratch work in the margins. Did the student set up the equation the right way? Carry the right terms? Show clean, sensible steps? The teacher marks all of that, hands back a score, and never once checks whether the final answer is correct.

That is how most AI agent evaluation works today. We inspect the trace — the tools the agent called, the order it called them in, the reasoning it narrated along the way — and we grade the quality of the work. What we rarely check is the thing the whole exercise was for: did the agent actually get the right result?

The outcome is what matters. A perfectly reasoned path to a wrong answer is still a wrong answer, and a slightly messy path to the right one is often fine. So why do we default to grading the steps? Because the steps come in a standard format and the answer doesn't. A trace has structure you can parse. An outcome is whatever the task produced — and there's no schema for "correct." We grade what's easy to grade, not what counts.

I want to make the case that this is backwards, and that fixing it is less an evaluation problem than a data problem.

What trace-based evaluation is good at, and where it stops

Let me be fair to the mainstream approach first, because it earns its place. Trace-based evaluation is genuinely useful, and for some jobs it's exactly what you want.

If you need to know whether an agent picks the right tool for a task, calls it with the right arguments, and doesn't wander through six redundant retrievals to get somewhere it could have reached in two, the trace tells you all of that. For a user-facing agent, the trace also shows you whether the responses came back with the right tone and the right content. When something breaks, the trace is the first place you look; it's how you diagnose why a run went the way it did. None of that is going away, and none of it should.

But two things push against grading traces alone.

First, not every agent is user-facing. Plenty of agents run headless: they process, transform, and act, and no human reads their intermediate chatter. For those, "did it work through the problem correctly" isn't even a question worth asking.

Second, and more important: the more real agency an agent has — the more it performs consequential actions on your behalf — the more the outcome matters relative to the work done to reach it. An agent that reconciles your books, provisions infrastructure, or files a change in a downstream system is judged by the state it leaves behind, not the elegance of its route there.

And here's the catch that trace grading can't escape: the quality of an outcome is frequently impossible to judge from the tool calls. Tool calls have side effects, and the side effects are the part you actually care about. Say your agent replies to a message in Slack, and the particular way it phrases that reply trips an automation that schedules a meeting nobody wanted. Reading the trace, you'd see a reasonable-looking Slack message go out. You would have to trace every downstream consequence of that message to catch the problem. It's far easier to check the calendar API and confirm no meeting got scheduled. You can read every span in the trajectory and still not know whether the final state of the system is right. To know that, you have to go look at the final state itself.

What the trace tells you vs. what the final state tells you
‍
Trace: which tools were called, in what order, with what arguments, and how the agent reasoned between them. Great for diagnosis and debugging.
‍
Final state: whether the world the agent was acting on ended up correct: the record updated, the artifact built, the dataset generated to spec. The actual result.

Why the outcome is the hard part to measure

So why does almost everyone skip the outcome and grade the trace anyway? The answer is tied to the format problem I mentioned earlier.

There is a standard for representing the work an agent does. OpenInference and the OpenTelemetry conventions it builds on give you a shared schema for spans: every tool call, every LLM invocation, every retrieval step, encoded the same way across frameworks. Point an eval harness at a trace and it knows how to read it, because the shape is agreed upon.

There is no equivalent standard for an answer. The "answer" to an agentic task isn't a string you can diff against a golden output. It might be a fully configured calendar. It might be a built artifact, like a working service or a migrated schema. It might be a generated dataset. These are abstract end-states, and you can't judge them by reading the steps that produced them any more than you can grade a cake by reading the recipe. You have to taste it.

By the way, this is exactly where the strongest counter-argument to my position lives. A well-argued recent piece makes the case that grading outcomes alone will lie to you: an agent can reach the right result through a broken, unsafe, or lucky path, and if you only check the endpoint you'll bless behavior you shouldn't. A right answer via a reckless route is a landmine waiting for production.

But the fix isn't to retreat to grading traces. The fix is to grade the outcome at the trajectory level: to verify that the key checkpoints along the way were hit and that the final state is correct, rather than demanding the agent reproduce one blessed reference path step for step. Rigid trajectory grading has its own failure mode: it penalizes valid alternative routes to a correct result, and there are usually many valid routes.

Notice what "verify the final state is correct" quietly requires, though. To check the endpoint against what's correct, you have to know what correct is. And that turns out to be the whole game.

The outcome you can't verify is the outcome you don't control

You can only judge whether a final state is right if you already know what right looks like. And you only know what right looks like if you control the data the agent was acting on.

If the agent operated on a world you generated, you know that world completely: every record, every relationship, every value, and therefore every correct end-state that could follow from it. If it operated on data you didn't create yourself, whether that's a copy of production or a dataset you pulled from somewhere else, you're stuck reverse-engineering the right answer after the fact, and you're often just guessing.

We ran into this ourselves building Tonic Fabricate, our synthetic data product. When we evaluate Fabricate, we don't only care that the agent wrote syntactically correct code and responded with the correct tone and helpful content. We also deeply care that the agent produced high quality synthetic data: does the generated world match what was asked for, hold its relationships together, and behave the way real data would?

We can grade that honestly for one reason. We generated the seed, so we know the ground truth the output should satisfy. The outcome is verifiable because the data underneath it is ours. Take that away and the same evaluation becomes guesswork.

Ground truth is a data problem

Ask most teams where their agent evaluation actually bottlenecks, and it's rarely the grader. It's the ground truth for agent evals — the known-correct answers every meaningful eval has to be checked against. Writing them by hand is slow. Retrofitting them onto production data is worse, because you're inferring what "correct" was for situations you didn't design.

A good practitioner guide on agent evaluation puts the requirement plainly: a strong eval task needs unambiguous specifications paired with known-correct reference solutions. The advice is right. The trouble is that producing those reference solutions is treated as manual labor, done downstream, after the data already exists.

Flip the order and the bottleneck mostly dissolves. When you generate the world the agent operates in, you know every dimension of it as a byproduct of creating it. Ground truths can be generated against the seed data instead of hand-authored or reconstructed later. Structured synthetic data carries the metadata (the schema, the relationships, the distributions) that lets you construct verifiable tasks and their answers programmatically, from single-hop lookups to multi-hop reasoning chains.

This is the same mechanism behind a model we trained on nothing but Fabricate-generated synthetic data, where complete synthetic worlds came with structured ground truth attached. The eval didn't need someone to write the answer key. The answer key fell out of owning the world.

The teams that struggle with agent evaluation metrics aren't short on graders. They're short on data they control well enough to know the correct answer in advance.

The data layer is the thing almost no one owns

Disposable test environments are common now. There are good options for spinning up an isolated place to run an agent and tear it down after. But look closely at what most of them isolate: code execution. They give the agent a clean machine to run in. That's valuable, but it's not the hard part of testing an agent with real agency.

The hard part is the world the agent acts on. Standing up a realistic version of the third-party systems an agent touches — a Jira with believable projects and tickets, a Salesforce with coherent accounts and opportunities, a Slack with conversations that hang together — without mutating a real environment is a data problem first and an infrastructure problem second. You need the data to be internally consistent, and you need something the agent can actually call.

That's the layer almost no eval approach generates and owns. It's the layer Tonic.ai has been building for eight years, with Tonic Structural for structured data de-identification, Tonic Textual for unstructured data de-identification, and Tonic Fabricate generating net-new synthetic worlds from scratch. Fabricate builds both the data and the mock APIs the agent calls against, the endpoints that behave like a real inbox, CRM, or ticketing system, backed by the synthetic data underneath.

We've stood these environments up and torn them down ourselves, enough times to know the data is where the work actually is. Owning it is what makes verifying an outcome possible in the first place.

Where this leaves you

I'm not here to tell you to throw out trace grading. That would be its own kind of wrong.

You need both. The trace tells you why an outcome happened: it's how you debug, how you catch the reckless-but-lucky path, how you understand what your agent is doing. The outcome tells you whether it worked at all. And the outcome is what gives the trace its meaning: a trajectory is only worth grading once you know whether it arrived somewhere correct. Read together, they're honest. Read alone, either one lies to you.

The catch is that the honest half, the outcome, is the half you can't measure unless you know the correct final state, and you can't know that unless you control the data underneath it. That's why I keep calling this a data problem rather than a grading problem. The grader is downstream. The data is the foundation.

The teams that evaluate agents well over the next few years won't be the ones with the cleverest metrics. They'll be the ones who treat the world their agent is tested against as something to build and own deliberately, not something to reconstruct after the fact and hope they got right.