How to Evaluate AI Agents (and Not Get Burned)
Evaluating an AI agent is harder than testing code because it is non-deterministic, takes many steps, and a correct-looking final answer can hide a broken path. Grade the real outcome, not the transcript's claim. Check the tools it called, not just the words it said. Use an LLM as a judge, but validate that judge against humans. Start with 20 to 50 tests drawn from real failures, and know that the eval set you build from your own traffic is the durable asset.
You would not give a new hire the production database password on day one and walk out of the building. Yet that is roughly what a lot of teams do with an AI agent: wire it to real tools, point it at customers, and trust that because the demo looked great, the thing works. Then it confidently tells a customer their refund is processed, and the refund is not processed, and now you are on the phone.
Evaluation is how you avoid that call. And evaluating an agent is genuinely harder than testing normal software. Here is why, and what to actually do.
Why this is hard: the math is against you
Regular code is deterministic. Same input, same output, write a test, done. Agents break all three assumptions. They are non-deterministic, so the same request can take a different path each run. They have no single right answer for many tasks. And they act over many steps, which is where the real trap lives.
Because reliability multiplies across steps instead of averaging, a perfectly respectable per-step success rate falls apart over a long task:
And that assumes errors are independent. In reality a misread in step one poisons every step after it, so it is usually worse. This is the whole reason final-answer testing is not enough for agents.
Grade the outcome, not the transcript
Here is the single most useful rule, and it comes straight from how the serious labs think about it. Verify the real end state, not what the agent says. If the task was "issue the refund," check that the refund actually exists in the system, not that the agent typed "your refund is processed." Agents are very good at sounding successful. Language is cheap. State is truth.
The tempting overcorrection is to grade the whole path and demand the agent follow the exact steps you imagined. That backfires too, because a good agent will sometimes find a smarter route than the one you scripted, and you do not want to fail it for being clever. Anthropic has a lovely example of a model finding a booking loophole that technically failed the test but actually served the user better. So the real rule is nuanced: grade the outcome by default, and grade the trajectory only where the process itself carries risk, like making sure it never called the "delete everything" tool on the way to a correct answer.
Speaking of which, check the tools. Did it call the right tool, with the right arguments, in a sane order? That is its own axis of correctness, separate from whether the final answer looked fine.
Use an LLM as a judge, but put it on trial too
You cannot hand-grade thousands of runs, so you use a model to judge the outputs against a rubric. This works, and it is standard. But the judge has biases: it can prefer longer answers, favor its own writing style, and be swayed by which option comes first. Ensembling and reversing option order reduce the random noise, but they do not fix the biases all the judges share.
So you validate the judge the way you would validate any instrument: against human labels. Give it an explicit "unknown" option so it stops inventing grades when it lacks information. Score each dimension separately instead of one vague overall number. And read the transcripts yourself, because you will not know if your grader is fair until you do. An unvalidated judge is just a confident opinion at scale.
Evals are a spec, not a chore
The reframe that changes how teams work: writing evaluations is not a box you tick after building. It is how you decide what "good" even means. Two engineers reading the same feature description will disagree about edge cases; an eval suite forces you to settle those disagreements up front, in concrete tests. Anthropic makes this point directly, evals are specifications. The teams that write them early tend to build better agents, not because evals are magic, but because they were forced to define success before they shipped.
You do not need a hundred perfect cases to start. Take 20 to 50 tasks pulled from real failures and bug reports, build known-good outcomes, balance the good and bad cases, and wire it into CI as a gate. Then add evaluation on live traffic once you are in production, because that is where the failures you did not imagine show up.
The part worth remembering
As models get cheaper and more interchangeable, the thing you actually own is not the model. It is the eval set, the golden collection of your own real cases, graded against your own business logic. Anyone can rent a frontier model. Almost no one can copy your evaluation of what "correct" means for your product. That is the asset.
This is how we build AI agents and copilots: the eval suite comes with the agent, not after it. And it is a fair thing to demand of anyone building agents for you, including us. If a vendor cannot show you how they measure their agent, they are hoping, not engineering. Ask us to show ours.
References
- Anthropic, Demystifying evals for AI agents: grading outcomes, evals as specifications, and starting small.
- RAGAS agentic and tool-use metrics: goal accuracy, tool-call accuracy, and faithfulness.
- LangChain, evaluating trajectories versus outputs: scoring the full execution path, offline and online.
Frequently asked questions
Three reasons. Agents are non-deterministic, so the same input can produce different steps each run. They act over many steps, so a correct final answer can hide a broken path and errors compound. And many tasks have no single right answer, so simple reference grading breaks. You have to measure outcomes and trajectories across multiple runs, not a single pass.
Start with task success: did it actually achieve the goal, verified by the real end state rather than what it claims. Then tool-call accuracy: the right tool with the right arguments. Then, where the process carries risk, the trajectory. Add faithfulness for grounded answers, plus latency and cost. Grade the outcome by default and grade the path only where the path matters.
It means using a language model to score another model's output against a rubric. It is scalable and useful, but it has real biases: it can favor longer answers, prefer its own style, and be swayed by option order. So you validate the judge against human labels, give it permission to answer 'unknown,' and score dimensions separately. An unvalidated judge is an unmeasured instrument.
Small and real. Take 20 to 50 tasks drawn from actual failures and bug reports, build known-good reference outcomes, and grade the final state rather than the transcript. Read the transcripts yourself so you know your grader is fair. Wire it into CI as a regression gate, then add evaluation on live traffic once you deploy.
Building something with AI agents?
We design and ship production AI agents for teams in fintech, crypto, and beyond.
Book a call