There’s a moment that arrives for everyone who ships an agent. It works on your test questions, you’re pleased, you put it in front of real users, and then someone reports that it did something baffling. It looked up the wrong record, or looped three times when it should have looped once, or confidently gave an answer built on a tool result it misread. You go to investigate and realise, with a sinking feeling, that you have no idea what actually happened inside that run. The answer is all you kept. The whole chain of decisions that produced it is gone.
That moment is what this article is about. Evaluating and observing agents is genuinely harder than evaluating a plain model, and the reason is the loop. A chatbot gives one answer you can judge. An agent produces a whole trajectory: a sequence of decisions, tool calls, observations, and steps, any one of which can go wrong while the others go right. To run agents responsibly in production, you need to see that trajectory and you need ways to judge it. Let’s build up both.
Why agents are hard to evaluate
Three properties of agents conspire to make evaluation difficult, and it’s worth naming them so the solutions make sense.
The first is that agents are multi-step. A single user request might trigger eight model calls and five tool calls. The final answer being wrong doesn’t tell you which of those thirteen steps went off the rails. You could have a perfect plan wrecked by one bad tool result, or a flawless execution of a bad plan.
The second is that agents are non-deterministic. Run the same request twice and the agent may take different paths, use different tools, phrase things differently. That’s often fine, there can be many good ways to solve a task, but it means you can’t just check for one exact expected output the way you’d test ordinary code.
The third is compounding. As we saw with multi-agent systems, small errors early in a trajectory don’t stay small. A slightly wrong retrieval leads to a slightly wrong inference leads to a confidently wrong answer. By the end, the mistake is large and its origin is buried.
Put together, these mean you can’t evaluate an agent by only looking at its final answers. You have to be able to look inside the run.
Observability: seeing inside the run
Before you can evaluate anything, you need to see it. This is observability, and for agents it’s non-negotiable. The core practice is tracing: recording every step of every run so you can reconstruct exactly what happened.
A good trace captures, for each step, what the model was asked and what it decided, which tool it called and with what arguments, what the tool returned, how many tokens were used, how long it took, and how much it cost. String those per-step records together and you have the full story of a run: the trajectory, laid out, ready to inspect.
Figure 1: A single run unrolled as a trace. Each step records the model’s decision, the tool call and its result, plus tokens, latency, and cost. The trace is what turns a mysterious answer into a story you can read.
Figure 1 shows a trace laid out step by step. When that baffling user report comes in, this is what saves you: instead of guessing, you pull up the trace and read exactly where things went wrong. Notice that the trace also carries tokens, latency, and cost per step, which ties straight back to the monitoring signals from the LLMOps track. Agents make those signals more important, not less, because one user request can quietly balloon into a dozen expensive calls.
Looking across many traces reveals a subtle benefit too: patterns tell you things no single trace can. If most runs solve a task in two steps but a stubborn few take nine, those nine traces are where your improvement work lives.
Evaluation: judging what you see
Observability shows you what happened. Evaluation is deciding whether what happened was good. For agents, there are two distinct things worth judging, and conflating them is a common mistake.
The first is outcome evaluation: did the agent achieve the goal? For the article-writing agent, did it produce a correct, well-sourced article? This is what ultimately matters to users, and it’s the primary thing to measure.
The second is trajectory evaluation: was the path sensible, even if the outcome was fine? Did the agent take a reasonable route, or did it stumble to the right answer through nine wasteful steps and two lucky recoveries? A good outcome from a bad trajectory is fragile, it’ll fail the next time luck runs out, so trajectory quality is a leading indicator of problems that outcome scores alone will miss.
You want both. Outcome tells you if users are being served today. Trajectory tells you whether that’ll still be true tomorrow.
Figure 2: The same task judged two ways. Outcome evaluation asks whether the goal was reached. Trajectory evaluation asks whether the path there was sound. The dangerous quadrant is a good outcome reached by a bad path.
Figure 2 maps these against each other, and the quadrant to watch is the top-right-to-bottom tension: a right answer reached by a bad path. Those runs pass an outcome-only check and then fail you in production when conditions shift.
How to actually run evaluations
Concretely, evaluation for agents rests on a few practices, all of which build on ideas from earlier tracks.
Keep a task set, a curated collection of representative requests with known good outcomes, the agent equivalent of the golden test set from the LLMOps track. Run the agent against it whenever you change anything, and compare.
Use LLM-as-judge for the fuzzy parts. Since agent outputs are open-ended, you can’t always check them with a simple string match, so you use a capable model, given a clear rubric, to score whether an answer is correct, complete, and grounded. It’s imperfect but scales, and paired with a smaller set of human-reviewed cases it works well.
Above all, follow the never regress rule. Every time the agent fails in production in an instructive way, distil that failure into a test case and add it to your set. Over time your task set becomes a memory of every mistake the agent has made, and no fix is allowed to reintroduce an old bug. That growing library of past failures is one of the most valuable assets a mature agent system has.
The through-line of this whole article is simple: you cannot improve what you cannot see, and you cannot trust what you have not measured. Trace everything, judge both the outcome and the path, and turn every failure into a permanent test. Do that and the baffling-user-report moment stops being a crisis and becomes a five-minute lookup.
In the final article of the track we confront the flip side of everything that makes agents powerful. An agent takes real actions in the real world, and that means it can be attacked and it can cause harm. Next up: securing AI agents.