Agents must get an entire multi-step sequence right — not just one response. Errors compound. Side effects are real. Valid paths to success are many. These properties make agent evaluation more complex than evaluating a single model output.
Task success rate — Percentage of tasks completed correctly end-to-end. The primary metric. Requires clear, measurable success criteria for each task.
Step accuracy — For tasks with known correct action sequences, percentage of steps executed correctly. Useful for pinpointing where in the loop the agent fails.
Efficiency — Tool calls, LLM calls, and elapsed steps to completion. An agent that takes 5 steps beats one that takes 50, even if both succeed.
Error recovery — When something goes wrong, how often does the agent recover and complete the task vs. getting stuck, looping, or producing an incorrect result?
Ground truth (gold standard): deterministic correct answers — code that passes tests, queries that return the right rows. Scalable and reliable.
LLM-as-judge: a separate LLM scores outputs against a rubric. Useful for open-ended tasks; less reliable than deterministic evaluation.
Human raters review agent behavior and outputs. Most reliable. Expensive and slow. Use for calibrating automated evaluators and high-stakes applications. Always combine with trace-level inspection of failure modes.
Build a test set of representative tasks with known correct outcomes. Automate evaluation where possible. Run on every significant prompt, tool, or model change. Track task success rate over time — regressions in agent behavior are as important to catch as code regressions.
Ask the AI assistant about agent evaluation methods, how to define success criteria, or how to build an automated evaluation pipeline.