Evaluation (Evals)

The dataset is the artefact that matters. Fifty inputs drawn from real usage, including the awkward and adversarial cases, are worth more than a thousand synthetic ones, because they encode the distribution you actually serve. Every production failure should end up in that set, which makes the suite grow in exactly the directions your system is weak. Scoring is then chosen per case: deterministic assertions where a correct answer exists, heuristics for format and constraint checks, a model-as-judge with an explicit rubric for qualitative dimensions, and human review for the subset where none of those are trustworthy.

For agents, evaluating the final answer alone hides most of the information. Trajectory-level evaluation asks whether the right tools were selected, whether arguments were well-formed, how many steps were taken and where the run went sideways — an agent that reaches a correct answer after nine wasted calls is a different system from one that reaches it in two, and only trajectory metrics distinguish them.

The misconception is that public benchmarks tell you what you need to know. They measure general capability on tasks that are not yours, on data that may have leaked into training, with prompts unlike your own. They are useful for shortlisting a model and useless for deciding whether your change helped. The only evaluation that answers that question is one built on your inputs and your definition of correct.