Evaluation Suite

An Evaluation Suite groups the criteria that matter for an agent into a single runnable artefact. The criteria can be LLM-as-a-Judge scores, deterministic checks against tool outputs, policy-adherence evaluations, regression tests against named scenarios, and aggregate metrics over production traces. The Suite runs on a schedule, on every change to the policy library, and on demand. The AI Operations Lead owns the Suite. Domain experts contribute criteria. The Suite is versioned, inspectable, and traceable to the decisions it scored. An Evaluation Suite is the evidence object the human governance gate consults when reviewing autonomous-improvement refinements.