LLM-as-a-Judge

LLM-as-a-Judge replaces hand-graded evaluation at scale. The judge model receives the production trace, the authored criteria, and the scoring rubric. It returns a score and a rationale per criterion. The pattern is one instrument among several in EvalOps; it complements deterministic checks and human annotation. The judge's criteria are authored by domain experts and the AI Operations Lead. The pattern is auditable: every score carries a rationale and a reference to the trace it scored. LLM-as-a-Judge is appropriate for criteria that require linguistic or semantic judgement. It is not appropriate for criteria that admit deterministic checks: those run deterministically.