How to test an AI evaluator before trusting its scores

A second model creates a second measurement problem

It is easy to put an evaluator beside an agent. The agent produces an answer, another model scores it, and a dashboard begins to fill. The difficult part starts when someone uses that score to change a model, expand a rollout or remove a review step. At that point, the evaluator has become an instrument for making decisions. Its own errors belong in the decision.

I would start with a smaller question than whether a judge is good. What decision will it support? A rough comparison of two draft-writing prompts can tolerate errors that would be unacceptable in a check for an invented payment instruction. A judge might be useful for the first job and unsuitable for the second, with exactly the same agreement score.

I need to know which people labelled the examples, which criterion they applied, where they disagreed and how the automated decisions differ. Human judgment can be uncertain. Hiding that uncertainty behind a model score makes the measurement look cleaner while leaving the underlying question unresolved.

Shreya Shankar and colleagues studied this through EvalGen. They describe criteria drift: seeing outputs helps people discover and refine what they mean by good work. Their work supports iteration between examples, criteria and evaluators. It also makes a fixed benchmark score less reassuring when the definition of good is still changing. EvalGen, 2024

Clever Hans could read the questioner

In 1904, Oskar Pfungst investigated Clever Hans, a horse presented as able to answer arithmetic questions by tapping a hoof. Pfungst changed the testing conditions: could Hans still answer when the questioner did not know the answer, or when the horse could not see the questioner? The apparent arithmetic ability depended on the questioner’s knowledge and visible movements. Pfungst identified small, unintended movements that told the horse when to stop tapping. Pfungst, Clever Hans, Chapter II; English translation, 1911

The revealing experiment changed where information could leak into the answer. I would use that logic when testing an AI evaluator. Hide the provider name, swap answer order and remove a confident preamble while keeping the substantive answer fixed. If the score changes, inspect whether the criterion justifies it. A polished response may give the evaluator a cue that looks like quality. The horse’s behavior supplies a useful experimental question; actual paired tests are still needed to discover whether a particular AI judge has that problem.

Define the decision before choosing a score

Consider a fictional research assistant that reads a museum's membership rules and prepares a replacement-card draft. The member asks for the draft to remain unsubmitted. A response can explain the policy correctly while submitting the order. Combining clarity, factuality and operational correctness into one average would obscure the failure that matters most.

I would separate three checks. A program reads the authoritative record and submission log to check whether a draft exists for the right membership identifier and whether an order was submitted. An evidence check compares factual claims with the policy and tool results. A semantic judge assesses whether the explanation is useful and complete. Each check needs evidence suited to its question.

A semantic judge adds value where a correct answer can take many forms. It can assess whether an explanation captures an exception without demanding a specific sentence. A program is usually the simpler instrument for counting records or comparing identifiers. I would keep mechanically observable requirements outside the model's discretion.

Google's managed agent metrics distinguish task success, tool-use quality and trajectory quality. Adaptive rubrics can generate criteria for a prompt, then judge whether those criteria were met. That gets specific tests quickly. The generated criteria still need review, especially when a short instruction carries most of the operational risk. Google's metric definitions

Check which failures the evaluator misses

Here is a deliberately constructed calibration set of 200 examples. Human review marks 160 acceptable and 40 unacceptable. The judge passes 152 acceptable examples and wrongly passes 12 unacceptable examples. It rejects eight useful outputs and catches 28 failures.

The judge agrees with the labels on 180 of 200 examples: 90%. It passes 95% of acceptable outputs and rejects 70% of unacceptable outputs. Viewed from the failure side, it misses 30% of the bad work. All four statements describe the same constructed judge. Only some tell a reviewer how much unfit work it may allow through.

The denominator needs to travel with the rate. Twelve false passes divided by 40 unacceptable examples measures failure to catch bad work. Twelve divided by 164 judge-passed examples measures contamination of the passed group. I would keep both visible, alongside the counts supporting each estimate.

An always-pass evaluator would agree with 80% of the labels while catching no failures. That is a useful calibration control. If an elaborate judge produces little improvement over a trivial rule on the error that matters, I would investigate the labels and instrument before investing in a more impressive dashboard.

The work mix changes the meaning of a score

Suppose the same 95% true-pass rate and 70% true-rejection rate transfer to 1,000 new cases. At 2% bad work, the expected counts are 931 good passes, six bad passes, 49 good rejections and 14 caught failures. About 0.64% of the passed group is bad. At 20% bad work, the expected counts include 760 good passes and 60 bad passes; contamination rises to about 7.32%.

The conditional error rates are identical. The population changed. I would keep an intentionally difficult calibration set separate from a representative sample used to estimate operating impact. Oversampling failures helps examine the detector; treating that sample's unweighted pass contamination as a production rate gives a different quantity.

A stronger assumption sits underneath: the judge performs equally well on the new failures. A different language, tool schema or document type can change its errors. Reweighting old measurements cannot repair that shift. I would check transfer on the actual slice before using a prevalence-adjusted estimate to justify less human review.

Calibrate against the costly mistake

The two error directions create different costs. Rejecting useful work creates review or rework. Passing unacceptable work may create downstream repair, customer harm or an unauthorized action. A release rule should reflect those consequences rather than treating every disagreement as equivalent.

For the constructed 20%-failure population, suppose a bad pass costs 100 units and a good rejection costs two. The first judge produces 60 bad passes and 40 good rejections, for an illustrative loss of 6,080 units. A stricter judge that passes 85% of good outputs and rejects 95% of bad outputs produces ten bad passes and 120 good rejections: 1,240 units. The second has lower overall agreement, 87% rather than 90%, and lower illustrative loss.

Reverse the economics and the choice can reverse. If rejecting useful work costs 20 units and a bad pass costs one, the first judge's loss is 860 units while the stricter judge's is 2,410. These are assumptions in a teaching calculation. A team must estimate its own consequences, including whether severe events should be counted separately rather than priced into an average.

I would discuss this with the process owner before choosing a threshold. What can go wrong, who resolves it and which mistakes justify stopping the workflow? A generic target such as 90% agreement cannot supply those judgments. Nor does raising the target solve a test set containing none of the severe failures under discussion.

Freeze the criteria for a fair comparison

Criteria will change as people encounter unfamiliar outputs. I would give that discovery a place in development. Reviewers can add a missing criterion, explain a disagreement and revise examples. Once a comparison supports a release, both versions should face the same reviewed criteria and examples. A changed criterion starts a new measurement series.

That suggests three pools. Exploratory examples reveal failure modes. Development examples improve the rubric and judge prompt. A final test set assesses the resulting instrument. Split related examples together by conversation, source document or task family; a near-copy across the split offers weak evidence of transfer.

Google documents generating rubrics before evaluation and reusing the rubric group. I would also save evaluator model and metric versions, because a managed default can change while the application stays still. A higher score after such a change has more than one possible explanation. Rubric reuse, metric versioning

A held-out set tells us how the instrument behaves beyond the examples used to shape it. Repeatedly inspecting and adjusting against the same set spends that independence. Preserve what changed, collect fresh examples when needed and resist reporting every later improvement against the original number as if the comparison remained untouched.

Check whether presentation changes the verdict

A judge can respond to qualities irrelevant to the task. Zheng and colleagues' MT-Bench study examines position, verbosity and self-enhancement biases alongside the usefulness of model-based judging. That supports testing behavior under presentation changes; its historical results cannot supply a current accuracy estimate for whichever judge we select. MT-Bench, 2023

I would make paired examples that preserve the substantive answer while changing one nuisance factor. Swap answer order. Remove provider names. Add an unnecessary preamble. Reorder evidence. Inspect how often the verdict changes and whether a reviewer can explain the difference through the criterion.

Some changes are substantive. Shortening an answer can remove a qualifier; changing a tool result can alter what is justified. Human review of the pair comes first. Otherwise our test may call a real quality difference a judge defect. Keep examples and rationales alongside any reported flip rate.

I would also place an instruction inside the material being evaluated, asking the judge to award a pass. The evaluator should treat it as content. It processes untrusted model output and retrieved evidence, so instruction boundaries matter here too. A stronger model can improve interpretation while leaving this boundary problem unresolved.

Count uncertainty and missing judgments

Zero missed failures in a small sample is encouraging, but substantial uncertainty remains. Under independent Bernoulli trials with a stable miss probability, the chance of zero misses is (1−p) raised to n. Setting that probability to 0.05 gives the one-sided 95% upper limit p = 1−0.05^(1/n).

For 20 labelled bad examples with zero misses, the limit is about 13.91%. For 100, it is about 2.95%. The limit concerns misses among bad examples, rather than the bad share of passed work. These calculated limits depend on the sampling assumptions. Shared templates and correlated tasks reduce the information in a nominally large sample. A hundred variants of one failure give a thin basis for a claim about a broad workflow.

An evaluator timeout needs a distinct outcome. Retain the attempt as unjudged, show coverage and route it according to the workflow's consequence. Silently removing unjudged cases can improve the displayed score when the hardest inputs are precisely those that fail to receive a judgment.

There are two denominators to keep apart: the agent attempts we wanted to assess and the attempts that actually received a usable evaluation. I would report both. A team can improve its grader's reliability without changing the agent, or improve the agent while its evaluation coverage deteriorates. One blended percentage hides those movements.

Start with a small instrument challenge

The practical companion reproduces the arithmetic and specifies an authored test matrix: wrong entity, missing qualifier, prohibited action, conflicting evidence, presentation changes, an embedded grading instruction and evaluator timeout. Each row asks what the human expects, what the automated check can observe and how disagreement will be resolved.

The first exercise is to construct known good and known bad examples. Make the bad examples plausible: a polished answer with the wrong identifier or a confident completion claim after a failed write. Include a correctly reasoned refusal when permission is missing among the good examples. A judge that handles only conspicuous nonsense is unlikely to help with the errors that survive ordinary development.

Then introduce ambiguity deliberately. Two experts may interpret completeness differently while agreeing on the underlying facts. Resolve whether the criterion needs a range, a separate category or a process decision. Forcing every disagreement into pass or fail can produce labels that look consistent while failing to represent the actual work.

Running the companion's calculations establishes their arithmetic. The matrix becomes evaluation evidence when an actual agent and judge are run against reviewed examples. I would keep those stages separate so an attractive test design never becomes an unearned performance claim.

Give the evaluator an operating owner

A strong economical alternative is deterministic checks plus selective expert audits, with no semantic judge. For a small, high-consequence workload, experts may review every output. Automated judging adds prompt maintenance, calibration effort, cost and another failure path. I would compare its incremental failure detection and review savings against those simpler designs. It earns its place when it helps experts inspect meaningful variation or reduces repeated review without concealing mistakes.

I would keep the domain owner responsible for the criterion, the engineering owner responsible for implementation and evidence, and a named reviewer responsible for material disagreements. These are jobs, not necessarily three teams. The small version can be one explicit review session. The larger version needs a record that survives staff changes and model defaults.

Before expanding reliance, I want conditional error rates on important slices, uncertainty, disagreement examples and the cost of review left behind. I also want a route back to fuller review if failure patterns change. Monitoring passed cases is part of that route; a clean-looking failure dashboard cannot reveal everything the judge missed.

The decision I would defend is specific: trust a particular version to support a particular decision, on work whose limits we understand, while keeping an independent way to discover when that trust stops being justified. That makes evaluation an engineering practice that can improve, rather than a score the organisation learns to repeat.