LLM-as-a-judge and human evaluation

Overview

• LLM-as-a-judge uses a model to grade an answer against a rubric.
• Human evaluation uses reviewers with defined criteria.
• Both produce judgments that need calibration and can have bias or inconsistency.

Why this matters

Open-ended language quality is difficult to capture with deterministic checks alone.

An everyday example

An editor grades clarity while an accountant checks exact totals.

A technical example

Rules check source IDs and claim receipts; a judge scores explanation completeness; staff review sampled consequential cases.

How it works

1. Write rubric
2. Create human-labeled calibration set
3. Compare judge agreement
4. Blind/randomize comparisons
5. Record reasons and disagreements
6. Recalibrate over time.

Architecture and flow

Output → deterministic checks + calibrated judge + human sample → release assessment.

The insurance assistant example

Staff assess whether an explanation is understandable without promising coverage that has not been confirmed.

When to use it

Subjective clarity/completeness checks and scalable screening alongside objective tests.

When to avoid it

Never use an uncalibrated model judge as the sole privacy, financial, or authorization gate.

Interview answer

“LLM judges help scale rubric-based evaluation, but I calibrate them with humans and retain deterministic checks for exact constraints.”

Benefits, trade-offs, and production considerations

• Judges can prefer verbosity, recognize their own style, follow injected answer text, or favor one position in comparisons.
• Track judge version, prompt, disagreement, and error rates on adversarial examples.