A judge can return clean, confident scores while missing the failures your team needs it to catch. A fluent verdict can look sound even when it conflicts with the human review your team trusts. Before an LLM judge supports a release, ranking, or safety decision, compare its verdicts with human reference labels.
This guide gives you a 13-step way to build and test that comparison. You will choose a task and label scale, prepare examples, measure agreement, inspect errors, and set a release gate. It also covers bias checks, prompt changes, and production monitoring. Human comparison must come before decisions rely on judge verdicts.
The practical goal is evidence of agreement on your own labeled examples, not confidence in a score alone.
Key Takeaways
- Use Cohen’s kappa to adjust observed agreement for chance agreement.
- Choose a label scale that matches the decision humans make.
- Keep prompt-tuning examples separate from held-out evaluation cases.
- Inspect class-specific errors, invalid outputs, and bias signals.
- Set a task-specific ship gate before a judge reaches production.
- Recheck agreement against fresh human labels over time.
Why Does Cohen’s Kappa Matter More Than Raw Agreement?
A judge that always chooses the most common label can look right often without detecting the failures that matter. Cohen’s kappa measures agreement between two raters while adjusting for agreement expected by chance. That adjustment helps you see when raw agreement looks high because one label dominates.
Raw agreement is the share of examples where the judge and human reference label match. It does not account for how often each rater uses each label. If most cases pass, a judge that calls nearly everything a pass may match many human labels while missing important failures. Agreement alone does not establish accuracy.
For example, two raters can agree on a label and both be wrong. Kappa cannot tell whether either rater matches reality or an authoritative standard. It measures alignment with the labels you chose, not correctness beyond them.
Use kappa to measure agreement with your chosen human labels, not to claim that either rater is correct.
The MT-Bench study reported that strong LLM judges reached 80% agreement with human evaluators, close to the reported agreement between humans. That result gives useful context, not a promise for your task. An LLM judge is a use-case-specific proxy, and its agreement depends on its prompt, model, and task complexity.
Legal research, support triage, and answer quality each ask for different judgments. A legal-research benchmark, for example, may score whether criteria were met without proving every answer fully correct. Human review should check whether the measured task matches the decision your team plans to make.
Treat kappa as evidence of agreement with your chosen human labels, not proof that either set of labels is correct.
Which Judge Task and Label Scale Should You Use?
Use the evaluation mode that matches the decision, then choose labels humans can apply consistently. Keep the task narrow: a judge should decide a defined criterion, not rate general quality without a clear rule. A simpler scale often gives a more dependable comparison.
Pairwise comparison asks which of two outputs is better. Criteria-based scoring checks an answer against named requirements. Reference-based evaluation compares the answer with a known response or source material. Each mode supports a different evaluation question.
| Mode | What the judge sees | Best-fit question | Main limitation |
|---|---|---|---|
| Pairwise comparison | Two candidate outputs, A and B | Which answer better meets the task? | Order can affect a verdict |
| Criteria-based scoring | One answer and a rubric | Did it meet each named criterion? | Compound criteria can blur results |
| Reference-based evaluation | Answer and trusted reference | Does the answer match the source? | A reference may be incomplete |
A three-level scale works when your team needs to separate failure, partial coverage, and full coverage. For example, humans can label 0 for failure or factual conflict, 0.5 for partial coverage, and 1.0 for a fully addressed question. Those values preserve a useful distinction without asking reviewers to score tiny differences.
Fine-grained scales can add noise when people cannot tell adjacent values apart. Unweighted Cohen’s kappa suits two raters using nominal labels, such as pass and fail. Weighted kappa suits ordered labels, while Fleiss’ kappa can measure more than two raters on nominal labels.
Linear and quadratic weights assign different partial credit to ordinal disagreements. A disagreement between 0 and 0.5 may count differently from one between 0 and 1. The weighting scheme for ordinal labels must be stated with the result.
For ordered labels, write down the weighting scheme before comparing scores because changing the weights changes the agreement result.
How Do You Build a Gold Set People Can Label Consistently?
Build the set from the workflows the judge will actually score, including both ordinary traffic and difficult boundary cases. Representative data lets you test common use and high-risk failures. Keep prompt-tuning examples separate from held-out evaluation examples.
Sample Common Cases, Boundaries, and High-Risk Examples
Choose cases that reflect real inputs, not only clean examples that make the rubric easy to apply. Include examples from routine use as well as rare events that could cause harm. A calibration set with only ordinary traffic can hide the failures your team most needs to find.
- Common traffic, such as routine support questions or typical user requests.
- Rare high-risk cases, such as safety issues or sensitive decisions.
- Known failures, including factual conflicts and missed instructions.
- Rubric-boundary examples where two labels may seem plausible.
Prompt-tuning examples help you refine the judge, but they cannot also serve as an untouched final test. Keep a held-out group that the team does not use during prompt changes. This separation protects the evaluation from repeated exposure.
Collect Independent Human Labels and Resolve Disagreements
Ask reviewers to label examples independently before they discuss them. Save each person’s original label and any notes, then resolve disagreements through a defined adjudication process. The adjudicated human reference label gives you a final comparison target without erasing the initial votes.
Sample-size planning depends on category count, label prevalence, expected kappa, statistical power, and alpha. Weighted-kappa planning also depends on the weighting scheme and category rates. A sample of 50-100 subjects is a common starting point, not a universal minimum.
One benchmark used three expert annotators on 1,994 samples, producing 5,982 annotations across six datasets. The datasets were SQuAD v2.0 (346), HotPotQA (342), Coral (318), TechQA (295), DC767 (347), and Enterprise-Knowledge RAG (346). Fleiss’ κ was 0.79 with quadratic weighting, and Krippendorff’s α was 0.79 using ordinal distance.
In a 350-task pilot, annotators needed 80% accuracy to advance. Production used three-annotator consensus with continuous quality control. A 2026 ECIR study found stratified sampling reduced human validation by up to 85% for 95% confidence.
Sample-size planning for kappa studies should fit the labels and design you will use, rather than follow one fixed count.
Keep the original reviewer labels beside the adjudicated reference so human disagreement stays visible.
LLM as a Judge Calibrate Kappa 13 Steps 2026: How to Set Up the Harness
Start with an isolated project so a calibration run can be repeated without changing the rest of your development environment. A small, clear setup makes it easier to find why two runs differ. Steps 1-2 are creating the project and installing its dependencies.
A Python virtual environment keeps evaluation packages apart from system Python. A requirements file records the packages used for the run, including evaluation and statistical tools. Separate folders help you track the source data and the results it produced.
- Create a project directory for the calibration code and configuration.
- Create and activate a Python virtual environment for that project.
- Record evaluation and statistical packages in a requirements file.
- Store input data and run results in separate folders.
Keep file names and versions clear enough that another teammate can find the exact test inputs. Store the human reference labels apart from generated verdicts, but join them through a stable example ID. That ID lets you compare each judge result with its matching human label.
Stratified sampling can direct annotation effort toward underrepresented labels and risk slices. A 2026 ECIR study found that it could reduce human validation by up to 85% while retaining 95% confidence in the estimated measure. Use a sampling plan that fits the estimate and the population you intend to describe.
If you plan to run a local judge, check your operating system, memory, model size, and runner compatibility before loading it. Local model setup options include desktop runners such as LM Studio and Ollama. Model licenses may also limit sharing or commercial use.
Save each run’s inputs and outputs together so a later comparison uses the same sampling and dependency setup.
What Should a Reliable Judge Prompt Specify?
A useful rubric tells the judge exactly what evidence earns each verdict and what to do when that evidence is missing. Observable rules help both people and the model apply the same criterion. Define what each label means before scoring examples.
A judge has four components to configure: the evaluation prompt, model selection, variable mapping, and feedback configuration or rubric. Keep their roles clear. The prompt gives the task, variable mapping supplies the right content, and the rubric defines the scoring rule.
Define the Criterion, Labels, and Edge Cases
Describe the criterion through evidence that a reviewer can point to. For factual support, the rule might ask whether each key claim appears in the supplied reference. State what counts as a pass, a fail, or partial coverage if your scale includes that label.
Explain how to handle missing evidence, conflicting sources, and answers that address only part of the request. Add representative positive and negative examples. Boundary examples matter because they reveal whether two reviewers interpret the wording in the same way.
Use a Low-Precision Output the Judge Can Apply
Use binary or low-precision scoring when finer distinctions do not affect the decision. Fine-grained numerical scores can invite inconsistent judgments about small differences. Your output schema should fix the allowed labels and give one decision rule for each.
State exactly which input, answer, and reference material the judge receives. Do not assume that a model sees a source document just because a human reviewer has it. If the judge needs a reference, include it in the supplied context and identify it clearly.
A fixed output schema can ask for a verdict, a short evidence span, and a brief reason. Tell the judge what to return when required information is absent, and how to signal an invalid result. Ask a human reviewer to apply the rubric to a boundary example before you treat its wording as settled.
How Do You Freeze the Judge Before Testing It?
Pin every part of the evaluation call before comparing its verdicts with human labels. One saved configuration makes the test easier to repeat and explain. Keep invalid outputs in the end-to-end denominator.
Save at least six items with each run: model version, full prompt, examples, output schema, decoding configuration, and supplied references. A judge model can change its behavior after an update, even when your prompt stays the same. Record the exact version rather than only the product name.
Build a client that sends examples in a consistent form and stores each response with its example ID. Normalize harmless formatting differences, such as extra spaces, only when they do not change the verdict. Do not rewrite unclear content or repair a malformed label without recording the issue.
Caching lets you reuse a response for the same request, which saves time and supports repeat runs. Retries can help with temporary request failures, but they should not turn a refusal or invalid response into a valid verdict without a record. Count parse failures and refusals as explicit outcomes.
If the judge is stochastic, report single-run agreement separately from any repeated-run or majority-vote setup. A majority vote is a different evaluation configuration, not a replacement for the single-run result. Keep the method visible in your report.
When answers rely on source material, preserve the exact references passed to the judge. Reviewers must be able to check a verdict against the same underlying text, especially in legal or discovery work. Source-document review practices can help teams assess traceability and processing errors.
Record invalid outputs as failures in the acceptance calculation rather than dropping them from the test set.
How Do You Calculate and Read Cohen’s Kappa?
Cohen’s kappa subtracts chance-expected agreement from observed agreement, then scales the remainder by the agreement still available. It is a chance-corrected agreement measure for two raters. Use the same paired examples for the judge verdict and human reference label.
The formula is κ=(P₀−Pₑ)/(1−Pₑ). P₀ is the observed share of matching labels. Pₑ is the share of agreement expected from the raters’ separate label proportions.
Make a table with one rater’s labels in rows and the other rater’s labels in columns. Fill each cell with the number of examples receiving that pair of labels. The diagonal cells show observed matches, while the row and column totals show each rater’s label use.
Calculate P₀ from matching cases divided by all cases. Calculate Pₑ from the raters’ marginal proportions, then place both values in the formula. Scikit-learn provides a Cohen’s kappa function, and equivalent statistical tools can do the same calculation.
The score ranges from -1 to +1. A value of 1 means perfect agreement, 0 means agreement equal to chance expectation, and a negative value means less agreement than expected. The guide to inter-annotator agreement also explains how label prevalence can affect the measure.
| Kappa range | Landis–Koch description | How to use the label |
|---|---|---|
| Above 0.8 to 1 | Almost perfect | Describe the measured agreement |
| Above 0.6 to 0.8 | Substantial | Check the task’s error costs |
| Above 0.4 to 0.6 | Moderate | Inspect disagreements by class |
| Above 0 to 0.4 | Slight to fair | Review prevalence and label use |
| Below 0 | Poor | Investigate the scoring process |
These Landis–Koch descriptions are descriptive labels, not universal release thresholds. A team may need stricter agreement for a high-risk decision than for a low-impact sorting task. Report the selected interpretation beside the confusion pattern and task risk, not as a standalone grade.
What Should You Inspect Beyond the Kappa Point Estimate?
An overall kappa can conceal both unstable estimates and a small group of failures with high operational cost. Check the uncertainty and the errors, not just the single score. Report raw agreement and the label crosstab beside kappa.
A bootstrap interval estimates how much the result might change across repeated samples from your data. Resample the independent unit, not isolated rows, when several examples come from one conversation, customer, or source document. Otherwise, closely related examples can make the interval look more certain than it is.
Quantify Uncertainty Without Breaking the Sampling Unit
For clustered examples, draw whole groups during each bootstrap resample. For example, resample conversations if each conversation supplies several test turns. The method should match how your examples were collected and how they are likely to repeat in use.
Some bootstrap samples may contain only one effective label. Kappa can then be undefined for that sample. State how many samples were undefined and how you handled them; do not quietly omit them from the uncertainty report.
Find Which Labels and Slices Drive Disagreement
Use these checks to find patterns hidden by the overall number:
- Crosstab the judge verdicts against the human reference labels.
- Report percentage agreement beside kappa.
- Separate false passes from false failures.
- Compare agreement by language or workflow slice.
Kappa can be low despite high percentage agreement when one label is common or raters use labels at different rates. The crosstab makes those patterns easier to spot. A high raw match rate may still hide rare but costly misses.
When false passes cause more harm than false failures, inspect each class separately. A judge that misses a dangerous answer should not appear acceptable just because it handles routine cases well. Compare language, product area, and workflow slices when those differences could change the decision.
State how undefined bootstrap samples were handled instead of hiding them from the uncertainty report.
How Can You Test Position and Verbosity Bias?
A judge that changes its preference when the same two answers switch positions may be reacting to layout instead of quality. Order dependence is a warning sign in pairwise evaluation. Treat a verdict flip after an A/B swap as a bias signal.
Position bias, verbosity bias, and self-preference are distinct judge biases. They need separate tests because one check cannot explain every distorted verdict. A judge may favor the first answer, the longer answer, or an answer from its own model family.
Reverse the Presentation Order for Pairwise Judgments
Present each pair once with candidate A first, then again with candidate B first. Keep the wording and content unchanged while swapping only the order. Track whether the verdict stays with the same candidate or flips.
A flip suggests the verdict may depend on position rather than answer quality. Do not count an order-dependent preference as a reliable win for either candidate. A 2025 study of 15 LLM judges across MT-Bench and DevBench found position bias varied by judge and task across more than 150,000 evaluation instances.
The study’s results also show why a single overall rate can miss differences between tasks. Quality gaps between answers affected position consistency. The 2025 position-bias study describes measures for position consistency, preference fairness, and repetition stability.
Test Whether Longer Answers Win Without Better Evidence
Build length-trap examples that pair a concise, correct answer with a longer answer that is wrong or padded. Ask humans to judge them under the same criterion. Then compare whether the judge favors length more often than human reviewers do.
Use examples from different model families when feasible to test self-preference. A judge may favor familiar wording or outputs from its own family, even when another answer better meets the rubric. Keep each bias measure distinct from the general agreement score.
Keep a separate record of order flips and length-trap results so they do not disappear inside overall kappa.
How Do You Compare Judges and Set a Ship Gate?
Choose the judge that aligns best with the human decision you need, not the model that returns the most confident verdicts. Compare on identical cases so changes reflect the candidate judge, not a different sample. Report class errors, invalid outputs, and stability with kappa.
Run each candidate on the same labeled examples with the same task and output rules. Compare its kappa, errors by class, invalid-output rate, and stability across repeated runs if the judge is stochastic. Confidence in a verdict is not a measure of agreement.
| Benchmark stage | Rule | Classification |
|---|---|---|
| 54 judges, 1,994 samples | Compare with human annotators | 27 models met Tier 1 criteria |
| Correlation screen | Require r≥0.80 | Keep models passing stage one |
| Kappa z-score screen | |z|<1 or z>1 | Human-like or super-consistent |
The Judge’s Verdict Benchmark compared 54 LLM judges with human annotators on 1,994 samples. Of the 27 models meeting Tier 1 criteria, 23 showed human-like agreement patterns and four showed super-consistent patterns. Its two-stage screen first required correlation r≥0.80, then applied Cohen’s kappa z-scores.
The benchmark illustrates a screening design, not a universal ship rule. A human-like agreement pattern does not by itself mean a judge fits your task. The human-human agreement on your task and the harm from each error should shape your acceptance gate.
A 350-task pilot used an 80% annotator accuracy threshold for production advancement. That number belongs to its pilot design, not every team’s release process. Set your own threshold around the intended use and the cost of a false pass.
Write the acceptance threshold and the false-pass limit before you choose a production judge.
How Do You Repair Disagreements Without Overfitting?
Start each repair with the reason for a disagreement, not a general rewrite of the whole prompt. Target the cause so a change fixes a known gap rather than masking unrelated errors. Keep development cases separate from untouched validation cases.
Group disagreements before changing the rubric. A useful review names what the judge missed, what evidence it used, and whether the same pattern appears in several examples. Human corrections can become ground truth for measuring and improving alignment.
Use Disagreement Patterns to Refine the Rubric
Match each cause to a focused repair:
- Ambiguous criteria need sharper wording tied to observable evidence.
- Missing evidence needs a rule for absent or conflicting references.
- Compound criteria need separate checks for each required condition.
- Missing boundary examples need a clear positive and negative pair.
Change one cause at a time when possible. If you rewrite several rules together, a better score will not show which change helped. Keep the human correction and the judge’s original verdict so the reason for each repair remains clear.
Validate Changes on Untouched Examples
Reserve an untouched set for final validation. Repeated prompt tuning against that set turns it into development data, so its score no longer gives an independent check. Collect fresh expert labels for later rounds when you need new validation examples.
Reviewers can correct sampled judge scores, and those corrections can serve as ground truth for alignment checks over time. Use that record to see whether the judge improves on the errors that matter. Do not count a correction as a successful fix until the changed judge passes on cases it did not see during tuning.
After the prompt passes on untouched cases, preserve that version and its results as the next comparison baseline.
How Do You Monitor a Calibrated Judge in Production?
A passing calibration result describes the judge on the cases it was tested against, not every future production input. Fresh human checks help reveal when live traffic changes the result. Start by reviewing 5-20% of traffic, plus every error and low-score output.
Compare sampled production verdicts with fresh human reference labels. A monthly recalibration is a suggested default, with earlier checks after a judge-model swap or prompt rewrite. Recheck sooner when you add a tool, change policy, or serve a new user segment.
Track score-distribution changes separately from changes in agreement with humans. A shift in score rates can signal a change in incoming cases, but it does not prove the judge has become less aligned. Fresh labels show whether its decisions still match human review.
Errors and low scores deserve complete review because they may point to risk that a random sample misses. Keep an eye on important workflow and language groups, especially when their traffic mix shifts. A judge can keep its overall score while struggling with a smaller group.
Do not treat a launch result as permanent proof; new inputs and system changes can alter judge performance.
Preserve the prompt, model version, dataset version, and agreement result together so a later drop can be traced to a specific change.
Conclusion
Trust a judge only after its verdicts have been checked against the human decisions that define success for your task. Kappa is useful evidence when you read it with human labels, costly error patterns, and the intended use. A single score cannot tell your team whether a judge is safe for every decision.
Before the judge controls a release decision, establish a baseline on real examples. Keep the reviewer labels and the judge configuration with the result, so later changes have a fair comparison point.
Choose one representative batch, have qualified reviewers label it independently, and calculate the first agreement baseline.
FAQ
How many examples should I label before calculating Cohen’s kappa?
A sample of 50-100 subjects is a common starting point, but the needed size depends on your study design. More categories and rare labels often require more examples to estimate agreement well. Planning also depends on expected kappa, statistical power, and alpha. Include enough cases from important risk groups to make the results useful for your real decision.
Can Cohen’s kappa be used when three or more people label the same examples?
Fleiss’ kappa is one option when more than two raters label nominal categories. It may not fit every design, especially when labels are ordered or reviewers leave some examples blank. Consider whether ordinal distance matters and how missing labels affect the comparison. Choose a measure that matches the number of raters and the type of label.
Does a high kappa prove that an LLM judge is accurate?
No, a high kappa shows agreement between the judge and the human labels, not accuracy by itself. Human reviewers can share the same mistake, or the reference labels may not reflect an authoritative standard. Compare the labels with a trusted source when one exists. Inspect consequential errors, especially false passes that could lead to harm.
Why can kappa be low when reviewers agree on most examples?
Kappa can be low when one category dominates or raters use labels at different rates. Those patterns affect the amount of agreement expected by chance, even when raw agreement seems high. Report the label crosstab and percentage agreement with kappa. Then check whether one rater overuses a category or misses a less common class.
Should I use weighted kappa for pass/fail labels?
No, binary pass/fail labels are usually nominal, so unweighted kappa is the typical choice. Weighted kappa is for categories with a real order, where some disagreements should count as closer than others. Do not add weights just to make a score look more favorable. State the weighting scheme when you report an ordinal result.
How often should I recalibrate an LLM judge?
Monthly is a suggested default for checking agreement against fresh human labels. Recalibrate sooner after a judge-model swap, prompt rewrite, new tool, policy change, or shift in user groups. A stable score distribution does not prove that agreement remains steady. Compare sampled verdicts with human decisions to check alignment directly.
Can I tune a judge prompt and measure its final score on the same examples?
No, use separate development and held-out sets for a fair final check. Repeated tuning against the held-out examples turns them into development data, so their score no longer gives an independent test. Keep the untouched cases out of prompt edits. For later rounds, collect fresh examples and expert labels to test whether the repair generalizes.
