What separates a draft that a team ships from one that quietly gets rewritten three times? Usually not the model’s marketing page. It is whether anyone measured the output against a real standard before it went out the door. That question gets harder, not easier, as model families grow into tiers with different speeds, prices, and strengths. The Opus, Sonnet, and Haiku naming pattern has become shorthand for a simple idea: a bigger, slower model for the hardest work, a balanced middle, and a fast, cheap tier for volume. Whether a future generation ships under a 5.5 label changes the calendar, not the method. The method is what this article covers. (Large language model)
Why tier names tell you less than you think
A tier name signals a position in a lineup, not a guarantee about any single task. The expensive tier may handle long, multi-step reasoning better, while the fast tier may match it on short, well-specified work. That gap is exactly why “which model is best” is the wrong question. The useful question is “which model clears my quality bar for this task at the lowest cost and latency.”
Model routing is the practical answer. Route each task type to the smallest tier that passes your own evaluation. Teams that skip this step either overspend on the top tier for simple work or underdeliver on hard work by defaulting to the cheapest option. Neither failure shows up in a benchmark table. Both show up in review cycles.
There is a second trap: treating a family’s internal naming as stable. Providers rename, deprecate, and re-snapshot models constantly. Any evaluation that does not record the exact model identifier and date is not reproducible six months later.
The four layers of a quality evaluation
Good evaluation stacks four layers. Each answers a different question, and skipping one leaves a blind spot.
| Layer | What it answers | Typical method | Main weakness |
|---|---|---|---|
| Automated metrics | Does output match a reference? | Exact match, BLEU, ROUGE, similarity scores | Rewards surface overlap, not usefulness |
| Rubric scoring | Is the output good by stated criteria? | Human or model graders against a rubric | Grader bias, vague criteria |
| Pairwise preference | Which of two outputs do people prefer? | Side-by-side human votes, Elo-style ranking | Measures preference, not correctness |
| Task outcome | Did the output work in production? | Pass/fail on real downstream tasks | Expensive to build, slow to run |
Most teams start at the top and stop there. The durable signal lives at the bottom. A model that wins on a similarity metric but fails a downstream task has told you nothing useful.
Automated metrics: fast, cheap, and easy to game
Reference-based metrics are fine for narrow jobs like translation consistency or structured extraction, where a correct answer exists. They fall apart on open-ended writing, where many good answers exist and none match a reference string. Use them as a smoke test, not a verdict.
Rubric scoring: the workhorse
A rubric turns “good” into checkable criteria. For a support reply, that might be: states the fix in the first two sentences, cites the relevant policy, avoids promises about refunds, stays under 150 words. For a code explanation, it might be: correct, runnable, explains the tradeoff, flags edge cases.
The rubric is where most quality problems actually live. If two reviewers disagree about whether an output passed, the rubric is too vague, not the model too weak. Write criteria that a new team member could apply without asking a question.
Using a model as a grader, carefully
LLM-as-judge means asking a strong model to score outputs against a rubric. It is cheap, fast, and scales to thousands of samples. It also carries documented biases. Research on judging with MT-Bench and Chatbot Arena identified position bias, where a grader favors whichever answer appears first, verbosity bias, where longer answers score higher regardless of quality, and self-preference, where a model favors text it produced. The paper on LLM-as-a-judge biases and the MT-Bench evaluation method lays out these failure modes and mitigation strategies.
Practical fixes that work in real pipelines:
- Randomize answer order on every comparison so position bias averages out.
- Cap or normalize length before scoring, or add an explicit “penalize padding” instruction.
- Use a different model family as the grader than the one being graded.
- Spot-check 5 to 10 percent of automated scores against human labels every cycle.
That last step is the one teams skip. It is also the one that catches a grader drifting away from human judgment after a prompt change.
Pairwise comparison and the preference trap
Pairwise comparison, where people pick the better of two outputs, underlies the public leaderboards that many buyers cite. The Chatbot Arena pairwise human preference methodology is a well-known example: voters see two anonymous responses and choose a winner, and the votes feed a ranking system.
Preference data is genuinely useful. It captures things rubrics miss, like tone and readability. But preference is not correctness. A confident, well-formatted wrong answer often beats a hedged, accurate one in a blind vote. Treat leaderboard position as one input, never as a procurement decision.
Building a task-specific evaluation set
Generic benchmarks have a contamination problem: models may have seen test items during training, which inflates scores in ways nobody can fully audit. This is a known threat to validity across the field. The fix is a private evaluation set drawn from your own work.
A workable process:
- Collect 50 to 200 real tasks per workflow, pulled from past tickets, drafts, or code reviews.
- Write the ideal output or a pass/fail criterion for each, using your rubric.
- Split into a development set for prompt tuning and a held-out set you touch only for final scoring.
- Score every candidate tier on the same held-out set, same prompts, same temperature.
- Record model identifiers, dates, and settings alongside every score.
Fifty well-chosen tasks beat a thousand scraped ones. The point is representativeness, not volume. If your real work is 70 percent short classification and 30 percent long synthesis, your evaluation set should mirror that split, or you will optimize for the wrong thing.
For readers building this out, a broader comparison of how different model families behave on draft quality can inform which tasks to include in the set, and a hands-on practical Claude writing workflow shows how rubric criteria translate into repeatable prompts.
Tier routing in practice: a worked example
Suppose a content team handles three task types: short product descriptions, long research briefs, and code sample generation. Here is how routing might look after a two-week evaluation.
| Task type | Volume | Quality bar | Likely tier | Why |
|---|---|---|---|---|
| Product descriptions | High | Pass/fail rubric, 8 of 10 criteria | Fast tier | Short, well-specified, easy to verify |
| Research briefs | Medium | Human review required | Top tier | Long context, synthesis, fewer errors tolerated |
| Code samples | Medium | Must run without edits | Middle tier, escalate on failure | Balanced cost, testable output |
The escalation rule matters more than the initial choice. Send failures up one tier, re-score, and log which failures the top tier actually fixed. Over a month, that log tells you whether the expensive tier earns its price on this workflow. Often it does for one task type and not the others.
If the team is also weighing vendors, a side-by-side look at Claude versus ChatGPT draft quality across workflows helps frame what to test, and a wider AI model comparison across current flagship tiers shows how the same evaluation framework applies when the lineup changes.
Where evaluations quietly fail
Five mistakes show up again and again.
Judging on cherry-picked examples. Someone runs ten prompts, gets a great result, and declares the tier switch a win. Ten prompts measure luck. Use the held-out set.
Ignoring refusal behavior. A model that refuses a legitimate request is failing, and a model that complies with a harmful one is failing differently. Score false refusals and unsafe compliance as separate categories. Frameworks like the NIST AI Risk Management Framework treat this kind of measurement as part of risk management, not an afterthought.
Forgetting latency and cost. A tier that scores two points higher but takes four times as long may lose on a chat product. Put latency and cost per task in the same table as quality scores.
Changing prompts mid-evaluation. If the prompt changes between scoring the middle tier and the top tier, the comparison is void. Freeze prompts, then score.
Not documenting versions. Model snapshots get retired. Without recorded identifiers and dates, nobody can reproduce the result, and the evaluation becomes folklore.
Running the evaluation on a schedule
Quality drifts. Prompts get edited, providers update models, and the work itself changes. A quarterly re-run of the held-out set catches drift before customers do. Between re-runs, track a small live sample: a few dozen outputs per week, scored by a human reviewer against the rubric. (NIST Artificial Intelligence overview)
A useful cadence looks like this: weekly spot checks, monthly grader calibration against human labels, quarterly full re-score across tiers, and an immediate re-score whenever a model snapshot changes. For developer-facing workflows, pairing this with a practical guide to using Claude Code in development helps translate evaluation findings into concrete tooling decisions.
Conclusion
The tier names will keep changing, and a future generation may or may not carry a 5.5 label. What will not change is the underlying test: a private evaluation set drawn from real work, a rubric specific enough that two reviewers agree, a grader checked against human judgment, and a routing rule that sends each task to the cheapest tier that clears the bar. Teams that build that apparatus now can swap models in an afternoon. Teams that do not will keep arguing about which model is best, using ten prompts and a hunch. The first group ships faster, and it is not close.
FAQ
What is the difference between Opus, Sonnet, and Haiku tiers?
They represent positions in a model family, typically trading capability against speed and cost. The top tier handles the hardest reasoning, the middle tier balances both, and the fast tier serves high-volume, well-specified work. The names describe a lineup, not a guarantee about any specific task, so each tier still needs its own evaluation on your work.
Can I just use the top tier for everything and skip evaluation?
You can, but you will overpay and often wait longer for the same result. Many tasks, like short classification or templated writing, pass a quality bar on the fast tier. Without an evaluation, you have no way to know which tasks those are, so the top-tier default becomes a permanent cost with no measured benefit.
How many test tasks do I need for a reliable evaluation?
Fifty to two hundred real tasks per workflow is usually enough to make a routing decision. What matters more than count is representativeness: the set should mirror the mix of task types and difficulty you actually see. A thousand easy tasks will not tell you how a tier handles the hard 10 percent.
Is a model grader trustworthy enough to replace human review?
Not as a replacement, but as a scaler. Model graders carry documented biases toward position, verbosity, and their own output. Randomize order, control for length, use a different model family as the grader, and check a sample against human labels every cycle. With those controls, automated scoring handles volume while humans handle calibration.
How do I compare tiers fairly when prompts keep changing?
Freeze everything: prompts, temperature, evaluation set, and scoring rubric. Score all tiers on the identical held-out set in the same session, and record exact model identifiers and dates. If a prompt must change, re-score every tier from scratch rather than comparing a new prompt against an old score.
What should I do when a model version is retired?
Re-run the held-out set on the replacement version and compare against the archived scores for the retired one. This is why recording model identifiers and dates matters. Treat a version retirement as a trigger for a full re-score, not a routine update, because behavior can shift in ways the release notes do not mention.
