LLM evaluation for a product team is a fixed set of real inputs, each with an expected answer or a grading rubric, that runs against every prompt, model, and retrieval change before users see it. Teams that ship AI features well build that set before the first prompt, because it’s the only reliable way to tell whether the next change made the feature better or just different.

Why the test set comes before the prompt

Public benchmarks rank models on generic tasks. A product team needs something narrower that no leaderboard measures: whether its own feature answers its users’ requests correctly and within budget.

Without a test set, prompt work tends to run on impressions. Someone edits the system prompt, tries three questions, and ships. Yet small, meaning-preserving changes sometimes move results a lot: Sclar et al. measured differences of up to 76 accuracy points from prompt formatting alone, in few-shot settings with LLaMA-2-13B.

Writing the set first also forces product decisions that otherwise get made by accident: what the assistant says when the answer isn’t in the documents, which requests are out of scope, and when it hands off to a person.

The criteria will shift once grading starts. Shankar et al. call this criteria drift: people need criteria to grade outputs, but grading outputs is what helps them define the criteria. That argues for starting early with a rough rubric.

Where the examples come from

The best examples are requests users have already sent. We pull from four sources.

Real user requests. Search queries, chat transcripts, and form submissions. For a new feature, the queries people type into the current search box are a fair proxy.

Support tickets. They capture questions hard enough to need a person, and resolved ones come with a human answer.

Edge cases. We write these on purpose: empty or very long inputs, mixed languages, typos, ambiguous or out-of-scope requests, and questions whose answer isn’t in the data. That last case matters most for retrieval features, where the failure is a confident, invented answer.

Adversarial cases. Prompt injection is LLM01 in the OWASP Top 10 for LLM Applications, which separates direct injection, typed by the user, from indirect injection, hidden in a website or file the model reads. We add both, plus attempts to extract the system prompt or another user’s data. OWASP’s mitigation list includes adversarial testing that treats the model as an untrusted user.

Personal data comes out before production text lands in the set, because eval sets get copied into notebooks and CI logs. Our article on AI gateways and PII masking shows where masking fits.

How many examples are enough to start

We start with 50 to 100 examples covering every intent in scope and every known failure mode. That catches a change that breaks a whole category, but not small improvements.

For a pass/fail metric, the standard error of the pass rate is the square root of p(1-p)/n, and a 95% confidence interval is the pass rate plus or minus 1.96 standard errors, as laid out in Miller’s “Adding Error Bars to Evals”.

At 100 examples and an 80% pass rate, that interval is about plus or minus 8 points. A prompt that lifts the score to 83% on that set hasn’t shown anything yet, and quadrupling the set only halves the interval.

Three practices from the same paper get more out of a small set and keep its error bars honest:

  • Paired comparisons. Scoring old and new prompts on the same examples and comparing per-example differences cuts variance, as long as both versions agree on which examples are hard.
  • Clustered errors. Ten examples from one conversation aren’t ten independent data points. The paper recommends clustered standard errors. Keeping one example per source is a simpler way around the problem.
  • Power analysis. Before a model swap, it shows whether the set is large enough to detect the difference that matters.

From there, the set grows: every production failure becomes a new example. A held-out slice that nobody tunes prompts against separates real improvement from a prompt that memorized the test.

Expected answers and rubrics

Every example needs a definition of “correct” that two people would apply the same way. Three label types cover most features:

  • Exact answer for classification, routing, and yes/no decisions.
  • Structured answer for extraction, tool calls, and generated SQL: the expected JSON fields or query results (comparing results, not query text).
  • Rubric for open-ended text: what the answer must and must not contain.

We write rubric items as binary checks, such as “states that duplicate charges are refunded in full,” rather than 1-to-10 scores. Each check then asks graders one yes/no question.

Agreement between graders is the real constraint. In the Judging LLM-as-a-Judge study, expert human raters agreed with each other 81% of the time on first-turn pairwise comparisons without ties. That’s why the best labels come from the people who own the answer, and a double-labeled sample shows which criteria are vague.

One row, pretty-printed (the file stores one row per line as JSONL):

{
  "id": "billing-017",
  "source": "support_ticket",
  "input": "I was charged twice for the annual plan in March. Can I get one charge back?",
  "context_docs": ["billing-policy-v3#duplicate-charges"],
  "expected": {
    "type": "rubric",
    "must_include": ["duplicate charges are refunded in full", "link to the refund form"],
    "must_not_include": ["a specific refund date", "another customer's data"],
    "escalate": false
  },
  "tags": ["billing", "high-impact"],
  "labeled_by": "support-lead"
}

Tags allow a zero-regression rule for high-impact categories even when the overall score goes up.

Metrics: exact match, human grading, and LLM-as-a-judge

We pick the cheapest method that measures what matters.

Method Use it for Cost per run Main risk
Exact or structured match Classification, extraction, tool calls, SQL, and JSON Near zero Fails correct answers phrased differently
Human rubric grading Early rounds and judge calibration Slow and expensive Raters disagree, and criteria drift
LLM-as-a-judge Open-ended answers on every regression run One call per criterion (two with swapped answer order) Position, verbosity, and self-preference bias

LLM-as-a-judge is usually the first method teams reach for, and it’s the one that needs the most checking. Zheng et al. found that GPT-4 as a judge agreed with expert humans 85% of the time on comparisons without ties, slightly above the rate humans reached with each other. The same paper documents where judges fail:

  • Position bias. Most judges favored whichever answer came first. In a stress test with two near-identical answers per question, GPT-4 with the default prompt gave the same verdict after the answers were swapped in 65% of cases, and GPT-3.5 and Claude-v1 did worse. On real model pairs, the bias shrank as the quality gap grew and almost disappeared when one answer was clearly better.
  • Verbosity bias. When a numbered list was padded with rephrased copies of its own items, Claude-v1 and GPT-3.5 preferred the padded answer in 21 of 23 cases. GPT-4 resisted the trick far better.
  • Weak math grading. GPT-4 accepted an incorrect math answer as correct in 14 of 20 judgments with the default prompt. A reference answer in the prompt fixed most of those failures.
  • Self-enhancement bias. GPT-4 and Claude-v1 gave their own answers higher win rates than human raters did, but the authors state that their data cannot determine whether the bias is real. A later study by Panickssery et al. found that models such as GPT-4 and Llama 2 recognize their own outputs with non-trivial accuracy, and that self-recognition correlated linearly with self-preference in fine-tuning experiments.

These are 2023 models, so the numbers describe failure modes worth testing for, not properties of today’s judges. The mitigations are cheap. The paper’s conservative fix runs each comparison twice with the answer order swapped and counts a win only when both orders agree. We also give the judge a reference answer or rubric, grade one binary criterion per call, and use a judge from a different model family than the one being graded.

The judge itself gets checked against a hand-graded sample, the same loop the EvalGen tool from Shankar et al. is built around.

Regression runs on every prompt and model change

We treat the prompt, model version, retrieval settings, tool schemas, and sampling parameters as one versioned config. Any change to it triggers a run.

That includes changes made by the model provider. Chen, Zaharia, and Zou compared the March and June 2023 versions of GPT-4 and found its accuracy at telling prime from composite numbers fell from 84% to 51%, while some other tasks improved. Teams that handle this well pin model versions where the API allows it and rerun the full set before moving to a new one.

Deterministic checks take seconds, so they run on every pull request, and the judged pass over the full set runs nightly and before each release. The useful report lists the examples that flipped from pass to fail, old and new answers side by side. A higher total with new failures in a high-impact tag is still a regression.

Release gates and signals after launch

Cost and latency per request are release gates too. They belong in the same report as quality because prompt changes trade them off: an extra retrieval pass can raise the pass rate and break the budget. We log tokens, cost, total latency, and time to first token for every example and gate on the 95th percentile, not the average.

The NIST AI Risk Management Framework calls for the same discipline in its Measure function: AI systems tested before deployment and regularly in operation, with measures of uncertainty and documented results. For one feature, an eval set plus a release log is that record.

After launch, three online signals are worth instrumenting:

  • Thumbs up and down. Only the users who choose to click are represented. They work for picking sessions to review, not as a score.
  • Edits. For copilots, how much users change a suggestion before accepting it is a direct quality signal.
  • Escalations. Handoffs to a person, reworded repeat questions, and abandoned sessions. In search, a reformulated query right after an answer often signals a miss.

A weekly review of flagged sessions turns those failures into labeled examples in the set.

Release checklist

Before release, we recommend confirming each line:

  • The set covers every intent in scope plus edge and adversarial cases, with an untouched held-out slice.
  • Pass rate meets a threshold agreed before the run, reported with a 95% confidence interval.
  • No pass-to-fail flips on high-impact tags.
  • Prompt-injection cases cause no data leaks and no tool calls outside the allowed scope.
  • Judge agreement with human labels rechecked on a fresh sample.
  • Cost per request and p95 latency, including time to first token, within budget.
  • Model version pinned, with prompt, model, and retrieval settings versioned together.
  • Previous config restorable by feature flag.

Where to start

The order we recommend:

  1. A first batch of 50 real requests from search logs, chat transcripts, or resolved tickets, with personal data stripped.
  2. Edge and adversarial cases on top, including an indirect prompt injection if the feature reads documents or web pages.
  3. A label on every example, written by the person who owns the answer.
  4. Only then the first prompt, a full run, and a read of every failure before anything changes.
  5. Deterministic checks in CI and a nightly judged run.
  6. Release thresholds from the checklist above, agreed before the release candidate exists.

Our AI features engagements follow the same order: the first week goes to the evaluation set, built from your real requests, so every later prompt, model, and retrieval change can be measured against it before release.