"It Works on My Prompt" Is Not a Test Plan: How to Eval LLM Features Before Launch

How to build an eval suite a product team can actually run: golden datasets, LLM-as-judge with calibration, regression gates in CI. What to measure, what to ignore, and when you have enough signal to ship.

You tested your AI feature five times, it answered reasonably all five, and now it feels done. That's the single most common mistake we see before a launch. Five manual runs cover nothing: not the user who types in all caps, not the one who pastes a 12-page PDF, not the Spanglish prompt, not the day the model decides to hallucinate an invoice number. An LLM is non-deterministic; two runs of the same input can diverge. Your gut sense that "it answered fine" is a sample of size one drawn from a distribution you never saw.

An eval suite fixes this by turning "seems to work" into a number you can watch across versions. You don't need a research team or expensive tooling. You need a representative dataset, honest metrics, and a gate that runs itself in CI. Here's how to build one a product team can actually maintain without it rotting three weeks in.

The golden dataset is 80% of the work

A golden dataset is a set of cases with an input and an expected output (or an acceptance criterion) that represents real usage. Your eval is never better than its dataset. And the best dataset isn't invented: you pull it from real traffic or closed-beta logs, anonymized.

• Start with 50-100 well-chosen cases, not 5,000 generic ones. Coverage beats volume.
• Stratify on purpose: happy paths, edge cases, garbage input, prompt-injection attempts, mixed languages, empty or oversized inputs.
• Keep your bugs. Every time something fails in production, that case enters the dataset with its correct output. A regression won't ship twice.
• Version the dataset in the repo (JSONL) and treat it like code: PRs, review, history.

Measure the task, not abstract "quality"

"Is the answer good?" isn't measurable. Break it into dimensions that depend on your use case. A data extractor and a support chatbot don't get judged by the same ruler.

• Extraction/classification: exact match, per-field F1, precision and recall. This is deterministic and cheap; use it wherever you can.
• Format: does the JSON parse? Does it respect the schema? A Pydantic or Zod check catches 90% of integration incidents.
• Faithfulness (RAG): does the answer rest on the retrieved context or invent it? Measure groundedness and whether it cited the right source.
• Safety: system-prompt leak rate, out-of-scope answers, PII exposure.
• Cost and latency: tokens per response and p95 time. A quality gain that triples cost may not be worth it.

LLM-as-judge: useful, but calibrate it or it lies

For subjective things (tone, helpfulness, coherence), using another model as a judge scales far better than hand-review. But an uncalibrated judge has known biases: it prefers longer answers, favors the first option in comparisons, and is lenient toward text that sounds confident even when it's wrong. Trust it blindly and you're measuring the judge's taste, not your product.

Calibration is simple and non-negotiable: hand-label 50-100 cases, run the judge on the same ones, and measure agreement (Cohen's kappa or correlation). If the judge doesn't match humans, fix the rubric before you trust it. Use a short, anchored scale (pass/fail, or 1-3 with each level explicitly defined), not a fuzzy 1-10. And when comparing two versions, alternate the order to cancel the position bias.

The regression gate lives in CI or it doesn't exist

An eval you run by hand when you remember is useless. The value shows up when it runs itself on every change to a prompt, model, or RAG chain, and blocks the merge if the score drops. Set a threshold; don't chase 100%.

• Define a floor per metric (e.g. extraction F1 ≥ 0.90, groundedness ≥ 0.95) and fail the PR if it dips.
• Run the deterministic subset on every commit; run the LLM-judge subset nightly or pre-release, since it costs money and time.
• Pin temperature to 0 for anything exactly checkable; that cuts run-to-run noise.
• Track the score over time. A slow 3-point drop per release is a boiling frog nobody notices without a chart.

When you don't need any of this

Honestly: standing up a calibrated LLM-as-judge for an internal prototype three people use is over-engineering. The suite's effort should scale with the feature's blast radius.

• Prototype or internal demo: a handful of cases in a notebook is enough. Don't halt the sprint for this.
• Feature touching money, health, legal, or personal data: strict evals, hard gate, no exception.
• High user volume: even if a single error is cheap, at scale it gets expensive; invest in the suite.
• If a human always reviews the output before acting, you can be looser on the automated gate and measure in production.

When you have enough signal to ship

There's no perfect score, only a score good enough for the risk you're taking. The question isn't "does it always get it right?" but "is the worst plausible case tolerable, and would I catch it if it got worse?" A human-in-the-loop feature can ship at 90%; one acting alone on money might need 99% plus a plan for the remaining 1%.

Before you say yes, confirm three things: the score has stabilized (adding more cases to the dataset no longer moves it), you have a gate that alerts if it drops post-launch, and you know what happens when the model fails, not just when it succeeds. A fallback, an honest error message, or a human escalation path is worth more than one extra point of accuracy.

The criterion in one sentence

If you can't put a number on "did it get better or worse?" between two versions, you're not evaluating yet, you're guessing. Before shipping any AI feature, ask: do I have a dataset that represents real usage, am I measuring the dimensions that actually matter for this task, does my automated judge agree with humans, and is there a gate that wakes me if the score drops tomorrow? If all four are yes, you have signal. If any is no, that's what you build next, not the launch.