← Back
evaluation incident seeded

A contaminated eval suite reports 100% while production misses novel phrasing

June 24, 2026

Symptom

CI shows a perfect 100% pass rate across 312 golden eval cases for four straight releases, yet support review finds that roughly 40% of real customer queries either trigger the wrong workflow or return an answer that ignores the user's actual constraint. The failures cluster around paraphrases, abbreviations, and multi-intent requests that never appear in the dashboard.

Root Cause

The eval fixtures were generated by copying the same prompt bank used during development into `evals/golden/*.jsonl`, then freezing expected outputs after prompt tuning had already been done against those exact inputs. The model and routing prompts were repeatedly adjusted until those 312 phrasings passed, so CI measured memorized development examples rather than holdout behavior. Production traffic used different wording, extra constraints, and misspellings that were absent from the harness, leaving the failing 40% unmeasured.

Diagnosis Steps

  1. Diff `prompts/dev_bank/*.yaml` against `evals/golden/*.jsonl` and count exact or near-exact duplicated user messages.
  2. Export 200 recent failed production queries from the support triage queue and label them with the same grader used in CI.
  3. Run `npx promptfoo eval --config promptfooconfig.yaml --filter-pattern realworld-smoke` and compare pass rate against the golden suite.
  4. Cluster failures by intent and wording pattern to identify missing paraphrases, abbreviations, and multi-constraint requests.
  5. Check the CI gate config for `passRateThreshold: 1.0` and confirm it only references the contaminated golden dataset.
  6. Create a clean holdout split whose examples were sampled after the latest prompt iteration and were never visible during development.

Fix

Rebuilt the eval harness into three named splits: `dev-regression` for known examples, `holdout-realworld` from 500 anonymized production queries sampled after the last prompt edit, and `mutation-paraphrase` generated from production intents but reviewed manually before use. CI still requires 100% on `dev-regression`, but release blocking now keys on at least 92% task success and no severity-1 misses on `holdout-realworld`. The first clean run exposed a 61.8% pass rate, and the routed-query prompt was fixed against the new failure clusters rather than the copied prompt bank.

Prevention

Keep development prompts and release-gating evals in separate datasets, record fixture provenance, and require every new prompt iteration to be scored on a holdout set that was created after the iteration began.

Stack

promptfoo 0.93.0 GitHub Actions Node 20 GPT-4o mini JSONL eval fixtures

Tags

evaluation evals contamination ci prompting

Date

June 24, 2026

Cost Impact

Four releases shipped with a false green gate, creating about 70 extra support escalations before the holdout eval was added.

For a month, every pull request showed the same line:

promptfoo: 312/312 checks passed, passRate=100.0%

That looked strong until the support queue was sampled. Out of 200 recent production queries, 79 failed the same task-success rubric. Customers wrote “refund last invoice but keep seats active”, “pause renewal for EMEA only”, or “why did ACH retry twice”; the golden suite mostly contained the tidy development prompts engineers had been using in demos. Same intents, different language, very different result.

The duplicate path was concrete. prompts/dev_bank/billing.yaml had been copied into evals/golden/billing.jsonl during the first evaluation push. Then the routing prompt was tuned in small commits until all 312 examples passed. Nobody meant to train on the test. The harness just gave the team a comfortable number because the release gate was pointed at examples that had already shaped the system.

The replacement kept that set, but renamed it honestly: regression coverage. A second split came from anonymized production traffic sampled after the prompt work, with fixture metadata recording source=prod_sample, sampled_at, and the owning intent. That split was the one GitHub Actions used for release blocking. The next green build meant the model survived new wording, not just yesterday’s rehearsal.