LLM Evaluation With Golden Datasets: The Only Way to Ship With Confidence

A golden dataset is a versioned file — YAML, JSONL, or a database table — where each row is: an input, one or more assertions about what a correct output looks like, and metadata (category, priority, source). Nothing more.

- id: refund_simple
  category: happy_path
  input: 'Can I get a refund for order 12345 shipped last week?'
  assertions:
    - kind: mentions_order
      value: '12345'
    - kind: intent
      value: 'refund_request'
    - kind: no_hallucinated_policy
      forbidden_phrases: ['30-day guarantee', 'lifetime warranty']

- id: refund_expired_ambiguous
  category: edge_case
  input: 'This is old — bought it a year ago, hardly used it, want money back'
  assertions:
    - kind: escalate_to_human
      required: true

The instinct is to grade every output with GPT-5. It's slow, expensive, and noisy. Ninety percent of assertions can be checked with deterministic rules:

• Exact match / substring — 'response contains order ID'.
• JSON schema — 'response parses to {intent: string, confidence: number}'.
• Regex — 'response contains a valid ISO date'.
• Forbidden list — 'response does not mention competitor names'.
• Length bounds — 'response is 50-500 tokens'.

Use LLM-as-judge only for genuinely subjective assertions — tone, factuality against a source doc, semantic equivalence. When you do, pin the judge model (don't 'use the latest') and periodically calibrate against human labels.

Don't wait for a labeling budget. Start with 20 examples you write yourself:

1. 5 easy happy-path cases (the demos you'd show your CEO).
2. 5 edge cases you already know are tricky.
3. 5 real user inputs from logs or beta.
4. 3 adversarial cases (prompt injection attempts, weird formatting, empty inputs).
5. 2 'never do this' cases (things the last version regressed on).

Twenty examples take an afternoon. They will catch 80% of the regressions the next big prompt change would have shipped. Everything after this is refinement.

# .github/workflows/llm-evals.yml
on: pull_request
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: python -m evals.run --dataset golden.yaml --out results.json
      - run: python -m evals.compare --baseline main --results results.json
        # fails if pass rate drops more than 2% on any category
      - uses: actions/upload-artifact@v4
        with: { name: eval-report, path: results.json }

On every PR that touches a prompt, a model choice, or the retrieval stack, the workflow runs the golden set, compares to the baseline on main, and posts a comment. Regressions past a threshold block the merge.

Every production incident where the LLM did something wrong becomes a new row in the golden set — with the fix as an assertion. This is the single practice that turns a 'nice to have' eval suite into an ever-improving safety net.

Set a runbook step: 'after every LLM incident, add the offending input as a golden example within 48 hours'. Assign the on-call. In a year you'll have 200-500 examples covering exactly the failure modes your users actually hit — infinitely more valuable than a benchmark someone else designed.

Reserve LLM judges for the assertions rules can't check:

• Factuality against a specific source document ('does this response state anything not in the doc?').
• Semantic equivalence for open-ended responses ('does this convey the same answer as the gold reference?').
• Tone and register ('is this professional but warm?').
• Multi-turn coherence.

Use structured judges: prompt the judge for a JSON verdict with a rubric, temperature 0. Calibrate weekly by having a human label 30 random cases and comparing agreement. When it drops below 85%, retune the judge prompt or swap the model.