Evaluating LLMs and Classical ML: A Practical Metrics Guide

The wrong metric is how good teams ship subtly bad models — a fraud model that scores 99.4% accuracy while missing 40% of the fraud, a search reranker that beats the baseline on offline CTR but loses on satisfaction, a summarizer that scores well on ROUGE while hallucinating dates. Every one of those disasters started with a metric that measured the wrong thing.

This guide is the exact metric playbook I use across projects. It pairs closely with RAG that actually works (where evaluation is half the work), prompt engineering for agents (where the judge prompt is the eval), observability for AI systems (where offline evals meet live signals), and MLOps on a shoestring (which pipelines the whole thing).

Accuracy lies whenever your classes are imbalanced. In fraud, spam, churn, medical screening, or content moderation, a model that predicts never can score 99% and be worthless. The full triple to always report:

• Precision — of the ones we flagged, how many were right? Drives user trust and cost of false positives.
• Recall — of the ones that were bad, how many did we catch? Drives coverage and cost of misses.
• PR-AUC — quality across all thresholds. Compare models here, then pick the threshold from a business trade-off.

The threshold is a product decision, not a modeling decision. Show stakeholders the precision–recall curve and let them pick the operating point. A fraud team may accept 30% precision for 90% recall; a marketing team wants 90% precision at 30% recall.

from sklearn.metrics import precision_recall_curve, average_precision_score
import numpy as np

precisions, recalls, thresholds = precision_recall_curve(y_true, y_scores)
pr_auc = average_precision_score(y_true, y_scores)

# Pick threshold: max F_beta where beta encodes business trade-off.
# beta > 1 favors recall (fraud), beta < 1 favors precision (paid ads).
beta = 2.0
f = (1 + beta**2) * (precisions * recalls) / (beta**2 * precisions + recalls + 1e-9)
best = np.nanargmax(f[:-1])
print("threshold", thresholds[best], "P", precisions[best], "R", recalls[best])

Calibration is the other half. If your model says 0.8, does 80% of the 0.8-bucket actually flip positive? Plot a reliability diagram; if it curves, apply Platt scaling or isotonic regression. Uncalibrated probabilities break every downstream decision system.

Macro-F1 hides the class that's dying. Always publish a per-class table (precision, recall, support) alongside the aggregate. When classes are hierarchical (product categories, medical codes), report hierarchical-F1 that gives partial credit for near-misses.

For multilabel (an item can have multiple tags), sample-averaged F1 rewards models that get most labels right on most examples; label-averaged F1 punishes long-tail failures. Pick the one that matches how users experience the failure.

For search, recommendations, or any list of results, the standard is NDCG@k on a versioned, human-labeled gold set of 200–1,000 queries with 5–20 rated candidates each. Ratings should be graded (0–3), not binary; the discount factor in NDCG only pays off when relevance is graded.

Click-through rate is a downstream product metric. It's great for A/B tests and terrible for offline model comparison because it depends on the UI, layout, and time of day. If offline NDCG and online CTR disagree, trust CTR — but keep NDCG as your fast iteration signal.

For RAG retrieval specifically, recall@k is the honest metric — the reranker will fix ordering; retrieval only has to include the answer. See the full pattern in RAG that actually works.

Free-text outputs — summaries, chat replies, code, translations — need a rubric with 3–5 criteria. My default rubric for most chat systems:

1. Faithfulness — does the answer stay within provided context / verifiable facts?
2. Completeness — did it answer the whole question, not just the easy part?
3. Tone/format — does it match the requested style and structure?
4. Safety — no policy violations, no PII leak, appropriate refusals.

Encode the rubric in a judge prompt. Score with a strong model (frontier tier). Sample 100 outputs quarterly, have humans score them with the same rubric, and compute Spearman correlation. If Spearman drops below 0.6, retrain the judge prompt — the judge has drifted or your data distribution shifted.

JUDGE_PROMPT = """You are a strict evaluator. Given the QUESTION,
the ANSWER, and the reference CONTEXT, score on 1-5:

1. Faithfulness: does ANSWER stay within CONTEXT? (1=fabricates, 5=fully grounded)
2. Completeness: does ANSWER cover the QUESTION? (1=misses, 5=complete)
3. Tone: appropriate register and format? (1=off, 5=perfect)

Return JSON: {"faithfulness":n,"completeness":n,"tone":n,"reasons":"..."}
Do not explain outside the JSON."""

def judge(question, answer, context):
    resp = llm.chat(
        model="strong-judge-model",
        temperature=0,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": JUDGE_PROMPT},
            {"role": "user", "content": f"QUESTION: {question}\nCONTEXT: {context}\nANSWER: {answer}"},
        ],
    )
    return json.loads(resp.choices[0].message.content)

BLEU and ROUGE remain useful only for tasks with a single correct answer — short translations, structured extraction, exact-match code. For open-ended generation they under-rank paraphrases and over-rank surface overlap.

End-to-end quality is a lagging indicator that hides which half is broken. Report both:

1. Retrieval quality — did the retrieved chunks contain the answer? Measure recall@k against a gold set of (question, chunk_id) pairs.
2. Answer faithfulness — does the answer stay within what the chunks say? LLM judge with the chunks in context.

Additionally track answer relevance (did we answer the actual question?) and context precision (how many retrieved chunks were actually used?). Frameworks like Ragas and TruLens automate these but the numbers only matter if your gold set is real.

For agents (multi-step tool use), end-to-end task success is the north star but it's too coarse to debug. Instrument each step:

• Tool-selection accuracy — did the agent pick the right tool for this step?
• Argument correctness — were the tool arguments well-formed and semantically right?
• Step efficiency — did it converge in ≤ N steps, or loop?
• End-to-end success — did the final state match the gold?

See the exact instrumentation pattern in building production AI agents with n8n and prompt engineering for agents.

A single number is not a metric — it's an anecdote. Every reported score needs a 95% confidence interval, computed by bootstrapping 1,000 resamples of your eval set. If the CI of model A overlaps model B, you didn't beat it.

import numpy as np
def bootstrap_ci(scores, n=1000, alpha=0.05):
    rng = np.random.default_rng(42)
    means = [rng.choice(scores, len(scores), replace=True).mean() for _ in range(n)]
    lo, hi = np.percentile(means, [100*alpha/2, 100*(1-alpha/2)])
    return float(np.mean(scores)), float(lo), float(hi)

mean, lo, hi = bootstrap_ci(per_example_scores)
print(f"{mean:.3f} [{lo:.3f}, {hi:.3f}]")

The single most under-used practice: publish a slice table. Break the aggregate by language, tenant, question type, product line, model version, and time-of-day. Regressions almost always live in a slice, not the whole.

For multilingual systems specifically, English usually dominates the eval set and masks bad performance in other languages. Weight the aggregate by traffic share per language, or report per-language rows. See building multilingual AI (Arabic + English) for the exact pitfalls.

Quality on its own is a fantasy metric. A 1-point NDCG gain that doubles latency or triples cost is usually a regression. Every model comparison chart should have three axes: quality, p95 latency, cost per 1k requests.

The cost side lives with your warehouse cost controls and LLM cost dashboard. The latency side lives in your observability layer.

• Eval set: git-tracked with a semantic version. Any change to examples or labels bumps the version.
• Judge prompt: versioned in the same repo. Changing the judge changes the metric — treat it like a model release.
• Model + params: model id, temperature, seed, and prompt template id are stored with every score.
• Run manifest: for each eval run, write a JSON with dataset version, judge version, model version, and seed. That is the unit you compare against next quarter.

Without this, you'll spend a week arguing whether model B is better than last quarter's model A. With it, the comparison is a git diff.

Offline evals are fast, cheap, and lie a little. Online metrics (A/B tests, live thumbs, task completion) are slow, expensive, and truthful. Run both, and treat any disagreement as new information:

• Offline up, online flat → your eval set doesn't match production distribution. Add cases.
• Offline flat, online up → your eval set is missing the criterion users actually care about.
• Both up → ship.
• Both down → roll back immediately, then investigate.

• Confidence intervals, not point estimates. Always bootstrap.
• Slice by segment (language, tenant, question type). Aggregate metrics hide regressions.
• Cost and latency on the same chart as quality. Trade-offs must be explicit.
• Version datasets. Reference the version in every number.
• Diff table (this model vs previous, per slice, with CI arrows) — leadership can read it in 30 seconds.