How to Build an LLM Evaluation Framework for Production

Why evaluation decides whether your AI improves

Evaluating LLMs is one of the least glamorous parts of AI engineering and one of the most important. Without a rigorous evaluation system you cannot tell whether a prompt change made things better or worse, or whether a new model version is an upgrade or a regression. The system drifts, and nobody notices until customers do.

An LLM evaluation framework turns “does this feel better?” into a repeatable release decision. It combines a representative dataset, explicit quality criteria, automated checks, and a process for reviewing the cases automation cannot judge reliably. It sits inside the wider production-readiness work in our AI production readiness checklist: without it, model upgrades and prompt edits become unmeasured product changes.

This guide covers both halves of the problem: the decisions a CTO needs to make about what quality means and who owns it, and the engineering mechanics that make those decisions enforceable.

Start from a user decision, not a benchmark

Public benchmarks tell you what a model can do on someone else’s data. They say little about your users, your documents or your output format. Start instead with the decisions your system helps users make, then describe what a good and a bad result looks like in that context. A support assistant may need grounded answers with correct citations; an extraction workflow may need a strict schema and high field accuracy; an internal coding assistant may need changes that compile and pass tests.

For each flow, define:

• a curated input set that represents real use;
• expected properties or reference answers;
• the quality criteria that matter, such as correctness, groundedness, format compliance and safety;
• latency and cost limits alongside quality measures;
• a threshold, and an owner who decides on exceptions.

Writing this down per flow matters more than any tooling choice. It gives engineering, product and the business owner a shared definition of “good enough to ship”.

Build the golden dataset carefully

Your evaluation is only as good as its inputs. The most common mistake is generating synthetic test data with the same model you are testing: the model grades its own blind spots as correct, and the evaluation becomes circular. Build the dataset from:

• real production queries, anonymised before they enter the test set;
• edge cases your team has found through adversarial testing;
• inputs from customer support tickets that historically caused problems;
• distribution-representative samples across every use-case category, so a rare but important flow is not drowned out by a common one.

As a working rule, aim for at least 200 to 500 examples before you trust aggregate numbers for a flow. Treat that as a floor for confidence rather than a target: coverage of the flows and failure types that matter is worth more than the raw count, and a small, well-labelled set beats a large noisy one.

Version the dataset alongside the code and refresh it at least quarterly. Production traffic shifts as users find new ways to use the product, and a test set frozen at launch slowly stops describing reality.

Offline and online evaluation: you need both

Offline evaluation before every release

Offline evaluation runs before any change reaches production: a new model, a prompt edit, a retrieval change, a new tool or a policy update. It runs in CI against the golden dataset, and any change that degrades the score beyond your agreed threshold blocks the deployment.

Pick metrics that fit the task:

• Exact match or F1 for factual extraction and classification.
• ROUGE or BERTScore as a rough signal for summarisation, never as the only one.
• G-Eval or LLM-as-judge for open-ended generation quality.
• Rule-based checks for format adherence, schema validity, required fields and banned content. Always include these: they are cheap, deterministic and catch the regressions users notice first.

Online evaluation after release

Offline tests cannot anticipate everything real users do, so keep monitoring production traffic. You cannot review every response, but you can sample. Combine automated signals (format failures, latency spikes, refusals, error rates) with periodic human review of a privacy-safe sample.

Track these over time for each workflow: a response quality score, refusal rate, formatting error rate, and semantic drift from a baseline. User signals such as corrections, thumbs-down ratings, escalations to a human and abandoned sessions are noisy individually but useful in aggregate.

Use LLM-as-judge where there is no single right answer

For nuanced Q&A, tone-of-voice consistency or creative generation, string-matching metrics fall short. The LLM-as-judge pattern uses a separate, capable model to score outputs against a rubric you define. It scales far better than human review, but only if you treat the judge as a component that also needs testing.

The rules we apply:

• Use a judge model different from the one you are evaluating, so it does not share the same blind spots.
• Give the judge a structured rubric with explicit 1 to 5 scoring guidelines for each criterion, not vague instructions such as “rate the quality”.
• Include reference outputs in the judge prompt wherever you have them.
• Score one criterion at a time; a single overall score hides which dimension regressed.
• Measure agreement between the judge and human annotators periodically, and re-check it when you change the judge model or prompt.
• Version the judge prompt and model like any other dependency, so a score change can be traced to the system under test rather than to the ruler.

Test every system boundary

An LLM result depends on more than the model. Many regressions blamed on “the model got worse” are really retrieval returning the wrong passages, a tool changing its response schema, or a parser silently dropping fields. Evaluate each boundary separately as well as end to end:

• Retrieval: are the right documents returned, and does the answer stay grounded in them? Our guide to reliable production RAG covers retrieval evaluation in more detail.
• Tool selection and arguments: does the system call the right tool with valid parameters, and does it recover when a tool fails?
• Structured output parsing: does every response validate against the schema you depend on downstream?
• Fallback behaviour: when a dependency fails or the model refuses, does the user get a sensible path forward?

For systems built with LangChain and LangGraph, include state transitions and tool failures in the test set, not only final text. For agents, record what the agent touched as well as what it answered.

Capture the exact model version and inference configuration in every run, whether you use OpenAI or an open-source model from Hugging Face. Without that, a provider update or a change to your serving stack makes historical scores impossible to compare.

Use release gates that match risk

Not every flow deserves the same gate. High-risk flows, such as anything that gives advice, touches money or exposes customer data, should block a release when quality regresses. Lower-risk flows may be better served by sampled review or a feature-flagged rollout to a small audience. The point is not a perfect score; it is making trade-offs visible before customers encounter them.

Prompt changes deserve the same treatment as model changes. A “small” prompt edit that ships without evaluation is how responses quietly degrade two weeks later. If your prompts live in a versioned registry with tests, as described in prompt engineering at scale, wiring them into the same gate is straightforward.

Close the loop with production feedback

The most valuable test cases come from production failures. Review a privacy-safe sample of bad outcomes regularly, work out why each one happened, and add it to the golden dataset so the same failure cannot ship again. Over time, every production failure becomes a permanent offline test case.

This loop has a second benefit: it reveals when a retrieval, data or product-design problem is being mistaken for a model problem. Teams without it tend to respond to every quality complaint by swapping models, which is expensive and rarely fixes the underlying cause.

Make evaluation an operating habit

Treat the evaluation suite as part of the product, not a one-off project:

• Ownership: a named owner for each flow’s criteria and thresholds.
• Versioning: datasets, rubrics and judge prompts in version control, with results stored against each release.
• Cadence: offline evaluation on every change, online review on a fixed schedule, and a dataset refresh at least quarterly.
• Visibility: results that product and engineering can both read, so launch decisions use a common language.

That operating layer is part of a broader MLOps foundation for AI products. Without it, your AI system will silently degrade over time and your team will hear about it from customers first.

If your team needs help defining release criteria or building the evaluation pipeline, discuss your evaluation programme with Tensorplay.