LLM Evaluation Guide

As a research scientist, the question I ask about any AI system is the same one I'd ask about an experiment: what's the evidence this actually works, and under what conditions does it fail? Most GenAI projects skip this step entirely — they ship on "the demo looked good" and find out about failures from users. Here's how to evaluate an LLM system properly, before that happens.

Why "it looks right" isn't enough

LLM outputs are fluent by default — even a wrong answer usually reads as confident and well-formed, which is precisely what makes eyeballing outputs an unreliable evaluation method. A prompt or RAG pipeline that performs well on the three examples you tried by hand can fail on 30% of real user inputs you never tested. Evaluation is how you find that 30% before your users do.

Three approaches to LLM evaluation

Most mature evaluation setups use all three: programmatic checks catch structural failures instantly and cheaply, LLM-as-judge scores subjective quality at scale, and a smaller human review samples the results periodically to catch what the automated methods miss.

Evaluating RAG systems specifically

A RAG pipeline has two places to fail independently, and conflating them is the most common evaluation mistake:

• Retrieval quality — did the system find the right chunks at all? Measured with metrics like recall@k (did the correct chunk appear in the top K results?) and precision (how much of what was retrieved was actually relevant?).
• Generation faithfulness — given the retrieved chunks, did the model's answer actually stick to them, or did it hallucinate on top of correct context? This is measured separately from retrieval quality.

A RAG system can retrieve perfectly and still hallucinate in the generation step, or retrieve badly and still produce a plausible-sounding (but wrong) answer — which is why these need distinct tests, not one end-to-end pass/fail.

Using an LLM as a judge

LLM-as-judge means prompting a (usually stronger or differently-configured) model to score another model's output against a rubric — "is this response faithful to the provided context? Rate 1–5 and explain why." This scales far better than human review, but has known failure modes worth designing around:

• Judges tend to favour longer, more verbose answers regardless of actual quality
• Judges can be inconsistent between runs unless temperature is set low and the rubric is specific
• A judge sharing the same blind spots as the model being judged won't catch shared failure modes

Building an evaluation test set

A good evaluation set includes: typical, everyday inputs (most of your traffic); known edge cases (ambiguous questions, missing information, adversarial phrasing); and regression cases (past failures you've already fixed, kept permanently so they never silently break again). Aim for a set large enough to be statistically meaningful but small enough to re-run cheaply every time you change a prompt or pipeline step — this is what makes "evaluate before you ship a change" a realistic habit rather than a one-time audit.

Evaluation doesn't stop at launch

Real usage surfaces inputs no test set anticipated. Production AI systems need ongoing evaluation: sampling live conversations for review, tracking user feedback signals (thumbs up/down, follow-up corrections), and watching for quality drift when an underlying model version changes upstream. This connects directly to the monitoring and logging practices covered in our production AI guide.

In the Generative AI course, evaluation isn't a separate lecture bolted on at the end — it's built into how every project is graded: does it work on cases beyond the one you tested by hand? That habit is worth more long-term than any single technique in this article.