Evaluating LLM Apps
Know whether your app works and whether a change helped: build a test set, choose checks from exact match to rubrics and LLM judges (and their biases), measure retrieval and grounding separately, and monitor quality, cost and speed in production.
Why evaluation is the core skill
LLM apps fail quietly. A prompt tweak that fixes one example can break ten others, a provider can update a model, and users will find cases you never imagined. Without measurements you are guessing. The teams that build reliable LLM apps are the ones with a good evaluation set and the habit of running it on every change, just like unit tests in normal software.
Build a test set
- Collect realistic inputs: 30 to 100 to start, covering common cases, edge cases (very long, empty, wrong language, rude), and things the app must refuse.
- For each, write the expected behaviour: an exact answer, the key facts the answer must contain, or which source it must cite.
- Keep growing it: every bug report from users becomes a new test case.
- Keep some cases held out that you never look at while tuning prompts, for the same reason you keep a test set in ML.
Ways to score an answer
| Check | Good for | Watch out for |
|---|---|---|
| Exact match / code checks | classification, extraction, JSON validity, "does it cite a source?" | only works when there is one right answer |
| Contains key facts | RAG answers: are the 3 facts that matter present? | wording varies; keep the list short |
| Human rating with a rubric | tone, helpfulness, safety | slow and costly; write clear criteria so raters agree |
| LLM as a judge | scaling rubric checks to thousands of answers | judges prefer longer answers, the first option shown, and their own style; check them against human ratings |
For an LLM judge, give it a clear rubric with examples, ask for a reason before the score, use a simple scale (pass/fail or 1 to 3), and swap the order when comparing two answers.
Evaluate retrieval and generation separately
For RAG, measure the two halves on their own so you know which to fix:
- Retrieval: for each test question, mark which chunks are relevant. Recall@k is the fraction of questions where a relevant chunk appears in the top k; MRR (mean reciprocal rank) averages 1/rank of the first relevant chunk, rewarding finding it near the top.
- Generation: faithfulness (is every claim supported by the retrieved sources?) and answer relevance (does it actually answer the question?). Tools such as Ragas automate these with LLM judges.
Try it: recall@k and MRR
Each row is a test question and the rank at which the first relevant chunk was retrieved (or "not found"). Edit the ranks and change k.
Interactive: this part runs in your browser.
In production
- Log inputs, retrieved chunks, outputs, latency, token counts and cost for every request (and remove or protect personal data in those logs).
- Collect user feedback (thumbs up/down) and review a sample of conversations every week.
- Track cost per request and the slowest 5% of response times, not just averages.
- Re-run your evaluation set when you change prompts, models or retrieval, and before switching to a new model version.
Quiz
1. You switch to a cheaper model. How do you decide whether it is good enough?
2. A known bias of LLM judges is that they…
3. Recall@3 is 0.6. What does that mean?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Ragas documentation ↗ metrics for evaluating RAG pipelines
- Anthropic: Building effective agents ↗ workflows vs agents and the common patterns, from real projects