How to Evaluate LLM Outputs: Evals for Business Apps

Your AI support assistant looked great in the demo. You tried 10 questions, the answers were good, and it shipped. Three weeks later, someone tweaks the prompt to make it friendlier, and now it occasionally promises refunds your policy doesn’t allow. Nobody noticed until a customer quoted it back to you.

That’s what happens without evaluations, or “evals.” Large language models don’t behave like regular code. The same input can produce different outputs, a small prompt change can shift behavior in unexpected places, and a new model version can be better at one thing and worse at another. Evals are how you know, with evidence, whether your AI feature is good enough to ship and whether it stays that way.

What LLM evaluation means for a business app

An eval is a repeatable test: a set of inputs, a definition of a good output and a way to score the results. For a business app, it answers practical questions:

• Does the assistant answer correctly from our documents?
• Does it follow our policies and tone?
• Does it refuse things it shouldn’t do?
• Did the new prompt or model make it better or worse?

OpenAI’s evaluation guidance lists “vibe-based” evaluation, the “it seems like it’s working” approach, as an anti-pattern, along with waiting until after launch to add evals. Anthropic’s testing guidance makes a similar point from the other side: build task-specific tests that mirror real usage, including edge cases.

This is different from public leaderboards and benchmarks. Benchmarks tell you how a model does on general tests. Your evals tell you how your prompt, your data and your chosen model perform on your task.

Step 1: Define success before you build

Write down what “good” means in measurable terms. For a customer support assistant, that might be:

Pick targets that match the risk. A typo in a marketing draft is cheap. A wrong dosage, price or legal claim is not. Our guide on what AI hallucinations are explains why accuracy needs special attention.

Step 2: Build a small, realistic test set

You don’t need thousands of examples to start. A test set of 50 to 200 cases that reflects real usage catches most problems early. Include:

• Common questions your users ask most often, pulled from support tickets, search logs or sales calls
• Hard cases: ambiguous questions, multi-part questions and questions that need 2 documents to answer
• Edge cases: very long inputs, typos, missing information and questions where the right answer is “I don’t know”
• Out-of-scope requests the assistant should politely decline
• Adversarial cases: attempts to make it ignore instructions or reveal confidential information

For each case, record the input and either the expected answer, the key facts the answer must include or the behavior you expect (for example, “declines and offers to connect to support”).

Anthropic’s guidance says that more test cases with automated grading is generally better than a few hand-graded ones. Start small, automate the scoring and grow the set over time. Every real failure in production should become a new test case.

Step 3: Choose how to score each output

There are 3 main grading methods. Most teams combine them.

Anthropic’s documentation ranks code-based grading as fastest and most reliable, human grading as most flexible but slow and expensive, and LLM-based grading as fast and flexible but needing a reliability check before you scale it.

Key takeaway: Use code checks wherever a rule can decide, an LLM judge for nuance, and humans to set the standard and audit the judge. Never trust an automated judge you haven’t compared with human scores.

Step 4: Write a rubric an LLM judge can use

LLM judges work, with caveats. A widely cited research paper on the approach (Zheng and colleagues, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” 2023) found that strong LLM judges reached over 80% agreement with human preferences, about the same level at which humans agree with each other. The same paper documented position, verbosity and self-enhancement biases.

To get reliable scores:

1. Make the rubric specific. “Is it good?” fails. “Does the answer state the correct return window from the policy document? Answer yes or no” works.
2. Score 1 criterion at a time. Separate accuracy, tone and format into different checks.
3. Use simple scales. Pass/fail or 1 to 5 with a written definition for each level. Anthropic recommends constrained outputs like “correct” or “incorrect” because purely qualitative feedback is hard to assess at scale.
4. Let the judge reason first. Both OpenAI and Anthropic recommend having the grader think before giving a score.
5. Control for known biases. OpenAI’s guidance flags position bias (preferring whichever answer comes first) and verbosity bias (preferring longer answers). Use pass/fail or randomize order in pairwise comparisons, and tell the judge not to reward length.
6. Consider a different model as judge. Anthropic’s examples note that it’s generally best practice to evaluate with a different model than the one that produced the output.
7. Calibrate with humans. Have people score a sample of 30 to 50 outputs, compare with the judge and adjust the rubric until they agree on most cases.

Here’s a simple accuracy rubric you can adapt:

• Pass: All facts match the provided source; no invented details; answers the actual question.
• Fail: Any fact contradicts or isn’t supported by the source, or the answer misses the question.

Step 5: Run evals on every change

Evals are only useful if you run them consistently. OpenAI’s guidance recommends evaluating continuously: run evals on every change, monitor production for new kinds of failures and grow the eval set over time.

Run your full test set whenever you:

• Edit the system prompt or instructions
• Switch models or model versions
• Change retrieval settings, document sources or chunking (see what RAG is)
• Add tools or new capabilities
• Change temperature or other parameters

Track results in a simple table: version, date, pass rate per criterion, cost and latency. If a change improves tone but drops accuracy, you’ll see it before your customers do. This is the same idea as regression testing in regular software; our guide to manual vs automated testing covers the basics, and what MLOps is shows how evals fit into a production pipeline.

Step 6: Keep evaluating after launch

Your test set is a sample of reality, not reality itself. After launch:

• Review a random sample of real conversations each week.
• Collect user feedback with a simple thumbs up or down and a reason.
• Log failures and turn each one into a new test case.
• Watch operational metrics: escalation rate, refusal rate, response length, latency and cost per conversation.
• Re-run evals when your provider updates the model, even if you didn’t change anything.

Tie quality metrics to business outcomes where you can, such as resolution rate or time saved. Our guide on how to measure the ROI of AI shows how.

Where to start this week

1. Write 5 success criteria for your AI feature, with a target for each.
2. Collect 50 real inputs from logs, tickets or colleagues, including 10 hard or out-of-scope cases.
3. Write the expected answer or behavior for each.
4. Add code checks for anything rule-based, such as format or required phrases.
5. Write 1 rubric for accuracy and 1 for tone, and run an LLM judge on all 50.
6. Have a person score 20 of them and compare with the judge.
7. Save the results as your baseline, and rerun before every change.

You can do this in a spreadsheet and a short script. Specialized eval tools and platforms help once you have many test cases, multiple features or a team sharing results, but the thinking matters more than the tool. If you’re formalizing AI oversight, add evals to your AI governance checklist.

Conclusion

You wouldn’t ship regular software without tests, and AI features deserve the same discipline. Define what good looks like, build a realistic test set, score it with code checks, a calibrated LLM judge and human review, and rerun it every time something changes. A few days of setup turns “it seems fine” into evidence, and catches the refund-promising chatbot before your customers do.

Sources

• OpenAI API: Evaluation best practices — eval process, vibe-based anti-pattern, position and verbosity bias, calibrating with human labels and continuous evaluation
• Anthropic: Define success criteria and build evaluations — task-specific test design, volume over hand grading, code vs human vs LLM grading and tips for LLM-based grading
• Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (2023) — over 80% agreement between strong LLM judges and human preferences; position, verbosity and self-enhancement biases