A better answer. Or just a different one?
Run your changes against the same real examples. See what improved, what broke, and which answers still need work before you ship.
questions.
Real examples.
Expected answers.
One frozen dataset.
Original prompt
Updated prompt
Test question
Can I return my order after 45 days?
ImprovedBefore
Yes! You can return your order within 60 days.
After
The return window is 30 days, so an order delivered 45 days ago is outside the policy. [1]
THE QUESTION THAT MATTERS
Did my change actually make things better?
01The same cases.
02A clear definition of good.
03A result you can defend.
DATASETS02
Keep the mistakes you never want to repeat.
Turn a useful trace into a test case. Save the input, relevant context, and expected behavior in a dataset you can run against future changes.
Start from real evidence
Promote a production trace or span into a case with provenance, or create and import cases directly.
Define the expectation
Keep the observed output separate from the expected answer. Add metadata that explains why the case matters.
A production mistake becomes a test
Input
Can I return my order after 45 days?
Expected behavior
Explain that the 30-day return window has passed. Cite the returns policy.
Output diff
- Expected output
- The return window is 30 days, so an order delivered 45 days ago is outside the policy. [1]
- Actual output
- The return window is 60 days, so an order delivered 45 days ago is within the policy.
SCORERS03
Tell the test what “good” means.
Check an exact value with code, ask an LLM judge to apply a rubric, or combine several checks. Each result keeps the score and its reason.
Choose the right check
Use JavaScript, Python, LLM, or library scorers for the quality question you need to answer.
Test the scorer itself
Preview a rule against known good and bad examples. An execution error remains an error, not a zero score.
Rubric
Choice scores
- Pass
- 1
- Fail
- 0
REVIEWS & HUMAN SCORES04
Some answers need a human eye.
An answer can pass a code check and still be unhelpful. Give reviewers the original evidence and shared criteria to judge correctness, tone, and usefulness.
Build a review queue
Select useful traces and organize them into a session with instructions and assigned reviewers.
Review in context
Read inputs, outputs, and spans in the inspector. Annotate the exact passages that need attention.
A shared rubric, not a gut feeling
“The answer is accurate, but it could explain the customer’s next option.”
THE EVALUATE TOOLKIT
A repeatable answer to “did it get better?”
START WITH ONE REQUEST