Internship← AI Product Building learning pathAI Product Building · Module 6 of 10
Evaluations (evals)
Evals measure AI quality so you can improve it with confidence.
📖 Key ideas
- Build a test set of real inputs with expected or good outputs.
- Score with exact checks, rubrics, or another model as a judge.
- Track metrics: accuracy, helpfulness, refusal rate, latency, cost.
- Run evals before every prompt or model change.
- Collect user feedback (thumbs up/down) and add failures to the test set.
💡 Real-life example: Like mock exams before the board exam: the same paper every time shows if you are really improving.
EvalA test that scores AI output
Test setFixed examples used for evaluation
LLM-as-judgeUsing a model to grade outputs
RubricScoring guide with criteria