A model can have low training loss while producing poor outputs in production. Rigorous evaluation requires multiple layers: automated metrics, LLM-as-judge, human evaluation, and production monitoring.
Never evaluate a fine-tuned model in isolation. Compare it to the base instruction-tuned model on the same test set. Improvement on target metrics should not come at the cost of general capability regression.
Held-out test set — Created before training begins. Diverse phrasing and topic coverage. Measures generalization, not memorization.
Automated metrics — Exact match, F1, ROUGE/BLEU for tasks with objective answers. Fast and scalable. Unreliable for open-ended generation quality.
LLM-as-judge — Use a frontier model to score responses against a rubric. Strongly correlated with human judgment, much cheaper. Main weakness: may share biases with the evaluated model.
Human evaluation — Gold standard for final go/no-go decisions, calibrating LLM-as-judge, and identifying failure modes automated metrics miss.
Production monitoring — User feedback, automated quality sampling, and interaction logging. Reveals failure modes that offline evaluation never anticipated.
Fine-tuning can improve specific task metrics while degrading general language quality, instruction following, or safety behavior. Evaluate both your target metrics and a broad set of general capability benchmarks on every fine-tuned checkpoint.
Ask the AI assistant about evaluating fine-tuned models, LLM-as-judge, regression testing, or production monitoring strategies.