Training data quality is the single most important factor in fine-tuning success. A mediocre model on excellent data outperforms an excellent model on mediocre data. Every hour improving data returns more value than the same time spent on hyperparameter tuning.
Correct — Every example demonstrates exactly the behavior you want. Incorrect or borderline examples directly teach the wrong patterns.
Diverse — Examples cover the full range of inputs the model will encounter in production, including edge cases. Easy-only datasets leave gaps.
Consistent — All examples demonstrate the same style, format, and behavior. Inconsistency teaches the model that multiple behaviors are acceptable — and it will switch between them unpredictably.
Appropriate length — Responses match the length you actually want in production. Not artificially long to seem thorough, not artificially short.
No leakage — No PII, private customer data, or confidential information that should not be encoded in model weights.
Contamination occurs when training examples overlap with your evaluation set — the model appears to perform well because it memorized answers, not because it generalized. Always define your evaluation set first, then collect training data with no overlap.
200–1,000 examples. Getting consistent JSON output or brand voice requires fewer examples than complex reasoning tasks.
1,000–100,000+ examples. Broader task coverage and domain depth require more examples to generalize across the full input distribution.
Always evaluate on a held-out validation set and collect more data if performance is insufficient — don't add epochs as a substitute for more data.
Ask the AI assistant about building fine-tuning datasets, avoiding data contamination, or how much data you need.