Supervised Fine Tuning: Build a Reliable SFT Dataset
Cover Image
A fine-tuning run can complete without an error and still teach the model the wrong behavior. The cause can hide in the examples: overrepresented or conflicting examples, a prompt formatted differently from production, the wrong message marked as the target, or a validation set that repeats training questions with slightly different wording.
Supervised fine tuning (SFT) works by showing a model examples of the responses it should produce. That makes the dataset a description of the behavior you want. Before spending time or compute on training, define that behavior, curate representative examples, verify how your trainer turns them into tokens and loss targets, then test the result against held-out cases and existing capabilities. This guide walks through that process for instruction and chat models. It focuses on preparing the data, not choosing a cloud provider or tuning every training parameter.
Key takeaways
Define one observable behavior and measure a baseline before training.
Review targets, isolate related examples by split, and inspect the rendered loss mask.
Compare a pilot against the baseline on target tasks and regression cases.
Decide whether supervised fine tuning fits the problem
Start with the problem you want to solve, not the fact that a model can be fine-tuned. SFT is a reasonable candidate when you need a model to repeat a task pattern consistently: classify inputs, follow a specific response format, apply a nuanced translation style, or stop making a recurring instruction-following mistake. OpenAI's current model-optimization guide describes SFT as learning from examples of correct responses and recommends measuring a baseline, prompting first, fine-tuning when useful, and evaluating on representative test data. The guide also notes that its hosted fine-tuning platform is winding down and is no longer available to new users, so check provider availability before planning around a managed service. The current optimization guide is a useful example of why the data workflow should be independent of a single provider.
SFT is less direct when the main problem is that the model cannot access current or private facts. A training example can influence what the model says, but it does not work like a searchable database that guarantees retrieval of a particular record. If a user needs the latest inventory, policy, or account detail, give the model access to that source at inference time. Fine tuning can still teach the response pattern used with retrieved information, but it is a different job.
SFT also differs from preference training. In ordinary supervised examples, you provide a prompt and a target response. The trainer learns to reproduce the target. If the task requires learning which of two plausible answers is better, preference optimization may represent the feedback more directly. Do not make your SFT corpus do a job its labels cannot express.
Before collecting examples, write a short success statement. For instance: “Given a support message and an approved policy excerpt, return one of four issue categories and a short response that cites only the supplied policy.” That sentence tells you what a useful example must include, what the model should emit, and which errors matter. It also gives you something to test before training.
Define the behavior before collecting examples
A dataset should encode one clear target behavior. If the objective includes several goals, such as tone, factual accuracy, refusal policy, tool use, and JSON formatting, test each separately so an overall score does not hide a regression. Divide the work into tasks or label the goals so you can test them separately.
Turn the success statement into an example specification. Write down:
The input fields the model will see in production.
The response fields or content it must generate.
The information it may rely on, including whether context is supplied in the prompt.
The required format, such as a JSON object or a conversational reply.
What the model should do when the input is missing information, ambiguous, or outside scope.
The errors that matter most and how a reviewer will score them.
Then create a small acceptance set before you expand the training corpus. Include common cases, boundary cases, malformed inputs, and cases where the correct behavior is to ask a question or decline. Have someone who understands the task review the target answers. If reviewers cannot agree on what a good answer looks like, the training data will encode that uncertainty rather than solve it.
For a classification task, the target might be a single allowed label plus a short rationale if the production system needs one. For a response-generation task, it might be a concise answer with a fixed set of fields. For multi-turn chat, include enough preceding conversation to make the target turn understandable. Avoid adding long explanations just because they sound helpful. Extra text becomes part of the behavior the model is asked to imitate.
A good specification also draws a line around the model's job. If the system prompt already supplies a stable policy, decide whether training examples should repeat that policy or show only the user-facing behavior. If examples use incompatible policies, formats, or hidden instructions without distinguishing when each applies, the model sees conflicting patterns. AWS guidance on SFT data quality discusses consistent targets and duplicate or overrepresented examples. Keep the task definition stable or explicitly encode the intended variation.
Once the target is concrete, example selection becomes a coverage problem: which real inputs represent the behavior and the mistakes you need to fix?
Curate examples for correctness and coverage
Start from real or realistic inputs that resemble production. Synthetic examples can help fill a gap, but inspect them before they become targets. A model-generated answer may sound polished and still invent a fact, ignore a constraint, or answer a question that the input did not ask. The target should be checked against the source and task specification, not accepted because it reads well.
Review examples in small batches. For each pair or conversation, ask whether the input is understandable on its own, whether the response follows the rule, whether all claims are supported by information available to the model, and whether the target is the behavior you actually want. Keep a reason when you reject or edit an example. Those reasons reveal recurring data-generation problems that can be corrected upstream.
Coverage matters as much as correctness. A dataset full of easy, near-identical prompts may teach the model a narrow pattern. Group examples by task type, input source, language, user intent, difficulty, and important edge cases. Check whether common cases dominate so heavily that rare but consequential cases disappear. The objective is not to force equal counts across every category. It is to make the coverage deliberate and testable.
Avoid treating quantity as a quality measure. A 2025 ICML paper studies token-level quality and proposes a method for identifying uninformative or noisy parts of SFT data, which is a reminder that an otherwise useful record may contain a problematic span. A 2026 ICML study on data difficulty reports that the best difficulty level depends on the dataset size. Neither result gives teams a universal recipe for how many examples to collect or how difficult every example should be. They support a more modest practice: measure quality and coverage for your own task, then run controlled comparisons. The token-cleaning paper and the 2026 study of data difficulty make those limits explicit.
Deduplicate before splitting. Exact duplicates are easy to find with normalized text or hashes, but paraphrases and repeated templates can leak across splits too. If several examples come from one conversation, document, customer, or generated prompt family, keep related examples together when building train and validation sets. Otherwise, the model may see nearly the same case during training and evaluation, making the score look better than its behavior on new inputs.
Preserve the source and provenance of each example. Store an identifier, source, creation method, review state, task version, and any correction history alongside the training fields. You may not need to send those metadata fields to the trainer, but they help you trace a bad output back to the data that taught it. Keep a clean export step that selects only the fields the training framework expects.
The set should represent the behavior you need, not just the data that was easiest to collect. Next, make sure its structure survives conversion into the exact conversation the model will see.
Format conversations and verify the loss target
SFT trainers accept different shapes: plain text, prompt-completion pairs, or conversations represented as messages with roles. The correct shape depends on the model and training framework. Hugging Face TRL documents standard and conversational formats, as well as prompt-only and prompt-completion variants. Its dataset-format guide is a useful reference when preparing data for that trainer.
A conversation might be represented as structured messages:
{
"messages": [
{"role": "user", "content": "Sort this request into a category: ..."},
{"role": "assistant", "content": "billing"}
]
}
This is only a conceptual example. Check the schema required by the trainer you use. A different system may expect a prompt and completion, a JSONL record, or a rendered text sequence. Do not assume that a format accepted by one library will behave the same way in another.
Inspect the rendered conversation
For conversational data, the chat template matters. It inserts the model's role markers, message boundaries, and end-of-turn tokens. A template mismatch can make training examples differ from production prompts even when the visible words appear identical. Use the tokenizer or model's intended template, inspect several rendered examples, and confirm how the final assistant answer is terminated. TRL's chat-template documentation explains that templates define this structure and that some training features depend on template-specific markers.
Check which tokens receive loss
Check which tokens contribute to the loss instead of relying on the dataset shape alone. TRL supports completion-only loss for prompt-completion data. For conversational data, assistant-only loss must be enabled with assistant_only_loss=True, and the chat template must support the required assistant-span markers. Confirm this behavior for your installed version in the SFT Trainer documentation.
Do not infer that the mask is correct from a successful training run. Render a few records, inspect token IDs or the trainer's example preview, and confirm that:
The user and system messages appear as context, not as copied target text, when that is the intended objective.
The assistant answer is included in the target.
Tool calls, tool results, and final answers have the right roles and boundaries if your task uses tools.
End-of-turn tokens match the model's conversation convention.
Padding, truncation, and sequence packing do not cut off or join examples in an unintended way.
Packing combines short examples into longer sequences to improve efficiency. TRL supports different packing strategies, including strategies that truncate or split overflow sequences. Inspect the rendered records, sequence boundaries and loss masks under the strategy you actually use. Packing is a performance option to validate, not a repair for incorrect examples.
When the rendered examples look right, protect the evaluation split from contamination and decide what evidence will count as improvement.
Split and audit data before training
Build training, validation, and test sets from the same task definition, but keep their examples separate. A validation set helps you compare training choices. A final test set should remain untouched until you are ready to assess the selected model. If you repeatedly tune against the same test questions, it stops being an independent check. The scikit-learn guidance on data leakage explains why test data must not influence model choices.
Split by source or group when examples are related. For example, if several prompts are derived from one support ticket, keep them together. If synthetic data comes from a template, avoid putting near-identical template variants in both training and test. If a conversation has several turns, split the full conversation rather than individual messages. These choices reduce the chance that evaluation measures memorization of a source rather than performance on a new case.
Audit overlap with exact and near-duplicate checks. Define normalization for the task before exact matching: whitespace changes may be harmless in prose but alter code or structured output. For near-duplicates, use a similarity method suited to the data and review likely matches rather than dropping similar examples automatically. Two examples can share phrasing while testing different behavior. Preserve both only when the distinction is intentional and likely to occur in production. Our fine-tuning dataset quality checklist covers duplicate decisions and group-based splitting in more detail.
Define evaluation and regression checks
Keep an evaluation set that tests both the target skill and nearby behaviors that should remain intact. If the task is output formatting, test valid and invalid inputs, missing fields, nested values, and the application's parser. If it is support response style, score correctness, tone, refusal behavior, and whether the response invents unsupported information. Include examples that test other important capabilities the base model already has, so the team can notice regressions.
Use a consistent rubric. A model may produce a fluent answer that violates one critical requirement. Define which errors are disqualifying and which are minor. For structured output, validate the schema programmatically and separately review whether the values are correct. For open-ended output, combine human review with task-specific checks; one aggregate score can hide a regression in a minority class or language.
Run the baseline model against this evaluation set before training. Save the prompts, outputs, scores, model identifier, evaluator version, and date. If you have tried prompt changes, compare those too. A fine-tune should earn its place by improving on the baseline for the intended use, not merely by lowering training loss.
The evaluation design is what makes the small training run informative. With it in place, a pilot can tell you whether the dataset deserves to grow.
Run a small experiment and watch for regressions
Begin with a small training run using a stable, reviewed subset. Keep the base model, evaluation set, prompt, and scoring rubric fixed. Change one major variable at a time, such as adding a new class of examples or removing a suspected noisy group. Otherwise, you will not know what caused the difference.
Compare the trained model with the baseline on target cases and regression cases. Review examples where the new model improved, worsened, or changed its response unexpectedly. A lower training loss only means the model is predicting its training targets more closely. It does not show that it is more useful on unseen inputs.
Keep the experiment reproducible. Record the base model revision, tokenizer and chat-template version, dataset snapshot, data filters, training configuration, random seed if applicable, and evaluation code. If you make a new dataset version, describe what changed. Versioning helps teams identify when two runs used different examples or different prompt formatting.
A practical data loop looks like this: collect representative examples, review them, render them with the chosen template, train a small candidate, run the same evaluation, inspect failures, then update the corpus only when the evidence points to a data issue. If the model fails because the prompt omits required context, more target examples may not fix it. If the correct answer is missing from the data, add reviewed examples that address that gap. If the model already performs well with a prompt, a fine-tune may add maintenance work without enough benefit.
The value of SFT comes from matching a defined behavior with examples the model can learn from and tests that can detect unwanted changes. A short preflight checklist makes those steps repeatable.
A supervised fine-tuning dataset preflight checklist
Before launching a full training job, confirm each of these points:
The target behavior is specific, observable, and relevant to a real use case.
The current model has a measured baseline on representative evaluation examples.
Every training response is correct for the information available in its input.
Examples cover ordinary, difficult, ambiguous, and out-of-scope cases deliberately.
Exact and near-duplicate examples have been reviewed and grouped before splitting.
Train, validation, and final test data are separated by source or related group where needed.
The chosen record format matches the trainer and model.
The chat template and end-of-turn markers match the model's expected format.
The loss mask targets the intended completion or assistant messages.
Tokenization, truncation, padding, and packing have been inspected on real records.
The evaluation rubric measures both the target task and important regression behaviors.
Dataset, template, model, and evaluation versions are recorded.
The selected provider or training framework is currently available for the intended workflow.
No checklist can choose a universal number of examples for every SFT task. The data budget depends on what behavior must change, how consistent the source examples are, and how much variation appears in real inputs. Recent research also finds that the usefulness of example difficulty can depend on dataset size, so “harder” and “more” are not reliable goals by themselves.
If the model fails the evaluation, diagnose the failure before adding data. A missing behavior calls for targeted examples. Incorrect targets call for source correction. A formatting issue calls for checking the template and loss mask. A knowledge lookup problem may call for retrieval or live context. Keeping those causes separate makes each new training run easier to interpret.
Before the full run, inspect one accepted example, one rejected example and one held-out case. Confirm the rendered target, rejection reason and split assignment. Supervised fine tuning is easier to diagnose when you can trace the lesson in the data and measure what the model learned.
