Fine-Tuning an LLM: 6 Steps to Measure Improvement

Your model returns valid JSON in a demo, then wraps a production response in a paragraph of explanation. You add another instruction. The next response follows the schema but chooses the wrong category. Should you keep prompting, change models, or start training?

Fine-tuning an LLM means adapting a pretrained language model using additional training examples. It can help a model perform a specific task more consistently, but a completed training run does not establish that the application improved.

This guide follows one practical example: routing support tickets into a fixed set of categories. The same decision process applies to extraction, constrained writing and other repeatable tasks. Start by identifying the failure, establish a baseline, and keep a candidate only when its measured benefits justify its costs.

Key takeaway: Define what must improve and what must stay reliable before spending compute on adaptation.

The first decision is whether the failure calls for changing the model at all.

1. Decide whether fine tuning an LLM solves your failure

Sort a sample of failed requests into categories. A support assistant that cannot find a new refund policy has a different problem from a router that reads the correct ticket but repeatedly selects the wrong queue.

Retrieval supplies information at request time. Fine-tuning changes parameters through training. Both can coexist: a model might retrieve a policy and then apply a learned response format.

Think of retrieval as giving a colleague the current manual and fine-tuning as practicing a recurring procedure. Practice can teach information too, but it does not provide a convenient update mechanism or a source citation for every changing fact.

For the ticket router, start with a clear category definition, a few examples and schema validation. If mistakes remain systematic, collect them into an evaluation before preparing a training run.

2. Define the evaluation before changing weights

Write down a concrete acceptance rule. For example: improve routing correctness on ambiguous tickets while maintaining valid output and acceptable latency. Choose numerical thresholds from your application's requirements; there is no universal score that makes a fine-tune worthwhile.

Build separate collections for training, validation and final testing. Use validation to choose checkpoints and settings. Keep the final test collection untouched until you are ready to assess the selected candidate.

Split related records together. If a support conversation contains ten messages, scattering those messages across training and testing makes the test less independent. Similar issues arise with document revisions, repeated templates and near duplicates. The scikit-learn cross-validation guide explains why related groups need to stay separate during evaluation. See the related guide on quality checks for an LLM fine-tuning dataset for a more detailed data review.

For routing, useful checks include:

• Label correctness: did the response select the right queue?

• Contract compliance: did it return the permitted fields and values?

• Ambiguity handling: did it request clarification or escalate when necessary?

• Regression coverage: does it still handle common, previously reliable tickets?

Save the original request, expected result and raw model response for every case. Aggregate scores tell you where to look; individual failures tell you what happened.

Use a compact validation subset for frequent development checks, then run broader checks before accepting a candidate. With the evaluation fixed, you can choose an adaptation method without treating every training setting as a guess.

3. Choose full fine-tuning, LoRA or QLoRA

Supervised fine-tuning describes learning from examples of desired outputs. Full fine-tuning and adapter methods describe which parameters change during that process.

The official PEFT LoRA guide explains how adapters let training update a subset of parameters while the pretrained weights remain frozen. This makes LoRA a useful starting experiment when full training is impractical.

The original QLoRA paper introduces adaptation through a frozen, quantized base model. Its reported results describe particular experimental configurations; they are not a guarantee that any model and context length will fit your GPU.

Estimate memory for the complete workload: weights, activations, optimizer state, sequence length and batch settings. Run a short feasibility check before committing to a long job.

For the router, choose a model that already understands the ticket language and can follow the required format. Then test the smallest adaptation that might correct the remaining errors.

4. Run a small experiment with production-shaped inputs

Build examples that resemble actual requests, including confusing cases, missing details and acceptable escalation behavior. Avoid teaching the model that every request has a confident answer.

Review each example against the behavior you intend to serve. For a router, include tickets that mention more than one issue and specify which issue determines the destination queue.

Use the model's expected conversation format. The Transformers chat template documentation explains how role messages become token sequences. Incorrect control tokens or duplicated special tokens can undermine an otherwise reasonable experiment.

Inspect rendered examples before training. Check that the desired assistant response survives truncation and that your training objective covers the intended response tokens.

The TRL SFT Trainer reference documents supported dataset formats and training configuration. Use documentation matching your installed release, and record package versions alongside the base-model revision and adapter settings.

Change one major variable per comparison. If you replace the dataset, prompt and adapter configuration together, a better score will not explain which change helped. That discipline makes the next comparison much easier to interpret.

5. Compare the candidate on gains, regressions and cost

Evaluate the original model and selected candidate on the same requests with documented generation settings. Compare the strongest practical baseline you have, including prompt improvements you would otherwise deploy.

Training loss measures fit to the training objective. It does not directly measure correct ticket routing, useful escalation or end-to-end response time.

Check each category separately. A candidate can improve the overall average while getting a rare but important queue wrong more often.

Adaptation can also affect capabilities beyond the target task. Research on forgetting during fine-tuning finds that parameter-efficient methods can experience forgetting too. Preserve tests for the capabilities your application still needs.

For open-ended responses, combine task-specific checks with human review. If a model judge helps score outputs, compare its judgments with a human-reviewed sample and inspect disagreements.

When the gain is too small or inconsistent to justify the extra work, keep the baseline and investigate the errors. When the candidate earns acceptance, preserve the exact system you evaluated for deployment.

6. Deploy with a versioned rollback path

A deployed fine-tune includes more than an adapter file. Record the base-model revision, tokenizer, chat template, adapter artifact, serving configuration and prompt together. Also retain the dataset version and evaluation results used to approve it.

Test that bundle through the actual serving path. Changes in quantization, generation settings or formatting can affect behavior, so a notebook result alone is insufficient.

Introduce the candidate gradually and watch the same failure categories used during evaluation. Keep the previous bundle available so a regression has a straightforward recovery path.

Fine-tuning an LLM system becomes a repeatable engineering process when each run answers a specific question. For the ticket router, the question is whether adaptation routes real requests more reliably at an acceptable cost. A model that finishes training has completed a job. A model that passes those checks has earned a place in the application.