What AI distillation keeps, and what a smaller model can lose

One model learns from another

AI distillation trains a model using signals from another model’s behaviour. The source is usually called the teacher and the learner the student. Often the student is smaller, with the aim of handling particular tasks using fewer resources. That does not mean it receives a complete copy of everything the teacher can do.

An influential paper by Hinton, Vinyals and Dean describes transferring knowledge from expensive systems into something easier to deploy. With language models, training can use generated answers or richer information about which possible continuations the teacher considers likely. It does not require copying the teacher’s parameters, the internal values adjusted during training.

Imagine sorting customer emails

Suppose a team wants to sort incoming messages into billing, technical problems and address changes. A large model could generate examples and help supply the training signals for a smaller one. If the student learns the task well, asking the large model about every message may be unnecessary.

This is a hypothetical example, not a performance result. The student might become good at sorting those emails while remaining poor at summarizing a contract. Learning one task does not establish that it has inherited the teacher’s other abilities. Lower total cost is not automatic either: preparation, training and usage volume all count.

A convincing teacher can still teach mistakes

Training material needs checking. Errors or gaps in the teacher’s examples can become patterns the student learns. Nor does every method rely only on a fixed set of completed answers. The TRL documentation describes approaches where the teacher provides feedback on outputs generated by the student itself.

Evaluation should include examples withheld from training. For the email example, we would include ambiguous requests, spelling mistakes and messages raising two issues at once. Those are useful test-design suggestions, not a claim that a short checklist establishes reliability.

When a distilled model promises impressive performance, ask which tasks were tested and how. A percentage without that context cannot tell you whether it is ready to replace the teacher in your workflow.

Distillation is not quantization

Quantization lowers the numerical precision used to represent model values, reducing resource requirements. Distillation is training that uses signals from another model. The two techniques can be combined, but they describe different changes. Our 4-bit and 8-bit AI guide covers the numerical side.

Distillation can also take place during fine-tuning. Fine-tuning describes additional training to adapt a model. Distillation tells you where some of its teaching signals come from. When choosing a smaller AI model, ask for task-specific results, memory requirements and actual cost. “Distilled” alone answers none of those questions.