Model distillation, or knowledge distillation, is a training technique in which a smaller "student" model learns to reproduce the outputs of a larger "teacher" model. The student keeps much of the teacher's quality on the tasks it was trained for, while running faster and at a fraction of the cost per request.
The trade is breadth for efficiency. A large general model can do thousands of things reasonably well. A distilled model does a few things the company cares about, such as classifying support intents or extracting invoice fields, nearly as well as the teacher, at a speed and price that work at millions of requests. It is one of the main ways to bring inference cost down without giving up accuracy on the tasks that matter.
How a model is distilled
The idea predates large language models. Hinton and colleagues showed that a single small network could learn from the full probability outputs of a large model or an ensemble, which carry more information than the right answer alone, and named the technique distillation.1 With LLMs the usual recipe is simpler: run the teacher on many real inputs, keep its best answers, and fine-tune the student on them.
Teacher
A large, capable model, often a frontier model called through an API or a large open-weight model.
Student
A smaller model, often a few billion parameters, that can run on modest hardware or the company's own infrastructure.
Transfer data
Inputs that match production, with the teacher's outputs. Some methods also keep the teacher's step-by-step reasoning as extra training signal.3
Evaluation set
Held-out real cases with known correct answers, used to compare the student with the teacher before launch.
One training example from a teacher
Suppose a marketplace wants a fast model that reads each incoming message and returns the intent, the order number, and whether a person is needed. A large model labels tens of thousands of past messages; each becomes one line of training data for the student.
{"messages": [
{"role": "system", "content": "Classify the customer message. Return JSON with intent, order_id, needs_human."},
{"role": "user", "content": "Seller never shipped order 7712-AB and isn't answering. I want my money back."},
{"role": "assistant", "content": "{\"intent\": \"buyer_protection_claim\", \"order_id\": \"7712-AB\", \"needs_human\": false}"}
]}Illustrative example in the chat format most fine-tuning APIs accept. The intent labels and order format are placeholders a company would define.
Once trained, the student handles the routing step in a fraction of the time. The large model stays in the system for the harder work: writing the reply and deciding what to do with unusual cases.
Distillation and other ways to shrink or adapt a model
| Technique | What changes | Main gain | Main limit |
|---|---|---|---|
| Distillation | A new, smaller model learns from a larger one | Speed and cost on a defined task | Weaker outside the tasks it was trained on |
| Fine-tuning | An existing model is trained further on your examples | Behavior and format fit your use | Needs labeled data; size and cost stay the same |
| Quantization | The same model's weights are stored with fewer bits | Less memory, faster serving | Some loss of accuracy, especially at low precision |
| Small language model | A compact model, often itself distilled | Runs cheaply or on your own hardware | Less general knowledge and reasoning |
| RAG | Nothing in the model; documents are added to the prompt | Current, citable knowledge | Does not make the model faster or cheaper |
Distillation is a form of fine-tuning where the training labels come from a teacher model instead of people. The techniques combine: a distilled model is often quantized for serving.
When distillation pays off
It suits tasks that are narrow, frequent, and stable, where a large model already does well and the volume makes its cost or latency a problem.
Triage every message
Classify intent, language, and urgency on every incoming contact in milliseconds, before the main agent starts.
message → student → routeExtract document fields
Pull amounts, dates, and policy numbers from claims and invoices at volume, with the teacher reviewing low-confidence cases.
document → student → check → postAnswer within a breath
Handle the first turns of a call, such as identifying the caller's request, where a pause feels broken.
speech → fast model → replyKeep data in-house
Run a distilled open-weight model on the company's own infrastructure, so regulated data never reaches an outside API.
own servers → student → no external callDistillation risks and controls
Copies style, not substance
ControlStudents can sound like the teacher while missing its accuracy. One study found imitation models fooled human raters but closed little of the gap on tasks not well covered by the imitation data.5 Test on held-out real cases with automatic checks, not only human impressions.
License terms
ControlSome providers forbid using their outputs to train competing models; Anthropic's commercial terms are one example.6 Check the teacher's terms before you collect training data.
Narrow competence
ControlThe student fails on inputs unlike its training data. Detect low confidence and unusual requests, and send them to the larger model or a person.
Personal data in training sets
ControlReal tickets contain names, addresses, and payment details. Mask or remove personal data before it becomes training data, and record the legal basis for using it.
Drift over time
ControlNew products, policies, and phrasing make the student stale. Rerun the evaluation set regularly and retrain when scores drop.
Common questions
What is the difference between distillation and fine-tuning?
Fine-tuning trains a model further on examples. Distillation is a kind of fine-tuning in which the examples come from a larger teacher model, with the goal of a smaller, cheaper model that behaves like the teacher on a task.
Is a distilled model as good as the original?
On the tasks it was trained for, it can come close and sometimes match it. Outside those tasks it is usually clearly weaker. Measure it on your own cases before you rely on it.
Is distilling another company's model legal?
It depends on that provider's terms. Some forbid using outputs to build competing models. Open-weight model licenses vary too, so check before collecting data.
How much data does distillation need?
It depends on the task. A narrow classifier may need thousands of examples; broader skills need far more. Methods that train on the teacher's reasoning can reduce the amount needed.
References
- Hinton, G., Vinyals, O. and Dean, J. Distilling the Knowledge in a Neural Network. NIPS 2014 Deep Learning Workshop, 2015.
- Sanh, V. et al. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. NeurIPS 2019 EMC2 Workshop.
- Hsieh, C.-Y. et al. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. Findings of ACL 2023.
- DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. 2025.
- Gudibande, A. et al. The False Promise of Imitating Proprietary LLMs. 2023.
- Anthropic. Commercial Terms of Service, section D.4.