AI Glossary Training and customization

Model distillation

  • Tidio Agents team
  • Updated November 19, 2026
  • 5 min read
  • 6 sources
Definition

Model distillation, or knowledge distillation, is a training technique in which a smaller "student" model learns to reproduce the outputs of a larger "teacher" model. The student keeps much of the teacher's quality on the tasks it was trained for, while running faster and at a fraction of the cost per request.

The trade is breadth for efficiency. A large general model can do thousands of things reasonably well. A distilled model does a few things the company cares about, such as classifying support intents or extracting invoice fields, nearly as well as the teacher, at a speed and price that work at millions of requests. It is one of the main ways to bring inference cost down without giving up accuracy on the tasks that matter.

How it works

How a model is distilled

The idea predates large language models. Hinton and colleagues showed that a single small network could learn from the full probability outputs of a large model or an ensemble, which carry more information than the right answer alone, and named the technique distillation.1 With LLMs the usual recipe is simpler: run the teacher on many real inputs, keep its best answers, and fine-tune the student on them.

Real inputsTeacher modelFiltered outputsTrain studentEvaluategaps found: add examplesdeploy studenttickets · documents · conversations
The evaluation step decides everything. The student is only as good as the examples it saw and the tests it passed.

Teacher

A large, capable model, often a frontier model called through an API or a large open-weight model.

Student

A smaller model, often a few billion parameters, that can run on modest hardware or the company's own infrastructure.

Transfer data

Inputs that match production, with the teacher's outputs. Some methods also keep the teacher's step-by-step reasoning as extra training signal.3

Evaluation set

Held-out real cases with known correct answers, used to compare the student with the teacher before launch.

Example

One training example from a teacher

Suppose a marketplace wants a fast model that reads each incoming message and returns the intent, the order number, and whether a person is needed. A large model labels tens of thousands of past messages; each becomes one line of training data for the student.

jsonl · one training record
{"messages": [
  {"role": "system", "content": "Classify the customer message. Return JSON with intent, order_id, needs_human."},
  {"role": "user", "content": "Seller never shipped order 7712-AB and isn't answering. I want my money back."},
  {"role": "assistant", "content": "{\"intent\": \"buyer_protection_claim\", \"order_id\": \"7712-AB\", \"needs_human\": false}"}
]}

Illustrative example in the chat format most fine-tuning APIs accept. The intent labels and order format are placeholders a company would define.

Once trained, the student handles the routing step in a fraction of the time. The large model stays in the system for the harder work: writing the reply and deciding what to do with unusual cases.

Compared

Distillation and other ways to shrink or adapt a model

TechniqueWhat changesMain gainMain limit
DistillationA new, smaller model learns from a larger oneSpeed and cost on a defined taskWeaker outside the tasks it was trained on
Fine-tuningAn existing model is trained further on your examplesBehavior and format fit your useNeeds labeled data; size and cost stay the same
QuantizationThe same model's weights are stored with fewer bitsLess memory, faster servingSome loss of accuracy, especially at low precision
Small language modelA compact model, often itself distilledRuns cheaply or on your own hardwareLess general knowledge and reasoning
RAGNothing in the model; documents are added to the promptCurrent, citable knowledgeDoes not make the model faster or cheaper

Distillation is a form of fine-tuning where the training labels come from a teacher model instead of people. The techniques combine: a distilled model is often quantized for serving.

In practice

When distillation pays off

It suits tasks that are narrow, frequent, and stable, where a large model already does well and the volume makes its cost or latency a problem.

Customer support

Triage every message

Classify intent, language, and urgency on every incoming contact in milliseconds, before the main agent starts.

message → student → route
Back office

Extract document fields

Pull amounts, dates, and policy numbers from claims and invoices at volume, with the teacher reviewing low-confidence cases.

document → student → check → post
Voice agents

Answer within a breath

Handle the first turns of a call, such as identifying the caller's request, where a pause feels broken.

speech → fast model → reply
Compliance

Keep data in-house

Run a distilled open-weight model on the company's own infrastructure, so regulated data never reaches an outside API.

own servers → student → no external call
Risks

Distillation risks and controls

Copies style, not substance

ControlStudents can sound like the teacher while missing its accuracy. One study found imitation models fooled human raters but closed little of the gap on tasks not well covered by the imitation data.5 Test on held-out real cases with automatic checks, not only human impressions.

License terms

ControlSome providers forbid using their outputs to train competing models; Anthropic's commercial terms are one example.6 Check the teacher's terms before you collect training data.

Narrow competence

ControlThe student fails on inputs unlike its training data. Detect low confidence and unusual requests, and send them to the larger model or a person.

Personal data in training sets

ControlReal tickets contain names, addresses, and payment details. Mask or remove personal data before it becomes training data, and record the legal basis for using it.

Drift over time

ControlNew products, policies, and phrasing make the student stale. Rerun the evaluation set regularly and retrain when scores drop.

FAQ

Common questions

What is the difference between distillation and fine-tuning?

Fine-tuning trains a model further on examples. Distillation is a kind of fine-tuning in which the examples come from a larger teacher model, with the goal of a smaller, cheaper model that behaves like the teacher on a task.

Is a distilled model as good as the original?

On the tasks it was trained for, it can come close and sometimes match it. Outside those tasks it is usually clearly weaker. Measure it on your own cases before you rely on it.

Is distilling another company's model legal?

It depends on that provider's terms. Some forbid using outputs to build competing models. Open-weight model licenses vary too, so check before collecting data.

How much data does distillation need?

It depends on the task. A narrow classifier may need thousands of examples; broader skills need far more. Methods that train on the teacher's reasoning can reduce the amount needed.

Sources

Tidio Agents

Build an agent that resolves, not only answers

We design and build custom AI agents for customer service, shopping and back-office work, on your infrastructure, with the code handed over to you. Projects start with a 21-day private AI audit.

Request a call