Model Distillation Explained: Teachers and Students

Model distillation is a technique for training a small AI model to imitate a larger, more capable one, so that you get much of the quality at a fraction of the size, cost and delay. The large model acts as a teacher and the small one as a student. It is one of the main reasons fast, cheap models can now handle tasks that needed a very large system not long ago. This guide explains how model distillation works, where it helps, where it falls short and why it matters when an assistant chooses a model for your question.

No maths is required. If you can follow an analogy about learning from a good teacher rather than from a textbook, you can follow distillation. We also cover the questions people actually ask: whether a distilled model is “just a copy”, whether it can match the teacher, how it differs from quantization and fine-tuning, and what it means for the price you pay.

The idea in one picture

Imagine a student who learns a subject two ways. In the first, they read the textbook and are told only the final answer to each practice question: “B”. In the second, they sit with an expert who explains how likely each answer is and why the wrong ones are tempting. “Mostly B, but C is a reasonable mistake because the two are similar.” The second student learns more from the same questions, because the teacher passes on its sense of how things relate.

That is the heart of distillation. A large model, trained at great expense, produces not only answers but a spread of probabilities across possible answers. The small model is trained to reproduce those outputs. The method was described in a well-known 2015 paper, Distilling the Knowledge in a Neural Network, and the overview of knowledge distillation summarises the field.

Model distillation explained step by step

Train or choose a strong teacher. This is a large model that already does the task well.
Prepare inputs. A set of prompts, documents or examples that represent the work the student should do.
Collect the teacher’s outputs. For each input, record the answer, and often the probabilities or the written reasoning behind it.
Train the student. The smaller model is adjusted until its outputs resemble the teacher’s, often alongside the original correct labels where they exist.
Evaluate. Test the student on fresh examples and compare it with the teacher on quality, speed and cost.
Iterate. Add examples where the student fails, adjust the size or training and repeat.

The result is a model that is faster to run and cheaper to serve, usually with some loss in ability. The art lies in choosing what to keep.

Types of distillation

Teachers can pass knowledge in several ways. The distinctions matter because they decide what the student can and cannot inherit.

Type	What the student learns from	Strength	Limit
Output (soft label)	The teacher’s probability spread over answers	Rich signal about how options relate	Needs access to the teacher’s internal outputs
Response-based text	Teacher-written answers to many prompts	Works with any teacher you can query	Copies style and errors along with substance
Reasoning traces	Step-by-step explanations written by the teacher	Can teach multi-step problem solving	Student may imitate the form without the skill
Feature or layer matching	Internal representations at chosen layers	Transfers deeper structure	Needs similar architectures and full access
Self-distillation	A model’s own earlier outputs	Can smooth and stabilise	Limited by the model’s own mistakes

For language assistants, the second and third types are the most common in practice: a strong model writes many examples, and a smaller one is trained on them.

Why distillation matters

Speed

Smaller models answer faster, because fewer calculations are needed per word generated. For chat that feels instant, speed matters as much as quality. Our guide to AI latency explains why.

Cost

Running a very large model for every message is expensive. A distilled model costs less per answer, which is why services can offer generous free tiers or lower point costs for everyday questions. It is also why an assistant that routes simple requests to a small model and hard ones to a larger one can serve many users sustainably.

Running closer to the user

Small models can run on a laptop, a phone or a modest server. That helps with privacy, offline use and cost control. Our overview of small language models covers where they shine.

Specialisation

A distilled model can be built for one job, such as classifying support tickets, extracting fields from invoices or writing product descriptions in a given style, with very high accuracy for that job at low cost.

What distillation cannot do

Distillation is powerful, and it has limits you should understand before trusting a small model with something important.

The student rarely exceeds the teacher. It learns to imitate. Where the teacher is wrong, the student is usually wrong too, and often more confidently.
Breadth shrinks. A small model has less capacity. It may match the teacher on the tasks it was trained for and fall behind on rare topics, long documents or unusual instructions.
Hard reasoning is the weak spot. Multi-step problems, tricky maths and careful planning are where size tends to matter most.
Errors are inherited. Biases, outdated knowledge and invented facts in the teacher flow to the student. Read our pieces on AI bias and AI hallucinations for how these behave.
Quality depends on the examples. A student trained on a narrow or repetitive set of teacher outputs becomes narrow and repetitive.

Distillation versus quantization, fine-tuning and mixture of experts

Several techniques make models smaller, cheaper or more specialised, and they are easy to confuse. They also combine well.

Technique	What it changes	Needs a teacher?	Typical goal
Distillation	Trains a different, smaller model to imitate a larger one	Yes	Smaller, faster model with similar behaviour
Quantization	Stores the same model’s numbers with less precision	No	Less memory and faster running
Fine-tuning	Adjusts an existing model on new examples	No	Specialise behaviour, style or format
Mixture of experts	Uses only part of a large model for each request	No	Large capacity at lower running cost

Details are in our explainers on quantization, fine-tuning versus prompting and mixture of experts. A real-world model may be distilled, then quantized, then fine-tuned for a specific use.

Common myths about distillation

A few misunderstandings come up again and again, and clearing them helps you read product claims sensibly.

“Distilled means the same quality, only cheaper.” Usually it means close on the tasks that were measured, with gaps elsewhere. Ask what was tested.
“Small models are always worse.” A small model trained for one job can beat a general model at that job, and it is often far faster. The right comparison is on your own task.
“The teacher’s knowledge is copied exactly.” The student sees only the teacher’s outputs on the examples chosen, so whatever those examples miss is missing from the student too.
“Distillation is a trick to avoid training costs.” It moves the cost: the teacher still had to be trained, and generating good examples takes work and careful filtering.

Distillation, rules and permission

Because distillation uses a teacher’s outputs as training material, it raises questions about permission. Many providers’ terms restrict using their model’s outputs to train a competing model, and some forbid it outright. If you plan to distil from a service you do not own, read the terms carefully and, for commercial work, take advice. Training on a teacher you built, or one released under a licence that allows it, avoids the problem. Distilled models also inherit data issues from the teacher’s training, so provenance matters for any organisation with compliance duties.

What this means when you use an assistant

You rarely choose a model yourself, and that is the point of routing. Ask Mio’s pages state that Mio routes each request to the best model for the task: fast models for chat, coding models for code, long-context models for documents and image models for design. Smaller, faster models are well suited to everyday chat, and larger ones to hard problems and long documents. Our guide to model routing explains how that choice is made, and the article on AI inference covers what happens when a model produces an answer.

Practical consequences for you:

Simple tasks are cheap and quick. A normal chat reply costs 1 point on Ask Mio, according to the pricing page. Part of the reason that is possible is that smaller models can handle everyday requests well.
Hard tasks may cost more. A long or complex reply costs 3 points, and a coding answer 5 to 10, because they need more capable models and more output.
Check important answers. Whatever model answers, verify facts and numbers that matter. Mio’s Research mode searches the web and cites sources you can open.

A simple way to judge a distilled model

If you are choosing between a small model and a large one for your own project, run a simple test rather than relying on claims.

Collect fifty real examples of your task, including difficult and odd ones.
Run both models on them with the same instructions.
Score the results blind, ideally with someone who does not know which model wrote which.
Compare quality, speed and cost per thousand examples.
Look at the failures, not just the average. A model that is wrong rarely but badly may be worse than one that is wrong often but harmlessly.
Decide on a rule: use the small model where it passes, and send the rest to the larger one.

Our guide on how to evaluate an AI assistant describes a lightweight way to run this kind of comparison.

Frequently Asked Questions

Is a distilled model just a smaller copy of the big one?

Not exactly. It is a different, smaller model trained to imitate the teacher’s outputs, so it behaves similarly on the tasks it was trained for but has less capacity overall. It may match the teacher on common requests and fall behind on rare topics, long documents or difficult reasoning.

Can a distilled model be better than its teacher?

Occasionally on a narrow task, because it can be specialised and trained on carefully chosen examples. In general it learns to imitate, so it rarely exceeds the teacher’s overall ability, and it inherits the teacher’s mistakes. For broad tasks, expect it to come close rather than to surpass.

How is distillation different from quantization?

Distillation trains a new, smaller model to imitate a bigger one. Quantization keeps the same model but stores its numbers with less precision to save memory and time. They solve related problems in different ways and are often combined, with a model distilled first and then quantized for deployment.

Does distillation make AI cheaper for users?

Often, yes. Smaller models cost less to run and answer faster, so services can price everyday requests low and keep larger models for hard tasks. That is part of why routing simple questions to fast models and harder ones to stronger models can keep a plan affordable.

Is it allowed to distil from any model?

Not always. Many providers restrict using their outputs to train other models, and licences differ. Read the terms of any service you use as a teacher, and take advice for commercial projects. Using a model you own or one with a permissive licence avoids most problems.

Does Ask Mio use distilled models?

Ask Mio does not ask you to pick a model. It routes each request to what suits the task, using fast models for chat and stronger ones where needed. How a particular model was built is not something you have to manage, and you can judge the results on the free plan.

The Bottom Line

Model distillation explained simply: a small student model learns from a large teacher, trading some ability for much lower cost and delay. It explains why quick everyday answers are cheap, and why hard problems still call for larger systems. Test any small model on your own examples, remember that it inherits its teacher’s errors and check terms before distilling from a service you do not own. To see routing in practice, try the free plan and compare options on the Ask Mio pricing page.

Model Distillation Explained: Small Models, Big Teachers