TL;DR: Knowledge distillation in AI is a training technique where a small “student” model learns to copy a large “teacher” model. The student is trained on the teacher’s outputs, often its full probability scores rather than just the right answer, so it picks up much of the teacher’s skill at a fraction of the size and cost. It is one of the main reasons fast, cheap AI models exist, from DistilBERT to the small reasoning models DeepSeek released in 2025. The catch is that a student is rarely as good as its teacher, and distilling someone else’s model can break their terms of service.
What is knowledge distillation in AI, and why does almost every AI lab use it? The short answer: the biggest models are expensive to run, and distillation is a way to pack much of what they know into a model small enough to answer quickly on a server, a laptop or even a phone. This guide explains how it works in plain English, the main types, real examples and its limits.
What is knowledge distillation?
Knowledge distillation is a form of model compression. You start with a large, accurate model (the teacher) and use it to train a smaller model (the student). Instead of learning only from labeled data, the student learns to reproduce the teacher’s behavior.
The idea goes back to a 2006 paper on “Model Compression” by Cristian Buciluă, Rich Caruana and Alexandru Niculescu-Mizil, who showed that a big ensemble of models could be compressed into one smaller model. The name “knowledge distillation” was popularized in 2015 by Geoffrey Hinton, Oriol Vinyals and Jeff Dean in “Distilling the Knowledge in a Neural Network”, which is still the standard reference.
If you are new to how these models work, start with our guide to what a large language model is.
How knowledge distillation works, step by step
- Train or pick a teacher. This is a large, high-quality model, for example a frontier LLM or a big image classifier.
- Run the teacher on training data. For each input, record its outputs. In classic distillation that means the full probability distribution over possible answers, not just its top pick.
- Train the student to match. The student, a much smaller network, is trained to make its outputs as close as possible to the teacher’s. Usually this is combined with ordinary training on the true labels.
- Deploy the student. The student is cheaper and faster to run, which is the whole point.
Why “soft labels” matter
Say an image model sees a photo of a husky. The correct label just says “husky.” The teacher’s output might say 85% husky, 12% wolf, 2% malamute and almost 0% toaster. Hinton and colleagues called this extra information “dark knowledge”: it tells the student which wrong answers are reasonable mistakes. A student trained on these soft labels learns a richer picture of the problem than one trained on bare right-or-wrong labels.
To bring out that information, the 2015 paper uses a temperature setting that softens the teacher’s probabilities, so small scores like “12% wolf” carry more weight during training.
Main types of knowledge distillation
| Type | What the student copies | Typical use |
|---|---|---|
| Response-based (logit) distillation | The teacher’s final output probabilities | Classic method from the 2015 paper; classifiers, language models |
| Feature-based distillation | The teacher’s internal layer activations | Computer vision; when the student’s architecture is similar |
| Relation-based distillation | How the teacher relates different examples or layers | Research settings, embedding models |
| Sequence-level / synthetic-data distillation | Whole text answers the teacher generates | Modern LLMs: the teacher writes training examples, the student is fine-tuned on them |
With modern chatbots, the last type is the most common. The teacher model produces large numbers of high-quality answers, sometimes with step-by-step reasoning, and the student is fine-tuned on them. This works even when you can’t see the teacher’s internal probabilities, which is why it can be done through an API.
Real-world examples
- DistilBERT (2019). Hugging Face researchers reported in their DistilBERT paper that the distilled model was 40% smaller and 60% faster than BERT while keeping 97% of its language-understanding performance on their benchmarks.
- DeepSeek-R1 distilled models (2025). Alongside its R1 reasoning model, DeepSeek released six smaller dense models, from 1.5 billion to 70 billion parameters and built on Qwen and Llama, that it says were distilled from R1’s reasoning outputs (see the DeepSeek-R1 repository).
- Distillation as a product. OpenAI added a model distillation workflow to its API in 2024, letting developers store a large model’s outputs and use them to fine-tune a smaller one.
- “Mini,” “Flash” and “Lite” models. Many labs describe their small, fast models as trained with help from bigger ones. The details vary and are often not published, but distillation is widely used to build these tiers.
Why distillation matters
- Lower cost. Smaller models need less compute per answer, which cuts the cost of inference.
- Speed. Fewer parameters usually means faster responses.
- On-device AI. A distilled model can fit on phones and laptops, which helps privacy and works offline.
- Specialization. A student can be trained to match the teacher on just one task, such as customer support or code review, and do it well with a tiny model.
Distillation vs. quantization vs. pruning
These three techniques are often confused because they all make models cheaper:
- Distillation trains a new, smaller model to imitate a big one.
- Quantization keeps the same model but stores its numbers at lower precision, such as 4-bit or 8-bit.
- Pruning removes weights or whole parts of the model that contribute little.
They can be combined: a team might distill a model, then quantize the student to run on a phone. A related efficiency approach, mixture of experts, takes a different route by activating only part of a large model for each token.
Limits and controversies
The student rarely beats the teacher
A distilled model usually trails its teacher, especially on hard, open-ended tasks. It may match the teacher on common benchmarks while doing worse on unusual inputs, and it can inherit the teacher’s errors and biases.
Distilling someone else’s model
Because sequence-level distillation only needs a model’s text outputs, it is possible to distill a competitor’s model through its API. Major AI providers’ terms of service generally prohibit using their outputs to build competing models. In early 2025, OpenAI said it was reviewing indications that DeepSeek may have improperly used its models’ outputs; DeepSeek did not confirm this. Whatever the facts of any single case, “unauthorized distillation” has become a real legal and competitive issue.
FAQ
What is knowledge distillation in AI in simple terms?
It is teaching a small AI model to imitate a big one, so you get much of the big model’s ability at lower cost and higher speed.
Who invented knowledge distillation?
The idea appeared in 2006 work on model compression by Buciluă, Caruana and Niculescu-Mizil. Hinton, Vinyals and Dean named and popularized “knowledge distillation” in 2015.
Is a distilled model the same as a fine-tuned model?
Not exactly. Fine-tuning is the training step; distillation is a strategy where the training data or targets come from a teacher model. Many modern distilled LLMs are produced by fine-tuning a small model on a big model’s answers.
The Teck Hustlers Editorial Team covers technology, business, finance and gaming news for readers who want the story behind the headline. Every article is researched, written and edited by our in-house team, checked against company announcements, filings and reputable reporting, and updated when new facts emerge. We focus on what each development means for founders, professionals, investors and everyday users. Have a tip or a correction? Reach us through our contact page.
