# What is knowledge distillation?

Knowledge distillation trains a small student model to reproduce the outputs of a larger teacher model. The student is cheaper to run and keeps much of the teacher's quality on the tasks it was distilled on.

## How it works

In [supervised fine-tuning](/glossary/supervised-fine-tuning) the target at each position is one correct token. In distillation it is the teacher's whole probability distribution over the next token. It shows which other tokens were plausible, so each position carries more signal. The student is trained to match it, usually with a Kullback-Leibler (KL) divergence loss mixed with cross-entropy. The term comes from [Hinton, Vinyals and Dean (2015)](https://arxiv.org/abs/1503.02531).

With language models there are two forms:

- **From logits.** Both models read the same text, and the student matches the teacher's distribution. They must share a vocabulary.
- **From text.** The teacher writes answers and the student is fine-tuned on them as [synthetic data](/glossary/synthetic-data). It needs the teacher's text only, not its logits.

## When to use it

Distill when a large model does the job and a smaller one must serve it, for cost or latency. With no larger teacher, see [self-distillation](/glossary/self-distillation).

## In Halo

Teacher distillation is the logit form. Both models sit in memory, and their vocabulary sizes must match:

```yaml
model_name_or_path: Qwen/Qwen3.5-9B          # student
teacher_model: Qwen/Qwen3.6-35B-A3B          # teacher, same tokenizer family
distill_loss: kl_divergence
distill_alpha: 0.5                           # even split between distillation and hard-label CE
```

Start it with `halo launch teacher-distill`. The shipped recipe puts [LoRA](/glossary/lora) (low-rank adaptation) on the student to save memory. See the [distillation guide](/halo/docs/training-methods/distillation#teacher-distillation).

