What is knowledge distillation?

Knowledge distillation trains a small student model to reproduce the outputs of a larger teacher model. The student is cheaper to run and keeps much of the teacher's quality on the tasks it was distilled on.

How it works

In supervised fine-tuning the target at each position is one correct token. In distillation it is the teacher's whole probability distribution over the next token. It shows which other tokens were plausible, so each position carries more signal. The student is trained to match it, usually with a Kullback-Leibler (KL) divergence loss mixed with cross-entropy. The term comes from Hinton, Vinyals and Dean (2015).

With language models there are two forms:

  • From logits. Both models read the same text, and the student matches the teacher's distribution. They must share a vocabulary.
  • From text. The teacher writes answers and the student is fine-tuned on them as synthetic data. It needs the teacher's text only, not its logits.

When to use it

Distill when a large model does the job and a smaller one must serve it, for cost or latency. With no larger teacher, see self-distillation.

In Halo

Teacher distillation is the logit form. Both models sit in memory, and their vocabulary sizes must match:

YAML

model_name_or_path: Qwen/Qwen3.5-9B          # student
teacher_model: Qwen/Qwen3.6-35B-A3B          # teacher, same tokenizer family
distill_loss: kl_divergence
distill_alpha: 0.5                           # even split between distillation and hard-label CE

Start it with halo launch teacher-distill. The shipped recipe puts LoRA (low-rank adaptation) on the student to save memory. See the distillation guide.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL