Distillation
Fit a student to a teacher's token-level output distribution. The three methods differ in where the teacher comes from and whether training is on- or off-policy.
Which to pick:
- A separate, stronger teacher is available → teacher distillation.
- One model, gold answers in the dataset, no generation budget → self-distillation.
- One model, a verifier, and a vLLM rollout budget → online SDPG.
All three take EP, TP, ETP, EP+TP and EP+ETP on the student. None takes CP or PP: each needs a second whole-model forward that neither the sequence split nor the stage split can reproduce.
Every divergence, the gold-token gate and the hard-label cross-entropy score the logits one token chunk at a time under a checkpoint (token_divergence in halo/trainers/distillation/divergences.py), so backward holds each forward's logits and no fp32 [tokens, vocab] plane; the peak beyond the models is those logits planes.
Dataset
Teacher distillation and self-distillation read the SFT conversation format under conversation_field — default messages for teacher distillation, prompt for self-distillation, so set it to match your data.
JSON Lines
{"messages": [{"role": "user", "content": "Explain gravity."}, {"role": "assistant", "content": "Gravity is a fundamental force..."}]}Self-distillation also reads a gold-answer column (sdpg_answer_field, default answer) to build the teacher's hint. Online SDPG reads the online-GRPO {"prompt", "answer"} format, where the answer feeds both the verifier reward and the hint.
Source on GitHubUpdated 18 November 2026