Self-distillation trains a model with itself as the teacher, so no second model is needed. In the form Halo uses, the model reads the prompt with the answer revealed, and its plain forward pass is pulled toward that hinted one.
How it works
Knowledge distillation needs a stronger teacher. Self-distillation makes one out of the student by giving it more to read. The same weights run twice per step:
- Once on the prompt. This is the student.
- Once on the prompt plus a hint that reveals the gold answer. This is the teacher.
The student is pulled toward the teacher on the response tokens the two share, on top of an ordinary supervised fine-tuning loss. There is no second model and no generation.
When to use it
Use it when nothing stronger than your own model is available but the dataset has answers. It reads conversations plus an answer column.
In Halo
The shipped recipe sets:
YAML
sdpg_hint_template: "\n[Hint] The correct answer is: {answer}. Do NOT state that you were given the answer.\n"
sdpg_loss: reverse_kl
sdpg_beta_warmup_steps: 50
sdpg_beta_decay_steps: 100The hint is filled from the column named by sdpg_answer_field (default answer). The warmup and decay steps ramp the distillation term in after the supervised term has settled and phase it out near the end. max_length needs headroom for the hint, because neither branch is ever truncated.
Bash
halo launch self-distill examples/distillation/qwen3_5/self-distill-qwen3.5-9b.yaml -n 8See the distillation guide. Online distillation applies the same idea to text the model generates during training.
