Distillation
Distillation fits a student to a teacher's next-token distribution instead of to hard labels, which carries more signal per token than plain SFT. Halo has three variants. They differ in where the teacher comes from and whether the data is fixed or generated live.
Expert, tensor and expert-tensor parallelism apply to the student in all three; none takes context or
pipeline parallelism. The MoE recipes are under examples/distillation/gptoss/ and
examples/distillation/gemma4/.
Teacher distillation
Data is the SFT conversation format. Both models sit on every rank, the teacher as weights only, so memory is the binding constraint and the shipped recipe puts LoRA on the student. Before any teacher weight loads, the script raises unless both tokenizers map every token to the same id: in practice teacher and student share a tokenizer family.
Bash
halo launch teacher-distill \
examples/distillation/qwen3_5/distill-qwen3.5-9b-from-qwen3.6-35b-a3b.yaml -n 8distill_alpha weights the divergence against the cross-entropy term, and distill_loss picks the
divergence; cosine_similarity tolerates a teacher whose logit scale differs, and jensen_shannon
moves from forward KL (distill_jsd_beta: 0) to reverse KL (1).
Self-distillation
No second model and no generation. The same weights run twice per step: once on the prompt, and once on the prompt plus a hint that reveals the gold answer. The hinted forward is the teacher, and the student is pulled toward it on top of an ordinary SFT loss. Use it when nothing stronger than your own model is available but the dataset has answers.
Bash
halo launch self-distill examples/distillation/qwen3_5/self-distill-qwen3.5-9b.yaml -n 8max_lengthneeds headroom for the hint: neither branch is truncated, so an over-length row raises.assistant_message_templatemust byte-match what the chat template renders, andtrain_on_completions_onlymust stay on while the distillation term is.reference_kl_coef: 0loads no reference model. Raising it costs a second resident model, unsharded under expert or tensor parallelism.
Online SDPG
SDPG is Online GRPO plus the same answer-hinted teacher term. Reach for it when the
policy solves too few prompts for GRPO's group signal to work on its own. It needs everything the online
trainer does: a running rollout server and the {"prompt", "answer"} dataset.
No shipped config turns it on; start from the template, which carries the block commented out:
Bash
# trainer on GPUs 0-6, the vLLM server holding GPU 7
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 halo launch grpo/online \
examples/grpo/online/rlvr-online-grpo-template.yaml -n 7 \
--use_sdpg=true --sdpg_beta_base=1.0SDPG adds a no-grad teacher forward on the longer hinted sequence and keeps the student's and the teacher's logits alive until backward, so it runs out of memory at batch sizes Online GRPO survives.
What to watch
- Teacher distillation:
distillation_lossshould fall whilesft_lossstays sane. An OOM on the first step means both models did not fit; cutmax_lengthor move the student to LoRA. - Self-distillation and SDPG: watch
opd_losswith the live coefficientopd_beta. A coefficient pinned at zero means the schedule never ramped. - SDPG: an
opd_lossof exactly zero means no rollout in the batch earned a positive advantage. That is expected now and then, and a broken verifier if it persists.
Every knob, loss and refusal: Teacher ↗ · Self ↗ · Online SDPG ↗.
Source on GitHubUpdated 18 November 2026