Online distillation (SDPG) in Halo is the on-policy form of self-distillation. The model generates answers, earns verifiable rewards for them, and is also distilled toward its own answer-hinted forward pass.
How it works
Elsewhere the term can also mean a teacher and a student trained at the same time. This page is about Halo's method, SDPG.
SDPG is online GRPO (Group Relative Policy Optimization) plus the teacher term of self-distillation. Each round:
- The model generates rollouts on a separate inference server.
- The rollouts earn verifiable rewards, checked against the dataset's answer, and GRPO turns them into advantages.
- The same weights run again on the prompt plus a hint that reveals the answer. The model is distilled toward that hinted forward pass.
The training text is on-policy: generated live by the current weights, not read from a fixed dataset.
When to use it
Reach for it when the model solves too few prompts for GRPO's group signal to say anything on its own.
In Halo
SDPG is a switch on the online GRPO trainer. Everything that trainer needs applies unchanged: a running rollout server, weight sync, and a dataset with prompt and answer. No shipped config turns it on. Start from the template and flip the gate, here with the trainer on GPUs 0-6 and the server holding GPU 7:
Bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 halo launch rlvr \
examples/grpo/online/rlvr-online-grpo-template.yaml -n 7 \
--use_sdpg=true --sdpg_beta_base=1.0SDPG runs two extra full-vocabulary forwards per micro-batch, so it runs out of memory at batch sizes online GRPO survives. See the distillation guide.
