Online distillation (SDPG)

Online distillation (SDPG) in Halo is the on-policy form of self-distillation. The model generates answers, earns verifiable rewards for them, and is also distilled toward its own answer-hinted forward pass.

How it works

Elsewhere the term can also mean a teacher and a student trained at the same time. This page is about Halo's method, SDPG.

SDPG is online GRPO (Group Relative Policy Optimization) plus the teacher term of self-distillation. Each round:

  1. The model generates rollouts on a separate inference server.
  2. The rollouts earn verifiable rewards, checked against the dataset's answer, and GRPO turns them into advantages.
  3. The same weights run again on the prompt plus a hint that reveals the answer. The model is distilled toward that hinted forward pass.

The training text is on-policy: generated live by the current weights, not read from a fixed dataset.

When to use it

Reach for it when the model solves too few prompts for GRPO's group signal to say anything on its own.

In Halo

SDPG is a switch on the online GRPO trainer. Everything that trainer needs applies unchanged: a running rollout server, weight sync, and a dataset with prompt and answer. No shipped config turns it on. Start from the template and flip the gate, here with the trainer on GPUs 0-6 and the server holding GPU 7:

Bash

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 halo launch rlvr \
    examples/grpo/online/rlvr-online-grpo-template.yaml -n 7 \
    --use_sdpg=true --sdpg_beta_base=1.0

SDPG runs two extra full-vocabulary forwards per micro-batch, so it runs out of memory at batch sizes online GRPO survives. See the distillation guide.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL