MTP heads
mtp_loss_weight trains the multi-token-prediction head a family's release ships, as that release's architecture: the family's own decoder layers plus its projection and norms, at the release's key path. The SFT script alone carries the knobs (fields). At weight 0 nothing is built and no hook is installed.
Families
Every other tail is refused by name (MTP_REFUSALS, halo/models/mtp/spec.py): Flash-Next's draft layer reads hyper-connection streams, GLM-5.3's tail lacks layers its class builds and is stored in fp8, Step-3.7 owns a head per draft layer, Inkling needs renames, DeepSeek-V4 stores vendor names and quantized experts. A family with no tail gets a plain refusal. tests/cpu/models/test_mtp_head_spec.py holds every class declaring a tail to a spec or a refusal.
Ling-3.0-flash's remote forward runs its own head; see Bailing.
A Nemotron-H head is the chain both engines draft with: the norms and the projection, the attention block, the MoE block, the final norm, then the trunk's output embedding, with each block's residual folded before the next block's norm and no zeroed embedding at position 0. Each block takes its class from the trunk's last layer of its kind, and the attention block takes that layer's call, so packed rows keep their documents apart. A config declaring another mtp_layers_block_type, or a depth cut whose trunk holds no layer of a block's kind, refuses before the load. Super VL's text route reads language_model.mtp.* and writes mtp.*.
Objective
Index t reads the embedding of token t+1 and the trunk's hidden t, and predicts token t+2. The term is the cross-entropy at those targets, divided by the same count as the next-token loss, added inside the model's forward, so HF's later scale multiplies both alike. No draft scores across a document start under packing or padding-free rows.
mtp_loss_type: hardfits the gold token.mtp_loss_type: softis Soft-MTP. It keeps the loss value and the output embedding's gradient, and routes the head's gradient to the trunk's next-token distribution: a KL at the same step, with the routed term exactlymtp_loss_weightat any batch.mtp_train_scope: jointlets the term reach head, trunk, embedding and output embedding.mtp_train_scope: detachedtrains the head alone while the trunk takes the next-token gradient. The global clip norm still covers both.mtp_train_scope: head_onlyisdetachedover a frozen trunk, which runs with no graph.moe_balancing: autoresolves tononethere.
On joint and detached, a bias balancing mode takes the head's gate too: moe/* counts one more layer, and its slot stays the release's key. The router aux loss stays the trunk's, as the head's gate runs past the backbone forward that records it.
A run logs ntp_ce and mtp_ce, each cross-entropy per target over the world, beside loss, which holds both weighted over one normalizer. Under detached with max_grad_norm: 0, the trunk's weights match a run with no head byte for byte, and ntp_ce is the same under either head loss. ntp_ce is not a no-head run's loss, which is a mean of each rank's own average: on rows of unequal length the two differ, by up to 10% in the measured detached run, at equal weights.
Hard and soft
test_sft_mtp_head_real.py, Qwen3.6-35B at EP8: eval_mtp_acceptance_tv on 24 held-out first-turn rows of 256 tokens, at steps 0, 30, 60, 90, 120 and 150. A stale start is the Qwen3.5-35B head on the Qwen3.6 trunk; an own start is the release's head.

What hard does depends on the rate: from a stale start it climbs then falls at 1e-5 and holds at 1e-6, and from the release's own head it falls at all three. Only soft fits have been served, so this page names no winner.
On first-turn rows through --release, Nemotron-H's own head reads 0.8124 on Lightning and 0.8876 on Super VL before any fit. A soft fit at 1e-5 falls and the gate stops it (0.8076 and 0.8854). At 1e-6 Lightning holds at 0.8126 and the gate passes. Super VL did not run at that rate.
The same stale soft row ends at 0.8253 on the fused kernel and 0.8256 on FA4, and at EP2 at 0.8185 on FA2 and 0.8189 on FA4. On the trunk's own samples at temperature 1, an own start reads 0.8704, 0.8877, 0.8919, 0.8916, 0.8902, 0.8901. Under detached at 5e-6, acceptance follows the moving trunk over 200 steps: 0.8187 to 0.8158 soft, 0.8187 to 0.8136 hard.
Parallel axes
Step cost
joint and detached add the head's forward and backward and a second cross-entropy to a step. head_only drops the trunk's graph and its gradient checkpointing, and still runs the next-token cross-entropy for the metric; soft takes a gradient through it, so soft costs more than hard there and not where the trunk already trains on that loss. MFU's attention term walks the decoder list, so it counts a GLM-4 head's layer and not a mtp.* head's.
test_sft_mtp_head_cost.py, Qwen3.6-35B at EP8 on FA2: packed random tokens, 32,992 per rank per step, weight 0.3, gradient checkpointing except on head_only, the median of 50 steps.

The row's bars hold on these arms: joint at most 1.30x, soft within 2% of hard (0.9%, inside the quartiles), head_only at most 0.60x. Reserved memory is the allocator's cache filling a 268 GiB card, so read allocated for what a head adds. detached, head_only with hard, and the repeat of the no-head arm are not measured.
Loading
halo/model_loading/parallel/mtp_head.py runs once after the per-mode load. It builds the head on meta in a head-only shell and streams the tail through the lazy planner, so a per-expert or a fused tail both load, each rank reads its expert slice, and fp32 masters apply as they do on the trunk. The tail comes from a resumed checkpoint, then mtp_head_source, then the model's own. A missing, partial or extra tail and a quantized source are refused. init_from_scratch draws the head with the family's own init.
Checkpoints
A save walks the head as it walks the trunk, so a checkpoint holds it at the release's key path on every saver, and on every node's disk under per-node output storage. Its expert bank takes the family's gathered layout, which both engines read. A GLM-4 head's embed_tokens and shared_head.head are written off the trained embeddings, on one process too, where that save takes the gathered writer since save_pretrained writes the state dict alone. A stock load ignores the head. A Nemotron-H head's expert bank is per expert, so an untrained tail comes back byte for byte, its router bias in fp32.
A resume writes the bytes of a run that never stopped. It reads the checkpoint's head and ignores mtp_head_source; a resume built from the base (use_grouped_gemm: false) reads the usual source, and the FSDP2 restore overwrites it. A head cannot join a run midway: a checkpoint with none refuses at the load, or over a base load at the optimizer gate (have no saved optimizer state), which head_only to joint also raises. joint to head_only passes: the head's moments restore, and the trunk's go unread. A resume at mtp_loss_weight: 0 builds no head, so what it saves carries none; keep the last checkpoint that does.
Serving
vLLM reads the head through VLLM_SPECULATIVE_CONFIG, SGLang through --speculative-draft-model-path. The rollout weight sync carries none: the knobs live on the SFT script alone, every trainer that syncs refuses a head at construction, and a load through the stock class holds none, so the server drafts with the head of its launch checkpoint. See Rollout servers.
A head can follow a policy another trainer produced: a head_only fit with mtp_head_source runs over that trainer's export. Tiny rows from SMPO and offline GRPO exports on Qwen3.5 MoE, GLM-4 Lite and Nemotron-H keep the trunk's bytes and move the head. A stock reload loads the result, and reports GLM-4 Lite's head keys as unexpected, since its class builds no layer past num_hidden_layers.
test_sft_mtp_head_serving.py, 64 first-turn prompts at temperature 1, one draft token, by the engine's own counters, each against the release head on the same prompts:
Qwen3.6's release head alone reads 0.83 to 0.84 across launches: 0.8284, 0.8311, 0.8394, 0.8438 and 0.8371 on vLLM. The Nemotron-H rows are one launch each. No fit here shows a served gain.
- SGLang one draft token:
--speculative-algorithm EAGLE --speculative-draft-model-path <checkpoint> --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2. Its multi-step defaults read 0.62 on the same fit and are not comparable. - A checkpoint with no head: vLLM exits non-zero at load,
Following weights were not initialized from checkpoint. SGLang starts on one and every draft misses (0.0000), so read its counters before trusting a server. - Greedy output: a draft changes a forward's batch shape and so its bf16 rounding, so greedy output parts from a plain server's on about half of the prompts, the release head's too. Each first split is a near-tie: the widest measured is 0.5 logprob, and the row holds it to 1.0. vLLM takes neither of these models at fp32, so no bf16-free reading exists.
Acceptance
Every evaluation of a run with a head scores each held-out draft against the trunk's own distribution over the same token, on every rank, and leaves eval_loss the next-token loss.
head_only gates on them unless mtp_min_acceptance_gain is null. The run scores before its first step and at its last, and exits non-zero on every rank, after its teardown, when either acceptance ended below its first logged reading plus the gain; a resume keeps that first reading. The last step's save already holds the head judged, and the error names that checkpoint, which is not one to serve.
The loss cannot make this call: a released head at learning_rate: 1e-4 with no warmup lost acceptance while its loss fell, and the fit warns off learning_rate: 1e-5 over 20 warmup steps. A fit on rows other than the ones you serve can pass its gate and leave served acceptance where it was, as the Qwen3.6 recipe does below.
Recipes
Both Qwen3.6 recipes render qwen3.6-reasoning-effort.jinja, an empty think block before every reply. Without the block, the same reply tokens read a next-token cross-entropy of 3.03 and top-1 of 0.27, against 0.85 and 0.77 with it, on Halo, stock transformers and vLLM alike. So qwen3-multiturn.jinja gives a run an eval_loss near 3.3 that a frozen trunk keeps; a no-head SFT run reads 0.83 after its first two real optimizer steps at 1e-5, its grad_norm falling from 244 to 4.5.
The Qwen3.5-35B-A3B, Qwen3.5-122B-A10B, Qwen3-Next and two Qwen3.8-Flash-Next recipes also render qwen3-multiturn.jinja, and the loss it starts at differs by model: Qwen3.5-35B-A3B's recipe starts at 1.12, on rows packed to 33,000 tokens. Train and serve under one render.
Neither render shows a served gain. Each recipe's rows are ultrachat replies, not a server's own samples after an open think block, so the gate's reading here does not predict the served one; the template only keeps that reading meaningful. The joint recipe's cost is under Step cost, and what it does to a trunk or a served head over a fine-tune is not measured. The GLM-4.7 Flash recipe's numbers are from first-turn rows at its learning rate: acceptance 0.7685 to 0.7649, which the gate stops, with its served reading under Serving.
GPU rows
tests/gpu/trainers/sft/:
Refusals
Before the load (reject_unsupported_mtp_options, halo/trainers/sft_rules.py): context, tensor and pipeline parallelism, HSDP, an accelerate launch, adapters and quantized loads, lowp_precision, torch_compile, tree_packing, image rows, label smoothing, loss_type other than nll or chunked_nll, save_sharded_ep where the export writes trunk copies, and mtp_head_source under init_from_scratch. head_only also refuses gradient checkpointing and gradient_checkpointing_plan, freeze patterns, init_from_scratch, a balancing mode that updates off the trunk, fp32_non_ep_params, fp32_experts, and a gate with no eval split. A trainer whose forward takes no labels raises at construction, as does a head the freeze patterns froze whole. added_special_tokens past the checkpoint's rows raises at tokenizer setup: the resize replaces the output embedding the head's term holds. Grow the vocabulary first with patch_vocab.py, which carries the head.
A run trained under context or tensor parallelism gets a head through a head_only fit on its export, at rows short enough to run without either.
Source on GitHubUpdated 18 November 2026