GPT-OSS
Two sizes: 20B and 120B. MoE with attention sinks (per-head learnable scalars) and an interleaved expert layout de-interleaved at load on the grouped-GEMM and ETP paths, then re-interleaved on save.
Every axis runs on both sizes (Compatibility matrix). GptOssConfig ships a native HF base_model_ep_plan and no base_model_tp_plan, so its TP runs through the Halo selective-TP path. Under pipeline parallelism balance with bias_update or none (aux_loss is severed by the split); two-node recipe: examples/sft/gptoss/gptoss-20b-multinode-pp2-ep8.yaml (Pipeline Parallelism).
EP layer
EPGptOssMoELayer (halo/distributed/expert_parallel/families/gpt_oss.py) replaces GptOssMLP.
Routing: custom router returning
(logits, weights, experts); softmax top-k with FP32 weights for DeepEP.Storage: interleaved
gate_up_proj [E, H, 2M]([g0, u0, g1, u1, …]),down_proj [E, M, H], plus the optional expert biasesgate_up_proj_bias [E, 2M](interleaved the same way) anddown_proj_bias [E, H].Compute: Grouped GEMM (default on SM90+) reads de-interleaved
gate_proj_gmm/up_proj_gmmstored as separate contiguous tensors at init, avoiding NaN gradients on the stride-2 slice; under ETP it reads the sharded pair ETP stores asgate_proj/up_proj. The loop fallback iterates per expert withindex_add_.Activation: clamped SwiGLU,
(clamp(up) + 1) * clamp(gate) * sigmoid(alpha * clamp(gate)). Every path — grouped GEMM, ETP and the per-expert loop — runs the same fused Triton kernel (GluCombine.gptoss, latched once per layer and logged asglu_combine=gptoss,halo/kernels/fused_glu.py, forward + backward, fp32 accumulation).The kernel replaces
torch.compile, whose recompiles would storm on the varying per-step routed-token count and whose float arguments go stale once shapes are dynamic. The loop path splits the halves out of its interleaved projection output first.All three paths read
alpha/limitoffGptOssExpertsand reproduce its one fixed formula, so none needs the activation probe that arms the plain-SwiGLU kernel (GLM-4). GptOss owns these compute paths and never reaches the base_glu_combineseam.
Packing needs a flash backend
packing: true is refused on eager/SDPA/flex for this family — select_data_collator raises.
GPT-OSS builds its attention mask from kwargs that omit position_ids, so a packed row runs as one
dense causal sequence and the documents inside it attend across each other silently. The flash
interfaces do receive position_ids and isolate correctly, so pin flash_attention_2 (or a later
FA) whenever packing is on, and use padded batches otherwise
(Document isolation).
Tree packing (tree_packing: true) runs on this family: rows that
share a prefix store it once.
Attention sinks
GptOss attention adds a per-head learnable scalar to the attention logits before softmax, with two parallelism consequences.
TP sharding. shard_sinks_param() (halo/distributed/tensor_parallel/parallelize_attention.py) shards sinks per rank as plain tensors, not DTensors — the forward concatenates sinks with locally-sharded logits, and a DTensor would dimension-mismatch on the concat. The slice is recorded through register_hand_sliced so gather_hand_sliced_params() all-gathers it at save time.
Sink handling (halo/models/attention/gpt_oss_sinks.py). One SinksPolicy per run, resolved from reset_sinks / train_sinks at load, stamped on the model for the trainer gates, and recorded in a LoRA run's training_provenance.json so the merge tools re-apply it (a merge rebuilds the base from the hub, whose sinks are always live).
Neutralized (
reset_sinks: true, the SFT default):Noneunder FA2 (its kernel takes no sink argument),dtype.minand frozen everywhere else (~0 softmax mass, zero gradient). Every checkpoint writer re-emits the FA2-removed sinks asdtype.min.Live, frozen (
reset_sinks: false): the pretrained sinks stay in every softmax,requires_grad=False. This is the RL setting — the trainer scores with the sinks the rollout engine serves — and the only live policy an adapter run or weight-sync trainer accepts.Live sinks admit only sink-carrying implementations (FA4,
flex_attention,eager; FA3 with ans_auxbuild): a sink-dropping implementation shifts every logprob ~−3 nats against the served policy (gpt-oss-20b). The resolver refuses one before the load, and the model every loader builds is checked again (verify_live_sinks_carried), the builds no resolver picks a backend for included: a sentence-transformers backbone on a pinned label, and the reward-model scorer, whose checkpoint's own sinks decide. They are refused under Context Parallelism, whose kernels never see the column. On Hopper no fused kernel carries them:flex_attentionunder EP runs uncompiled (Known issues) andeagereverywhere, both holding the full score matrix, so memory bounds the row length.Trainable (
reset_sinks: false+train_sinks: true): live with gradients on; full fine-tuning under FA4 oreageronly. Another family's load refusestrain_sinks.The loader installs a rescale on both
flash_attn.cuteentry points (varlen and dense), so a sink trains without the kernel's own sink backward, which is not validated here. A grad-requiring sink runs the sink-less kernel withreturn_lse=Trueand appliesout * sigmoid(lse - sink)outside it — algebraically the fused sink, withd_sinkfrom the gate and completedq/dk/dvthroughdoutand the kernel'sdlsepath.tests/gpu/kernels/test_fa4_trainable_sink_rescale.pypins all four against an eager reference on both entry points. Frozen sinks and no-grad calls pass through untouched.Refused:
reset_sinks: true(a dtype-min sink has zero gradient),flex_attention(NaN sink gradients in its compiled backward), FA3 (no sink gradient), adapter runs (the adapter artifact has no sink slot), and weight-sync RL (no validated end-to-end sync of a moving sink; frozen live sinks stay equal on both sides).
CP, TP, ETP
CP —
GptOssAttention→GptOssUlyssesAttention(halo/distributed/context_parallel/families.py), RoPE before all-to-all and native GQA. Needsreset_sinks: true: the CP kernels drop the sink column, so the wrapper raises at the first forward on live sinks rather than normalizing every softmax wrongly.TP — a DTensor plan on attention only;
embed_tokens/lm_headstay replicated (sharding them faults with a cuBLAS illegal-memory error under EP+TP), MoE experts belong to EP. See the selective-TP plan.ETP — the interleaved-GLU layout needs re-interleaving back to
[g0, u0, g1, u1, …]before the checkpoint write. Each rank stores its slice of the de-interleaved pair asgate_proj/up_proj(biases likewise,down_proj_biason expert-TP rank 0), which grouped GEMM and the per-expert loop both read, so the storage and its checkpoints do not depend onuse_grouped_gemm.See ETP weight sharding.
Precision and kernels
fp32_non_ep_params: trueis safe — the router lives inside the EP layer (unlike Gemma 4).- Blackwell auto-selects FA4 under every sink policy. GPT-OSS is unaffected by the Qwen3.5/3.6 FA4→SDPA fallback.
- SFT alternatives. With sinks reset,
flash_attention_2(sinks = None) andsdpa(thedtype.mincolumn contributesexp(dtype.min)=0) also run end to end, both JIT-free. RL cannot use them (why). Which kernel carries a sink, and how the resolver reads it: Flash Attention. - Sink-free trainer, live engine.
reject_unsyncable_modelrefuses on-policy RL when the FA2 reset removed the sinks Parameter (reset_sinks: true+ FA2): the sync sends the trainer's named parameters, so the engine would keep serving the pretrained sinks against a sink-free trainer — permanently off-policy with no error at sync time. - Hopper FA2 runs the image's build, whose split-K kernels are stubbed and never picked by a forward Halo runs (Docker → Building).
- Variable-length RL (async GRPO) runs FA4 through a per-row dense forward (Flash Attention).
Configs
examples/sft/gptoss/gptoss-20b-multinode-ep.yaml is the infra-agnostic EP example for GPT-OSS 20B and doubles as the multi-node reference; gptoss-120b-multinode-ep.yaml is its 120B counterpart.
The DPO, SMPO, reward, classification and offline GRPO configs pin flash_attention_4, so they run on Blackwell only; it is also what those scripts pick there for an unset label. Elsewhere those scripts default an unset label to SDPA (Flash Attention), so on Hopper pass --attn_implementation=flash_attention_3. The self-distillation and embedding configs leave it unset, which those scripts resolve to FA4 on Blackwell and FA3 on Hopper.
The DPO, SMPO, reward, classification, offline GRPO, self-distillation and 20B multi-node SFT configs train without gradient checkpointing, the largest single throughput lever here: at EP8 the activations fit to 32k tokens per rank (159 GiB, Throughput Benchmarks). The 120B multi-node config keeps it, since it targets 80 GB parts.
A multi-stage fine-tune must agree on reset_sinks across stages — a continuation run keeps the sink policy of the export it starts from.
The SFT recipes train from the releases as OpenAI ships them, openai/gpt-oss-20b and openai/gpt-oss-120b. Their experts are stored in MXFP4 (gate_up_proj_blocks / _scales, down_proj_blocks / _scales), and the lazy EP and PP loaders decode them to bf16 while reading, each rank only its own experts, one expert at a time, through transformers' convert_moe_packed_tensors. The loaded config drops quantization_config, so every full-model save is plain bf16 and, once whole, a checkpoint stock from_pretrained reads back: a gathered save as written, a sharded one once merge_ep_shards.py has merged it, and a pipeline-parallel save on per-node storage once merge_pp_stages.py has merged every node's directory. An adapter save holds only the adapter weights. The BF16 mirrors unsloth/gpt-oss-{20b,120b}-BF16 still load, at about three times the download.
A whole-model load (tensor or context parallelism without an EP group, ep1, or EP with ep_lazy_loading: false) hands the release to from_pretrained, and transformers decides there. The Halo images ship no kernels package, so its MXFP4 quantizer dequantizes to bf16 itself and every rank stages the whole decoded model: gpt-oss-20b at EP8 on 8× H200 takes about 855 s that way, against 5 s lazily. The recipes that load a frozen reference or teacher beside the policy (preference, distillation, KL-regularized) keep the BF16 mirror.
MXFP4 still serves on vLLM, but not under weight sync: its expert loader has no branch for a bf16 expert tensor, so a synced update drops every expert weight while the biases land, silently (Rollout Servers). On-policy RL serves a BF16 checkpoint, whichever checkpoint the trainer loaded, since the sync pushes bf16 expert tensors.
Async GRPO with Environments: examples/grpo/environmental/gptoss/vllm/ plus the sglang/ ep1 siblings — both pinned engines read the interleaved expert pair the gather emits, so either rollout_backend takes the weight sync (Rollout Servers).
gptoss-20b-multinode-ep.yaml sets use_grouped_gemm: false: at its EP=16 the 20B keeps 2 experts per rank, few enough that the loop is competitive, and gpt-oss's square expert FFN (intermediate == hidden == 2880, not a multiple of the kernel's 128-tile K) pays a CUTLASS tail epilogue the loop avoids. Grouped GEMM is the default and wins every measured cell through ep8 at batch 4. See Grouped GEMM.
Chat template
Two templates ship for GPT-OSS under jinja-templates/gpt-oss/, selected with chat_template:. gpt-oss-harmony.jinja is the upstream OpenAI-harmony template from unsloth/gpt-oss-20b-BF16, passed to vLLM too, so the training render is byte-identical to the server-side one. It deviates twice. Tool and parameter description is guarded with is defined, so a tool without descriptions does not crash the render (the checkpoint's built-in template renders param_spec.description unconditionally and raises without it). A tool call's string arguments, the OpenAI schema rollouts store, render verbatim, where the built-in tojson double-encodes them; restoring the upstream file trains quoted, escaped arguments no harmony parser reads.
For SFT and self-distillation the shipped examples pin gpt-oss-multiturn.jinja with force_chat_template: true and assistant_message_template: <|start|>assistant<|channel|>final<|message|>: it renders every assistant turn as an empty analysis channel followed by the final channel the served model emits, and ends every assistant turn with <|return|>, so completion-only masking trains each assistant turn of a multi-turn row. Under harmony a plain assistant message renders channel-less as <|start|>assistant<|message|>… and only the final turn ends in <|return|> (earlier turns end in <|end|>, which is not an eos id), so that marker matches no turn at all and every row masks; the channel-less spelling of it would train only the last assistant turn.
For RL the training and serving templates must match, since log-probs are computed on the served prompt. GPT-OSS ships a built-in template, so overriding it needs both chat_template: jinja-templates/gpt-oss/gpt-oss-harmony.jinja and force_chat_template: true. Pass the same file to vLLM via VLLM_CHAT_TEMPLATE.
Serving for GRPO (vLLM)
Use the toolkit halo:vllm-0.32.0 image, not stock upstream vLLM: on B300 the stock vllm/vllm-openai image produces garbage GPT-OSS output with harmony on or off. The toolkit image (Dockerfile.vllm plus docker/vllm/patches/patch_vllm_source.py) forces vLLM's harmony pipeline off and generates coherently.
With harmony disabled, five settings are load-bearing.
Text tool parser. GPT-OSS emits tool calls as plain text (
commentary to=functions.<name> json{...}), unreadable by the stockopenai(needs harmony token IDs) andseed_ossparsers. Serve with--enable-auto-tool-choice --tool-parser-plugin /opt/gpt_oss_text_tool_parser.py --tool-call-parser gpt_oss_text(docker/vllm/plugins/gpt_oss_text_tool_parser.py).The plugin brace-balances the JSON, returns the
final-channel text ascontentso the CoT is dropped, rewrites the name-glued token (submit_solutionjson→submit_solution) to the longest declared tool name it starts with, and coerces malformed arguments into valid JSON so a bad call never 400s the next turn.Only the first call is extracted: without
<|call|>as an eos the model keeps writingto=functions.*text and hallucinates its own tool result. A header with no brace-balanced JSON after it is dropped rather than rescued, so the turn scores as a no-tool turn and the policy is pushed toward valid JSON. Non-tool RLVR needs no tool parser.Reasoning budget.
--reasoning-parser-plugin /opt/gpt_oss_reasoning_parser.py --reasoning-parser openai_gptossre-registers the harmony-only stock parser for the harmony-disabled path. It separates the CoT from the answer, and is what lets a request carrythinking_token_budgetat all (without it every such request 400s). When an analysis channel overruns the budget it forces the move into the final channel,<|end|><|start|>assistant<|channel|>final<|message|>, which the vLLM recipes name asrollout_reasoning_end_tokenso the loss drops those forced tokens (Objective).The budget counts only inside an analysis channel: the count starts at an analysis opening (
analysis<|message|>, the parser'sreasoning_start_str) and stops at any channel header (<|channel|>, itsreasoning_end_str), so a tool call, a commentary preamble and the final answer are never counted or cut. It is per analysis message, and counts up to the three closing tokens (<|end|><|start|>assistant) before the next header, so an analysis ending on its own within three tokens of the budget is turned into final.vLLM forces the parser's
reasoning_end_strunless told otherwise, and stops forcing at the first natural end marker. Two image patches make the gpt-oss cut whole:vllm_reasoning_forced_end_patchforces the parser'sreasoning_forced_end_str(the six-token move) instead, andvllm_thinking_budget_completion_patchfinishes that sequence past the<|channel|>it contains rather than letting the model pick the channel, where a reopened analysis would count afresh. The forced cut carries the role tokens, so it decodes byte-identical to the ending the model writes itself and the split finds the answer behind it.docker/vllm/plugins/verify_gptoss_plugins.pydrives Model Runner V2's own budget kernels (on the Triton interpreter) over simulated harmony-disabled streams at build time: a cut lands right after the budget, a cut model cannot reopen analysis, tool calls, preambles and straight-to-final turns are never cut, and a recycled slot forgets a previous cut. A render or marker drift fails the image build rather than a run.The parser keeps the commentary tool call and the
finalanswer incontent: vLLM extracts tool calls fromcontentonly, so a call left inreasoningwould be dropped and the turn mis-scored as a giveup.--return-tokens-as-token-ids— required whenevertrain_on_sampled_tokensis on (the default); the compose file passes it (Rollout Servers).rollout_stop_tokens: ["<|call|>"]— with harmony disabled<|call|>is not an eos, so the model keeps generating past its tool call and hallucinates the result for most of the turn. A tool-bearing environment without it is refused at trainer construction. A server-side--override-generation-configeos does not fix it (Async GRPO with Environments).Full padded vocab. GPT-OSS ships 201088 embedding rows against
len(tokenizer)200019; rows 200019–201087 are unassigned padding. Every harmony special sits below that line —<|return|>/EOS at 200002,<|call|>at 200012, highest added id 200018 — so a resize tolen(tokenizer)keeps them all.The trap is
tokenizer.vocab_size, which is 199998: shrink to that and the checkpoint loses EOS and emits garbage.scripts/before_training/patch_vocab.pyonly grows the embedding, so a patched checkpoint keeps every special and serves like the base — unless it was patched with--reset_sinks, which leaves it sinks-off, and the pretrained model served sinks-off degenerates (below).
Sinks stay on at serving (the default): served sinks-off, the pretrained model degenerates to repetitive garbage with zero tool calls. The trainer freezes the same sinks, so both sides score with them; what remains between the two bf16 forwards is routing and rounding noise (Engine reference).
GPT-OSS serves from SGLang for async GRPO (rollout_backend: sglang): SGLang's own built-in detectors replace both vLLM plugins — the compose default --tool-call-parser auto resolves gpt-oss off the chat template, and SGLANG_REASONING_PARSER=gpt-oss separates the analysis channel (both registered in 0.5.22); thinking budgets are rejected at config time for this backend, and the server container needs NCCL_CUMEM_ENABLE=1 (the compose default). Flags and constraints: Rollout Servers.
Router balancing
GPT-OSS defaults to switch-style aux-loss balancing: moe_balancing: auto resolves to aux_loss, and that mode turns output_router_logits on itself.
The coefficient a run reads is the checkpoint's, not the class default. GptOssConfig declares router_aux_loss_coef: 0.001, but every released config.json (openai/gpt-oss-* and the unsloth/*-BF16 mirrors) ships 0.9, large enough to dominate both the SFT loss and the router gradient.
The shipped SFT examples that stay on aux_loss override it back down to 0.001 in model_init_kwargs; do the same on any aux-loss run, DPO and KTO included (their TRL loss weighs the term by the model config's coefficient: Callbacks), rather than inheriting 0.9. A bias_update run needs no override — that mode zeroes the coefficient itself (below).
The DeepSeek-V3 aux-loss-free bias update is opt-in on the EP path (moe_balancing: bias_update) and adopts the hub router's own router.bias as the balancing state. The Parameter is re-registered as a persistent buffer under the same key: frozen out of gradient training, sign-updated by BalancingBiasCallback each optimizer step, and exported with every checkpoint.
vLLM and SGLang load router.bias and route with it (top-k on bias-inclusive logits, combine = softmax over the selected values). _route_with_bias computes that arithmetic, so both read the trained bias the same way; bf16 near-ties can still pick differently (Engine reference).
router_aux_loss_coef is forced to 0 for the run to avoid double-balancing (restored in the exported config). The bias lives in logit space, so the default γ is small for it; size router_balancing_rate per BalancingBiasCallback.
Not for on-policy RL. Online and env GRPO must use moe_balancing: none — adoption re-registers router.bias as a buffer, and the weight sync ships a buffer only where the family declares it in engine_loaded_buffers, so a synced engine routes on the pretrained bias (configure_moe_balancing downgrades it automatically). Bias-update needs the EP layers (ep_size > 1, or the default use_grouped_gemm); without them it raises at setup.
Source on GitHubUpdated 18 November 2026