GPT-OSS

Two sizes: 20B and 120B. MoE with attention sinks (per-head learnable scalars) and an interleaved expert layout de-interleaved at load on the grouped-GEMM and ETP paths, then re-interleaved on save.

Every axis runs on both sizes (Compatibility matrix). GptOssConfig ships a native HF base_model_ep_plan and no base_model_tp_plan, so its TP runs through the Halo selective-TP path. Under pipeline parallelism balance with bias_update or none (aux_loss is severed by the split); two-node recipe: examples/sft/gptoss/gptoss-20b-multinode-pp2-ep8.yaml (Pipeline Parallelism).

EP layer

EPGptOssMoELayer (halo/distributed/expert_parallel/families/gpt_oss.py) replaces GptOssMLP.

  • Routing: custom router returning (logits, weights, experts); softmax top-k with FP32 weights for DeepEP.

  • Storage: interleaved gate_up_proj [E, H, 2M] ([g0, u0, g1, u1, …]), down_proj [E, M, H], plus the optional expert biases gate_up_proj_bias [E, 2M] (interleaved the same way) and down_proj_bias [E, H].

  • Compute: Grouped GEMM (default on SM90+) reads de-interleaved gate_proj_gmm / up_proj_gmm stored as separate contiguous tensors at init, avoiding NaN gradients on the stride-2 slice; under ETP it reads the sharded pair ETP stores as gate_proj / up_proj. The loop fallback iterates per expert with index_add_.

  • Activation: clamped SwiGLU, (clamp(up) + 1) * clamp(gate) * sigmoid(alpha * clamp(gate)). Every path — grouped GEMM, ETP and the per-expert loop — runs the same fused Triton kernel (GluCombine.gptoss, latched once per layer and logged as glu_combine=gptoss, halo/kernels/fused_glu.py, forward + backward, fp32 accumulation).

    The kernel replaces torch.compile, whose recompiles would storm on the varying per-step routed-token count and whose float arguments go stale once shapes are dynamic. The loop path splits the halves out of its interleaved projection output first.

    All three paths read alpha/limit off GptOssExperts and reproduce its one fixed formula, so none needs the activation probe that arms the plain-SwiGLU kernel (GLM-4). GptOss owns these compute paths and never reaches the base _glu_combine seam.

Packing needs a flash backend

packing: true is refused on eager/SDPA/flex for this family — select_data_collator raises. GPT-OSS builds its attention mask from kwargs that omit position_ids, so a packed row runs as one dense causal sequence and the documents inside it attend across each other silently. The flash interfaces do receive position_ids and isolate correctly, so pin flash_attention_2 (or a later FA) whenever packing is on, and use padded batches otherwise (Document isolation).

Tree packing (tree_packing: true) runs on this family: rows that share a prefix store it once.

Attention sinks

GptOss attention adds a per-head learnable scalar to the attention logits before softmax, with two parallelism consequences.

TP sharding. shard_sinks_param() (halo/distributed/tensor_parallel/parallelize_attention.py) shards sinks per rank as plain tensors, not DTensors — the forward concatenates sinks with locally-sharded logits, and a DTensor would dimension-mismatch on the concat. The slice is recorded through register_hand_sliced so gather_hand_sliced_params() all-gathers it at save time.

Sink handling (halo/models/attention/gpt_oss_sinks.py). One SinksPolicy per run, resolved from reset_sinks / train_sinks at load, stamped on the model for the trainer gates, and recorded in a LoRA run's training_provenance.json so the merge tools re-apply it (a merge rebuilds the base from the hub, whose sinks are always live).

  • Neutralized (reset_sinks: true, the SFT default): None under FA2 (its kernel takes no sink argument), dtype.min and frozen everywhere else (~0 softmax mass, zero gradient). Every checkpoint writer re-emits the FA2-removed sinks as dtype.min.

  • Live, frozen (reset_sinks: false): the pretrained sinks stay in every softmax, requires_grad=False. This is the RL setting — the trainer scores with the sinks the rollout engine serves — and the only live policy an adapter run or weight-sync trainer accepts.

    Live sinks admit only sink-carrying implementations (FA4, flex_attention, eager; FA3 with an s_aux build): a sink-dropping implementation shifts every logprob ~−3 nats against the served policy (gpt-oss-20b). The resolver refuses one before the load, and the model every loader builds is checked again (verify_live_sinks_carried), the builds no resolver picks a backend for included: a sentence-transformers backbone on a pinned label, and the reward-model scorer, whose checkpoint's own sinks decide. They are refused under Context Parallelism, whose kernels never see the column. On Hopper no fused kernel carries them: flex_attention under EP runs uncompiled (Known issues) and eager everywhere, both holding the full score matrix, so memory bounds the row length.

  • Trainable (reset_sinks: false + train_sinks: true): live with gradients on; full fine-tuning under FA4 or eager only. Another family's load refuses train_sinks.

    The loader installs a rescale on both flash_attn.cute entry points (varlen and dense), so a sink trains without the kernel's own sink backward, which is not validated here. A grad-requiring sink runs the sink-less kernel with return_lse=True and applies out * sigmoid(lse - sink) outside it — algebraically the fused sink, with d_sink from the gate and complete dq/dk/dv through dout and the kernel's dlse path.

    tests/gpu/kernels/test_fa4_trainable_sink_rescale.py pins all four against an eager reference on both entry points. Frozen sinks and no-grad calls pass through untouched.

    Refused: reset_sinks: true (a dtype-min sink has zero gradient), flex_attention (NaN sink gradients in its compiled backward), FA3 (no sink gradient), adapter runs (the adapter artifact has no sink slot), and weight-sync RL (no validated end-to-end sync of a moving sink; frozen live sinks stay equal on both sides).

CP, TP, ETP

  • CP — GptOssAttention → GptOssUlyssesAttention (halo/distributed/context_parallel/families.py), RoPE before all-to-all and native GQA. Needs reset_sinks: true: the CP kernels drop the sink column, so the wrapper raises at the first forward on live sinks rather than normalizing every softmax wrongly.

  • TP — a DTensor plan on attention only; embed_tokens/lm_head stay replicated (sharding them faults with a cuBLAS illegal-memory error under EP+TP), MoE experts belong to EP. See the selective-TP plan.

  • ETP — the interleaved-GLU layout needs re-interleaving back to [g0, u0, g1, u1, …] before the checkpoint write. Each rank stores its slice of the de-interleaved pair as gate_proj / up_proj (biases likewise, down_proj_bias on expert-TP rank 0), which grouped GEMM and the per-expert loop both read, so the storage and its checkpoints do not depend on use_grouped_gemm.

    See ETP weight sharding.

Precision and kernels

  • fp32_non_ep_params: true is safe — the router lives inside the EP layer (unlike Gemma 4).
  • Blackwell auto-selects FA4 under every sink policy. GPT-OSS is unaffected by the Qwen3.5/3.6 FA4→SDPA fallback.
  • SFT alternatives. With sinks reset, flash_attention_2 (sinks = None) and sdpa (the dtype.min column contributes exp(dtype.min)=0) also run end to end, both JIT-free. RL cannot use them (why). Which kernel carries a sink, and how the resolver reads it: Flash Attention.
  • Sink-free trainer, live engine. reject_unsyncable_model refuses on-policy RL when the FA2 reset removed the sinks Parameter (reset_sinks: true + FA2): the sync sends the trainer's named parameters, so the engine would keep serving the pretrained sinks against a sink-free trainer — permanently off-policy with no error at sync time.
  • Hopper FA2 runs the image's build, whose split-K kernels are stubbed and never picked by a forward Halo runs (Docker → Building).
  • Variable-length RL (async GRPO) runs FA4 through a per-row dense forward (Flash Attention).

Configs

examples/sft/gptoss/gptoss-20b-multinode-ep.yaml is the infra-agnostic EP example for GPT-OSS 20B and doubles as the multi-node reference; gptoss-120b-multinode-ep.yaml is its 120B counterpart.

The DPO, SMPO, reward, classification and offline GRPO configs pin flash_attention_4, so they run on Blackwell only; it is also what those scripts pick there for an unset label. Elsewhere those scripts default an unset label to SDPA (Flash Attention), so on Hopper pass --attn_implementation=flash_attention_3. The self-distillation and embedding configs leave it unset, which those scripts resolve to FA4 on Blackwell and FA3 on Hopper.

The DPO, SMPO, reward, classification, offline GRPO, self-distillation and 20B multi-node SFT configs train without gradient checkpointing, the largest single throughput lever here: at EP8 the activations fit to 32k tokens per rank (159 GiB, Throughput Benchmarks). The 120B multi-node config keeps it, since it targets 80 GB parts.

A multi-stage fine-tune must agree on reset_sinks across stages — a continuation run keeps the sink policy of the export it starts from.

The SFT recipes train from the releases as OpenAI ships them, openai/gpt-oss-20b and openai/gpt-oss-120b. Their experts are stored in MXFP4 (gate_up_proj_blocks / _scales, down_proj_blocks / _scales), and the lazy EP and PP loaders decode them to bf16 while reading, each rank only its own experts, one expert at a time, through transformers' convert_moe_packed_tensors. The loaded config drops quantization_config, so every full-model save is plain bf16 and, once whole, a checkpoint stock from_pretrained reads back: a gathered save as written, a sharded one once merge_ep_shards.py has merged it, and a pipeline-parallel save on per-node storage once merge_pp_stages.py has merged every node's directory. An adapter save holds only the adapter weights. The BF16 mirrors unsloth/gpt-oss-{20b,120b}-BF16 still load, at about three times the download.

A whole-model load (tensor or context parallelism without an EP group, ep1, or EP with ep_lazy_loading: false) hands the release to from_pretrained, and transformers decides there. The Halo images ship no kernels package, so its MXFP4 quantizer dequantizes to bf16 itself and every rank stages the whole decoded model: gpt-oss-20b at EP8 on 8× H200 takes about 855 s that way, against 5 s lazily. The recipes that load a frozen reference or teacher beside the policy (preference, distillation, KL-regularized) keep the BF16 mirror.

MXFP4 still serves on vLLM, but not under weight sync: its expert loader has no branch for a bf16 expert tensor, so a synced update drops every expert weight while the biases land, silently (Rollout Servers). On-policy RL serves a BF16 checkpoint, whichever checkpoint the trainer loaded, since the sync pushes bf16 expert tensors.

Async GRPO with Environments: examples/grpo/environmental/gptoss/vllm/ plus the sglang/ ep1 siblings — both pinned engines read the interleaved expert pair the gather emits, so either rollout_backend takes the weight sync (Rollout Servers).

gptoss-20b-multinode-ep.yaml sets use_grouped_gemm: false: at its EP=16 the 20B keeps 2 experts per rank, few enough that the loop is competitive, and gpt-oss's square expert FFN (intermediate == hidden == 2880, not a multiple of the kernel's 128-tile K) pays a CUTLASS tail epilogue the loop avoids. Grouped GEMM is the default and wins every measured cell through ep8 at batch 4. See Grouped GEMM.

Chat template

Two templates ship for GPT-OSS under jinja-templates/gpt-oss/, selected with chat_template:. gpt-oss-harmony.jinja is the upstream OpenAI-harmony template from unsloth/gpt-oss-20b-BF16, passed to vLLM too, so the training render is byte-identical to the server-side one. It deviates twice. Tool and parameter description is guarded with is defined, so a tool without descriptions does not crash the render (the checkpoint's built-in template renders param_spec.description unconditionally and raises without it). A tool call's string arguments, the OpenAI schema rollouts store, render verbatim, where the built-in tojson double-encodes them; restoring the upstream file trains quoted, escaped arguments no harmony parser reads.

For SFT and self-distillation the shipped examples pin gpt-oss-multiturn.jinja with force_chat_template: true and assistant_message_template: <|start|>assistant<|channel|>final<|message|>: it renders every assistant turn as an empty analysis channel followed by the final channel the served model emits, and ends every assistant turn with <|return|>, so completion-only masking trains each assistant turn of a multi-turn row. Under harmony a plain assistant message renders channel-less as <|start|>assistant<|message|>… and only the final turn ends in <|return|> (earlier turns end in <|end|>, which is not an eos id), so that marker matches no turn at all and every row masks; the channel-less spelling of it would train only the last assistant turn.

For RL the training and serving templates must match, since log-probs are computed on the served prompt. GPT-OSS ships a built-in template, so overriding it needs both chat_template: jinja-templates/gpt-oss/gpt-oss-harmony.jinja and force_chat_template: true. Pass the same file to vLLM via VLLM_CHAT_TEMPLATE.

Serving for GRPO (vLLM)

Use the toolkit halo:vllm-0.32.0 image, not stock upstream vLLM: on B300 the stock vllm/vllm-openai image produces garbage GPT-OSS output with harmony on or off. The toolkit image (Dockerfile.vllm plus docker/vllm/patches/patch_vllm_source.py) forces vLLM's harmony pipeline off and generates coherently.

With harmony disabled, five settings are load-bearing.

  • Text tool parser. GPT-OSS emits tool calls as plain text (commentary to=functions.<name> json{...}), unreadable by the stock openai (needs harmony token IDs) and seed_oss parsers. Serve with --enable-auto-tool-choice --tool-parser-plugin /opt/gpt_oss_text_tool_parser.py --tool-call-parser gpt_oss_text (docker/vllm/plugins/gpt_oss_text_tool_parser.py).

    The plugin brace-balances the JSON, returns the final-channel text as content so the CoT is dropped, rewrites the name-glued token (submit_solutionjson → submit_solution) to the longest declared tool name it starts with, and coerces malformed arguments into valid JSON so a bad call never 400s the next turn.

    Only the first call is extracted: without <|call|> as an eos the model keeps writing to=functions.* text and hallucinates its own tool result. A header with no brace-balanced JSON after it is dropped rather than rescued, so the turn scores as a no-tool turn and the policy is pushed toward valid JSON. Non-tool RLVR needs no tool parser.

  • Reasoning budget. --reasoning-parser-plugin /opt/gpt_oss_reasoning_parser.py --reasoning-parser openai_gptoss re-registers the harmony-only stock parser for the harmony-disabled path. It separates the CoT from the answer, and is what lets a request carry thinking_token_budget at all (without it every such request 400s). When an analysis channel overruns the budget it forces the move into the final channel, <|end|><|start|>assistant<|channel|>final<|message|>, which the vLLM recipes name as rollout_reasoning_end_token so the loss drops those forced tokens (Objective).

    The budget counts only inside an analysis channel: the count starts at an analysis opening (analysis<|message|>, the parser's reasoning_start_str) and stops at any channel header (<|channel|>, its reasoning_end_str), so a tool call, a commentary preamble and the final answer are never counted or cut. It is per analysis message, and counts up to the three closing tokens (<|end|><|start|>assistant) before the next header, so an analysis ending on its own within three tokens of the budget is turned into final.

    vLLM forces the parser's reasoning_end_str unless told otherwise, and stops forcing at the first natural end marker. Two image patches make the gpt-oss cut whole: vllm_reasoning_forced_end_patch forces the parser's reasoning_forced_end_str (the six-token move) instead, and vllm_thinking_budget_completion_patch finishes that sequence past the <|channel|> it contains rather than letting the model pick the channel, where a reopened analysis would count afresh. The forced cut carries the role tokens, so it decodes byte-identical to the ending the model writes itself and the split finds the answer behind it.

    docker/vllm/plugins/verify_gptoss_plugins.py drives Model Runner V2's own budget kernels (on the Triton interpreter) over simulated harmony-disabled streams at build time: a cut lands right after the budget, a cut model cannot reopen analysis, tool calls, preambles and straight-to-final turns are never cut, and a recycled slot forgets a previous cut. A render or marker drift fails the image build rather than a run.

    The parser keeps the commentary tool call and the final answer in content: vLLM extracts tool calls from content only, so a call left in reasoning would be dropped and the turn mis-scored as a giveup.

  • --return-tokens-as-token-ids — required whenever train_on_sampled_tokens is on (the default); the compose file passes it (Rollout Servers).

  • rollout_stop_tokens: ["<|call|>"] — with harmony disabled <|call|> is not an eos, so the model keeps generating past its tool call and hallucinates the result for most of the turn. A tool-bearing environment without it is refused at trainer construction. A server-side --override-generation-config eos does not fix it (Async GRPO with Environments).

  • Full padded vocab. GPT-OSS ships 201088 embedding rows against len(tokenizer) 200019; rows 200019–201087 are unassigned padding. Every harmony special sits below that line — <|return|>/EOS at 200002, <|call|> at 200012, highest added id 200018 — so a resize to len(tokenizer) keeps them all.

    The trap is tokenizer.vocab_size, which is 199998: shrink to that and the checkpoint loses EOS and emits garbage. scripts/before_training/patch_vocab.py only grows the embedding, so a patched checkpoint keeps every special and serves like the base — unless it was patched with --reset_sinks, which leaves it sinks-off, and the pretrained model served sinks-off degenerates (below).

Sinks stay on at serving (the default): served sinks-off, the pretrained model degenerates to repetitive garbage with zero tool calls. The trainer freezes the same sinks, so both sides score with them; what remains between the two bf16 forwards is routing and rounding noise (Engine reference).

GPT-OSS serves from SGLang for async GRPO (rollout_backend: sglang): SGLang's own built-in detectors replace both vLLM plugins — the compose default --tool-call-parser auto resolves gpt-oss off the chat template, and SGLANG_REASONING_PARSER=gpt-oss separates the analysis channel (both registered in 0.5.22); thinking budgets are rejected at config time for this backend, and the server container needs NCCL_CUMEM_ENABLE=1 (the compose default). Flags and constraints: Rollout Servers.

Router balancing

GPT-OSS defaults to switch-style aux-loss balancing: moe_balancing: auto resolves to aux_loss, and that mode turns output_router_logits on itself.

The coefficient a run reads is the checkpoint's, not the class default. GptOssConfig declares router_aux_loss_coef: 0.001, but every released config.json (openai/gpt-oss-* and the unsloth/*-BF16 mirrors) ships 0.9, large enough to dominate both the SFT loss and the router gradient.

The shipped SFT examples that stay on aux_loss override it back down to 0.001 in model_init_kwargs; do the same on any aux-loss run, DPO and KTO included (their TRL loss weighs the term by the model config's coefficient: Callbacks), rather than inheriting 0.9. A bias_update run needs no override — that mode zeroes the coefficient itself (below).

The DeepSeek-V3 aux-loss-free bias update is opt-in on the EP path (moe_balancing: bias_update) and adopts the hub router's own router.bias as the balancing state. The Parameter is re-registered as a persistent buffer under the same key: frozen out of gradient training, sign-updated by BalancingBiasCallback each optimizer step, and exported with every checkpoint.

vLLM and SGLang load router.bias and route with it (top-k on bias-inclusive logits, combine = softmax over the selected values). _route_with_bias computes that arithmetic, so both read the trained bias the same way; bf16 near-ties can still pick differently (Engine reference).

router_aux_loss_coef is forced to 0 for the run to avoid double-balancing (restored in the exported config). The bias lives in logit space, so the default γ is small for it; size router_balancing_rate per BalancingBiasCallback.

Not for on-policy RL. Online and env GRPO must use moe_balancing: none — adoption re-registers router.bias as a buffer, and the weight sync ships a buffer only where the family declares it in engine_loaded_buffers, so a synced engine routes on the pretrained bias (configure_moe_balancing downgrades it automatically). Bias-update needs the EP layers (ep_size > 1, or the default use_grouped_gemm); without them it raises at setup.

Source on GitHubUpdated 18 November 2026