# GPT-OSS

Two sizes: 20B and 120B. MoE with **attention sinks** (per-head learnable scalars) and an interleaved expert layout de-interleaved at load on the grouped-GEMM and ETP paths, then re-interleaved on save.

Every axis runs on both sizes ([Compatibility matrix](https://whitecircle.com/halo/docs/agent-docs/models.md#compatibility-matrix)). `GptOssConfig` ships a native HF `base_model_ep_plan` and no `base_model_tp_plan`, so its TP runs through the Halo selective-TP path. Under pipeline parallelism balance with `bias_update` or `none` (`aux_loss` is severed by the split); two-node recipe: `examples/sft/gptoss/gptoss-20b-multinode-pp2-ep8.yaml` ([Pipeline Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#model-support)).

## EP layer

`EPGptOssMoELayer` (`halo/distributed/expert_parallel/families/gpt_oss.py`) replaces `GptOssMLP`.

- Routing: custom router returning `(logits, weights, experts)`; softmax top-k with FP32 weights for DeepEP.
- Storage: interleaved `gate_up_proj [E, H, 2M]` (`[g0, u0, g1, u1, …]`), `down_proj [E, M, H]`, plus the optional expert biases `gate_up_proj_bias [E, 2M]` (interleaved the same way) and `down_proj_bias [E, H]`.
- Compute: **Grouped GEMM** (default on SM90+) reads de-interleaved `gate_proj_gmm` / `up_proj_gmm` stored as separate contiguous tensors at init, avoiding NaN gradients on the stride-2 slice; under ETP it reads the sharded pair ETP stores as `gate_proj` / `up_proj`. The **loop fallback** iterates per expert with `index_add_`.
- Activation: clamped SwiGLU, `(clamp(up) + 1) * clamp(gate) * sigmoid(alpha * clamp(gate))`. Every path — grouped GEMM, ETP and the per-expert loop — runs the same fused Triton kernel (`GluCombine.gptoss`, latched once per layer and logged as `glu_combine=gptoss`, `halo/kernels/fused_glu.py`, forward + backward, fp32 accumulation).

    The kernel replaces `torch.compile`, whose recompiles would storm on the varying per-step routed-token count and whose float arguments go stale once shapes are dynamic. The loop path splits the halves out of its interleaved projection output first.

    All three paths read `alpha`/`limit` off `GptOssExperts` and reproduce its one fixed formula, so none needs the activation probe that arms the plain-SwiGLU kernel ([GLM-4](https://whitecircle.com/halo/docs/agent-docs/models/glm4.md#fused-swiglu)). GptOss owns these compute paths and never reaches the base `_glu_combine` seam.

## Packing needs a flash backend

`packing: true` is **refused** on eager/SDPA/flex for this family — `select_data_collator` raises.
GPT-OSS builds its attention mask from kwargs that omit `position_ids`, so a packed row runs as one
dense causal sequence and the documents inside it attend across each other silently. The flash
interfaces do receive `position_ids` and isolate correctly, so pin `flash_attention_2` (or a later
FA) whenever packing is on, and use padded batches otherwise
([Document isolation](https://whitecircle.com/halo/docs/agent-docs/data/collators.md#document-isolation-under-packing)).

[Tree packing](https://whitecircle.com/halo/docs/agent-docs/optimization/tree-packing.md) (`tree_packing: true`) runs on this family: rows that
share a prefix store it once.

## Attention sinks

GptOss attention adds a per-head learnable scalar to the attention logits before softmax, with two parallelism consequences.

**TP sharding.** `shard_sinks_param()` (`halo/distributed/tensor_parallel/parallelize_attention.py`) shards sinks per rank as plain tensors, not DTensors — the forward concatenates sinks with locally-sharded logits, and a DTensor would dimension-mismatch on the concat. The slice is recorded through `register_hand_sliced` so `gather_hand_sliced_params()` all-gathers it at save time.

**Sink handling** (`halo/models/attention/gpt_oss_sinks.py`). One `SinksPolicy` per run, resolved from `reset_sinks` / `train_sinks` at load, stamped on the model for the trainer gates, and recorded in a LoRA run's `training_provenance.json` so the merge tools re-apply it (a merge rebuilds the base from the hub, whose sinks are always live).

- **Neutralized** (`reset_sinks: true`, the SFT default): `None` under FA2 (its kernel takes no sink argument), `dtype.min` and frozen everywhere else (~0 softmax mass, zero gradient). Every checkpoint writer re-emits the FA2-removed sinks as `dtype.min`.
- **Live, frozen** (`reset_sinks: false`): the pretrained sinks stay in every softmax, `requires_grad=False`. This is the RL setting — the trainer scores with the sinks the rollout engine serves — and the only live policy an adapter run or weight-sync trainer accepts.

    Live sinks admit only sink-carrying implementations (FA4, `flex_attention`, `eager`; FA3 with an `s_aux` build): a sink-dropping implementation shifts every logprob ~−3 nats against the served policy (gpt-oss-20b). The resolver refuses one before the load, and the model every loader builds is checked again (`verify_live_sinks_carried`), the builds no resolver picks a backend for included: a sentence-transformers backbone on a pinned label, and the reward-model scorer, whose checkpoint's own sinks decide. They are refused under Context Parallelism, whose kernels never see the column. On Hopper no fused kernel carries them: `flex_attention` under EP runs uncompiled ([Known issues](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#known-issues)) and `eager` everywhere, both holding the full score matrix, so memory bounds the row length.

- **Trainable** (`reset_sinks: false` + `train_sinks: true`): live with gradients on; full fine-tuning under FA4 or `eager` only. Another family's load refuses `train_sinks`.

    The loader installs a rescale on both `flash_attn.cute` entry points (varlen and dense), so a sink trains without the kernel's own sink backward, which is not validated here. A grad-requiring sink runs the sink-less kernel with `return_lse=True` and applies `out * sigmoid(lse - sink)` outside it — algebraically the fused sink, with `d_sink` from the gate and complete `dq/dk/dv` through `dout` and the kernel's `dlse` path.

    `tests/gpu/kernels/test_fa4_trainable_sink_rescale.py` pins all four against an eager reference on both entry points. Frozen sinks and no-grad calls pass through untouched.

    Refused: `reset_sinks: true` (a dtype-min sink has zero gradient), `flex_attention` (NaN sink gradients in its compiled backward), FA3 (no sink gradient), adapter runs (the adapter artifact has no sink slot), and weight-sync RL (no validated end-to-end sync of a moving sink; frozen live sinks stay equal on both sides).

## CP, TP, ETP

- **CP** — `GptOssAttention` → `GptOssUlyssesAttention` (`halo/distributed/context_parallel/families.py`), RoPE before all-to-all and native GQA. Needs `reset_sinks: true`: the CP kernels drop the sink column, so the wrapper raises at the first forward on live sinks rather than normalizing every softmax wrongly.
- **TP** — a DTensor plan on attention only; `embed_tokens`/`lm_head` stay replicated (sharding them faults with a cuBLAS illegal-memory error under EP+TP), MoE experts belong to EP. See [the selective-TP plan](https://whitecircle.com/halo/docs/agent-docs/parallelism/tensor-parallelism.md#the-selective-tp-plan).
- **ETP** — the interleaved-GLU layout needs re-interleaving back to `[g0, u0, g1, u1, …]` before the checkpoint write. Each rank stores its slice of the de-interleaved pair as `gate_proj` / `up_proj` (biases likewise, `down_proj_bias` on expert-TP rank 0), which grouped GEMM and the per-expert loop both read, so the storage and its checkpoints do not depend on `use_grouped_gemm`.

    See [ETP weight sharding](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-tensor-parallelism.md#weight-sharding).

## Precision and kernels

- `fp32_non_ep_params: true` is safe — the router lives inside the EP layer (unlike Gemma 4).
- **Blackwell** auto-selects FA4 under every sink policy. GPT-OSS is unaffected by the Qwen3.5/3.6 FA4→SDPA fallback.
- **SFT alternatives.** With sinks reset, `flash_attention_2` (`sinks = None`) and `sdpa` (the `dtype.min` column contributes `exp(dtype.min)=0`) also run end to end, both JIT-free. RL cannot use them ([why](#attention-sinks)). Which kernel carries a sink, and how the resolver reads it: [Flash Attention](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#model-specific-handling).
- **Sink-free trainer, live engine.** `reject_unsyncable_model` refuses on-policy RL when the FA2 reset removed the sinks Parameter (`reset_sinks: true` + FA2): the sync sends the trainer's named parameters, so the engine would keep serving the pretrained sinks against a sink-free trainer — permanently off-policy with no error at sync time.
- **Hopper FA2** runs the image's build, whose split-K kernels are stubbed and never picked by a forward Halo runs ([Docker → Building](https://whitecircle.com/halo/docs/agent-docs/infrastructure/docker.md#building)).
- **Variable-length RL** (async GRPO) runs FA4 through a per-row dense forward ([Flash Attention](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#model-specific-handling)).

## Configs

`examples/sft/gptoss/gptoss-20b-multinode-ep.yaml` is the infra-agnostic EP example for GPT-OSS 20B and doubles as the multi-node reference; `gptoss-120b-multinode-ep.yaml` is its 120B counterpart.

The DPO, SMPO, reward, classification and offline GRPO configs pin `flash_attention_4`, so they run on Blackwell only; it is also what those scripts pick there for an unset label. Elsewhere those scripts default an unset label to SDPA ([Flash Attention](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#model-specific-handling)), so on Hopper pass `--attn_implementation=flash_attention_3`. The self-distillation and embedding configs leave it unset, which those scripts resolve to FA4 on Blackwell and FA3 on Hopper.

The DPO, SMPO, reward, classification, offline GRPO, self-distillation and 20B multi-node SFT configs train without gradient checkpointing, the largest single throughput lever here: at EP8 the activations fit to 32k tokens per rank (159 GiB, [Throughput Benchmarks](https://whitecircle.com/halo/docs/agent-docs/optimization/throughput-benchmarks.md#maximizing-throughput-sequence--batch)). The 120B multi-node config keeps it, since it targets 80 GB parts.

A multi-stage fine-tune must agree on `reset_sinks` across stages — a continuation run keeps the sink policy of the export it starts from.

The SFT recipes train from the releases as OpenAI ships them, `openai/gpt-oss-20b` and `openai/gpt-oss-120b`. Their experts are stored in MXFP4 (`gate_up_proj_blocks` / `_scales`, `down_proj_blocks` / `_scales`), and the lazy EP and PP loaders decode them to bf16 while reading, each rank only its own experts, one expert at a time, through transformers' `convert_moe_packed_tensors`. The loaded config drops `quantization_config`, so every full-model save is plain bf16 and, once whole, a checkpoint stock `from_pretrained` reads back: a gathered save as written, a sharded one once `merge_ep_shards.py` has merged it, and a pipeline-parallel save on per-node storage once `merge_pp_stages.py` has merged every node's directory. An adapter save holds only the adapter weights. The BF16 mirrors `unsloth/gpt-oss-{20b,120b}-BF16` still load, at about three times the download.

A whole-model load (tensor or context parallelism without an EP group, ep1, or EP with `ep_lazy_loading: false`) hands the release to `from_pretrained`, and transformers decides there. The Halo images ship no `kernels` package, so its MXFP4 quantizer dequantizes to bf16 itself and every rank stages the whole decoded model: gpt-oss-20b at EP8 on 8× H200 takes about 855 s that way, against 5 s lazily. The recipes that load a frozen reference or teacher beside the policy (preference, distillation, KL-regularized) keep the BF16 mirror.

MXFP4 still serves on vLLM, but not under weight sync: its expert loader has no branch for a bf16 expert tensor, so a synced update drops every expert weight while the biases land, silently ([Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#weight-sync)). On-policy RL serves a BF16 checkpoint, whichever checkpoint the trainer loaded, since the sync pushes bf16 expert tensors.

Async GRPO with Environments: `examples/grpo/environmental/gptoss/vllm/` plus the `sglang/` ep1 siblings — both pinned engines read the interleaved expert pair the gather emits, so either `rollout_backend` takes the weight sync ([Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#which-families-each-engine-serves)).

`gptoss-20b-multinode-ep.yaml` sets `use_grouped_gemm: false`: at its EP=16 the 20B keeps 2 experts per rank, few enough that the loop is competitive, and gpt-oss's square expert FFN (`intermediate == hidden == 2880`, not a multiple of the kernel's 128-tile K) pays a CUTLASS tail epilogue the loop avoids. Grouped GEMM is the default and wins every measured cell through ep8 at batch 4. See [Grouped GEMM](https://whitecircle.com/halo/docs/agent-docs/optimization/grouped-gemm.md#grouped-vs-the-loop-path).

## Chat template

Two templates ship for GPT-OSS under `jinja-templates/gpt-oss/`, selected with `chat_template:`. `gpt-oss-harmony.jinja` is the upstream OpenAI-harmony template from `unsloth/gpt-oss-20b-BF16`, passed to vLLM too, so the training render is **byte-identical** to the server-side one. It deviates twice. Tool and parameter `description` is guarded with `is defined`, so a tool without descriptions does not crash the render (the checkpoint's built-in template renders `param_spec.description` unconditionally and raises without it). A tool call's string `arguments`, the OpenAI schema rollouts store, render verbatim, where the built-in `tojson` double-encodes them; restoring the upstream file trains quoted, escaped arguments no harmony parser reads.

For SFT and self-distillation the shipped examples pin `gpt-oss-multiturn.jinja` with `force_chat_template: true` and `assistant_message_template: <|start|>assistant<|channel|>final<|message|>`: it renders every assistant turn as an empty `analysis` channel followed by the `final` channel the served model emits, and ends every assistant turn with `<|return|>`, so completion-only masking trains each assistant turn of a multi-turn row. Under harmony a plain assistant message renders channel-less as `<|start|>assistant<|message|>…` and only the final turn ends in `<|return|>` (earlier turns end in `<|end|>`, which is not an eos id), so that marker matches no turn at all and every row masks; the channel-less spelling of it would train only the last assistant turn.

For RL the training and serving templates must match, since log-probs are computed on the served prompt. GPT-OSS ships a built-in template, so overriding it needs **both** `chat_template: jinja-templates/gpt-oss/gpt-oss-harmony.jinja` and `force_chat_template: true`. Pass the same file to vLLM via `VLLM_CHAT_TEMPLATE`.

## Serving for GRPO (vLLM)

Use the toolkit `halo:vllm-0.32.0` image, not stock upstream vLLM: on B300 the stock `vllm/vllm-openai` image produces garbage GPT-OSS output with harmony on *or* off. The toolkit image (`Dockerfile.vllm` plus `docker/vllm/patches/patch_vllm_source.py`) forces vLLM's harmony pipeline off and generates coherently.

With harmony disabled, five settings are load-bearing.

- **Text tool parser.** GPT-OSS emits tool calls as plain text (`commentary to=functions.<name> json{...}`), unreadable by the stock `openai` (needs harmony token IDs) and `seed_oss` parsers. Serve with `--enable-auto-tool-choice --tool-parser-plugin /opt/gpt_oss_text_tool_parser.py --tool-call-parser gpt_oss_text` (`docker/vllm/plugins/gpt_oss_text_tool_parser.py`).

    The plugin brace-balances the JSON, returns the `final`-channel text as `content` so the CoT is dropped, rewrites the name-glued token (`submit_solutionjson` → `submit_solution`) to the longest declared tool name it starts with, and coerces malformed arguments into valid JSON so a bad call never 400s the next turn.

    Only the **first** call is extracted: without `<|call|>` as an eos the model keeps writing `to=functions.*` text and hallucinates its own tool result. A header with no brace-balanced JSON after it is dropped rather than rescued, so the turn scores as a no-tool turn and the policy is pushed toward valid JSON. Non-tool RLVR needs no tool parser.

- **Reasoning budget.** `--reasoning-parser-plugin /opt/gpt_oss_reasoning_parser.py --reasoning-parser openai_gptoss` re-registers the harmony-only stock parser for the harmony-disabled path. It separates the CoT from the answer, and is what lets a request carry `thinking_token_budget` at all (without it every such request 400s). When an analysis channel overruns the budget it forces the move into the final channel, `<|end|><|start|>assistant<|channel|>final<|message|>`, which the vLLM recipes name as `rollout_reasoning_end_token` so the loss drops those forced tokens ([Objective](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#importance-sampling-correction)).

    The budget counts only inside an analysis channel: the count starts at an analysis opening (`analysis<|message|>`, the parser's `reasoning_start_str`) and stops at any channel header (`<|channel|>`, its `reasoning_end_str`), so a tool call, a commentary preamble and the final answer are never counted or cut. It is per analysis message, and counts up to the three closing tokens (`<|end|><|start|>assistant`) before the next header, so an analysis ending on its own within three tokens of the budget is turned into final.

    vLLM forces the parser's `reasoning_end_str` unless told otherwise, and stops forcing at the first natural end marker. Two image patches make the gpt-oss cut whole: `vllm_reasoning_forced_end_patch` forces the parser's `reasoning_forced_end_str` (the six-token move) instead, and `vllm_thinking_budget_completion_patch` finishes that sequence past the `<|channel|>` it contains rather than letting the model pick the channel, where a reopened analysis would count afresh. The forced cut carries the role tokens, so it decodes byte-identical to the ending the model writes itself and the split finds the answer behind it.

    `docker/vllm/plugins/verify_gptoss_plugins.py` drives Model Runner V2's own budget kernels (on the Triton interpreter) over simulated harmony-disabled streams at build time: a cut lands right after the budget, a cut model cannot reopen analysis, tool calls, preambles and straight-to-final turns are never cut, and a recycled slot forgets a previous cut. A render or marker drift fails the image build rather than a run.

    The parser keeps the commentary tool call and the `final` answer in `content`: vLLM extracts tool calls from `content` only, so a call left in `reasoning` would be dropped and the turn mis-scored as a giveup.

- **`--return-tokens-as-token-ids`** — required whenever `train_on_sampled_tokens` is on (the default); the compose file passes it ([Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#vllm)).
- **`rollout_stop_tokens: ["<|call|>"]`** — with harmony disabled `<|call|>` is not an eos, so the model keeps generating past its tool call and hallucinates the result for most of the turn. A tool-bearing environment without it is refused at trainer construction. A server-side `--override-generation-config` eos does not fix it ([Async GRPO with Environments](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/episodes.md#tool-calls)).
- **Full padded vocab.** GPT-OSS ships 201088 embedding rows against `len(tokenizer)` 200019; rows 200019–201087 are unassigned padding. Every harmony special sits below that line — `<|return|>`/EOS at 200002, `<|call|>` at 200012, highest added id 200018 — so a resize to `len(tokenizer)` keeps them all.

    The trap is `tokenizer.vocab_size`, which is **199998**: shrink to that and the checkpoint loses EOS and emits garbage. `scripts/before_training/patch_vocab.py` only **grows** the embedding, so a patched checkpoint keeps every special and serves like the base — unless it was patched with `--reset_sinks`, which leaves it sinks-off, and the pretrained model served sinks-off degenerates (below).

Sinks stay **on** at serving (the default): served sinks-off, the pretrained model degenerates to repetitive garbage with zero tool calls. The trainer freezes the same sinks, so both sides score with them; what remains between the two bf16 forwards is routing and rounding noise ([Engine reference](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#engine-reference)).

GPT-OSS serves from **SGLang** for async GRPO (`rollout_backend: sglang`): SGLang's own built-in detectors replace both vLLM plugins — the compose default `--tool-call-parser auto` resolves gpt-oss off the chat template, and `SGLANG_REASONING_PARSER=gpt-oss` separates the analysis channel (both registered in 0.5.22); thinking budgets are rejected at config time for this backend, and the server container needs `NCCL_CUMEM_ENABLE=1` (the compose default). Flags and constraints: [Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#sglang).

## Router balancing

GPT-OSS defaults to switch-style **aux-loss** balancing: `moe_balancing: auto` resolves to `aux_loss`, and that mode turns `output_router_logits` on itself.

The coefficient a run reads is the checkpoint's, not the class default. `GptOssConfig` declares `router_aux_loss_coef: 0.001`, but every released `config.json` (`openai/gpt-oss-*` and the `unsloth/*-BF16` mirrors) ships **0.9**, large enough to dominate both the SFT loss and the router gradient.

The shipped SFT examples that stay on `aux_loss` override it back down to `0.001` in `model_init_kwargs`; do the same on any aux-loss run, DPO and KTO included (their TRL loss weighs the term by the model config's coefficient: [Callbacks](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#moe-balancing-modes)), rather than inheriting 0.9. A `bias_update` run needs no override — that mode zeroes the coefficient itself (below).

The DeepSeek-V3 **aux-loss-free bias update** is opt-in on the EP path (`moe_balancing: bias_update`) and adopts the hub router's own **`router.bias`** as the balancing state. The Parameter is re-registered as a persistent buffer under the same key: frozen out of gradient training, sign-updated by [`BalancingBiasCallback`](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#balancingbiascallback) each optimizer step, and exported with every checkpoint.

vLLM and SGLang load `router.bias` and route with it (top-k on bias-inclusive logits, combine = softmax over the selected values). `_route_with_bias` computes that arithmetic, so both read the trained bias the same way; bf16 near-ties can still pick differently ([Engine reference](https://whitecircle.com/halo/docs/agent-docs/training-methods/async-grpo/objective.md#engine-reference)).

`router_aux_loss_coef` is forced to 0 for the run to avoid double-balancing (restored in the exported config). The bias lives in logit space, so the default γ is small for it; size `router_balancing_rate` per [BalancingBiasCallback](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#balancingbiascallback).

**Not for on-policy RL.** Online and env GRPO must use `moe_balancing: none` — adoption re-registers `router.bias` as a buffer, and the weight sync ships a buffer only where the family declares it in `engine_loaded_buffers`, so a synced engine routes on the pretrained bias (`configure_moe_balancing` downgrades it automatically). Bias-update needs the EP layers (`ep_size > 1`, or the default `use_grouped_gemm`); without them it raises at setup.
