# Step-3.7 Flash

`Step3p7ForConditionalGeneration` (`stepfun-ai/Step-3.7-Flash`) — 198B total / ~11B active composite VLM (196B language backbone + 1.8B vision encoder), native in transformers (`model_type` `step3p7`, text config `step3p5`; no remote code). 45 text layers interleave 12 full-attention and 33 sliding-window layers, every head carries a sigmoid attention gate, and a 288-expert sigmoid-routed MoE carries most of the parameters.

Per-axis support: [Compatibility matrix](https://whitecircle.com/halo/docs/agent-docs/models.md#compatibility-matrix). Pipeline parallelism takes the text tower only: a text-only run of the composite is admitted and re-emits the vision tower in every checkpoint, image data is refused, and stage boundaries land on the period-4 `layer_types` ([Pipeline Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#model-support)).

## Architecture

- **Attention interleave** — `layer_types` repeats full, sliding, sliding, sliding from index 0: 12 full-attention layers (64 heads, half-RoPE with `partial_rotary_factor` 0.5, at θ 5e6, llama3 scaling factor 2 over 131072 original positions → 262144 max) and 33 sliding-window layers (96 heads, window 512, full RoPE at θ 10000).

    All layers are GQA with 8 KV heads, with QK-norm and a per-head sigmoid gate on the attention output (`g_proj`; applied after the attention kernel, so it rides every backend). The per-layer head count lives in `per_layer_config`, and a bare `config.num_attention_heads` read **raises**, so head counts resolve through `get_config_field`'s per-layer seam.

- **Norms** — Gemma-style RMSNorm: normalize in fp32, scale by `weight + 1`.
- **MoE** — layers 3–44 (the first 3 are dense MLP): 288 routed experts, sigmoid top-8 with a selection-only correction bias (`gate.e_score_correction_bias`, added to fp32 sigmoid scores for top-k; the combine weights are the unbiased scores, renormalized), `routed_scaling_factor` 3.0, one shared expert (`share_expert_dim` 1280), `moe_intermediate_size` 1280.

    Layers 43–44 clamp the routed SwiGLU at 7 (`swiglu_limits`, applied **after** the activation) and the shared expert at 16 (`swiglu_limits_shared`); every other layer runs unclamped.

- **MTP** — the checkpoint ships 3 MTP tail layers (indices 45–47); transformers never builds them (their keys are ignored on load), so every export ships without them. The config field `num_nextn_predict_layers: 3` survives as metadata, so vLLM and `generate(use_mtp=True)` refuse an export, and each draft layer owns its head with attention widths that differ from the trunk's layers, so `mtp_loss_weight` [refuses the family](https://whitecircle.com/halo/docs/agent-docs/optimization/mtp-heads.md).

## Checkpoint

The release is plain bf16 (~403 GB of shards) — train from the repo id directly, no conversion step. The sibling `-FP8` / `-NVFP4` repos are serving artifacts, not training inputs.

The hub spells the experts as per-layer fused 3D stacks — `moe.gate_proj [E, M, H]` / `moe.up_proj` / `moe.down_proj` — which transformers' weight converter concatenates into the native fused `experts.gate_up_proj [E, 2M, H]` on load and splits back on save, alongside the renames (`moe.router_bias` ↔ `gate.e_score_correction_bias`, `share_expert.*` ↔ `mlp.shared_experts.*`, the vendor-namespace vision tower). Embeddings are untied (a separate `lm_head.weight`).

An EP-gathered save writes its **weights** in the hub layout the serving engines read: like every family's, it reverts the load's conversions through transformers' save-side revert on every streamed chunk ([Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#serving-on-vllm--sglang)) — here prefix renames back, `moe.gate.weight` / `moe.router_bias` / `share_expert.*`, the fused `gate_up_proj` split back into `moe.gate_proj` + `moe.up_proj`, the vendor-namespace vision tower.

The EP SFT round-trip test pins the on-disk keys against a plain `save_pretrained` of the same config, the expert halves bit-exact against the live gathered tensor, the fp32 `moe.router_bias`, and reload-loss equality.

Nothing restores that layout from a module-spelled file: `scripts/after_training/unfuse_moe_experts.py` refuses the family (its hub layout stores no per-expert tensors), and a transformers load + `save_pretrained` of a module-layout source reverts nothing (the load consumed no conversions). Sharded EP saves (`save_sharded_ep: true`) are refused for the family for the same reason: the offline merge streams key by key and cannot apply the revert.

The `config.json` beside those weights is the **source checkpoint's own**, not transformers'. vLLM 0.32.0 cannot serve the family through transformers' native `step3p7` config class (its config parsing raises `AmbiguousGlobalPerLayerAttributeError` on it) and reads it only through the release's `config.json` and the modules its `auto_map` names, which spell it in the vendor's keys: `moe_num_experts`, `moe_top_k`, `moe_layers_enum`, `attention_other_setting`, per-layer `rope_theta`.

transformers 5.20 absorbs those at load (`Step3p7TextConfig.attribute_map` plus the `kwargs.pop` derivations in `__post_init__`) and re-emits only the native ones, so its own serialization is unservable and nothing on the save side can invert it.

The family spec therefore declares `exports_source_config_schema`, and every toolkit config write carries the source's config and remote-code modules forward with this run's changed values applied under the source's spellings, a `patch_vocab.py` `vocab_size` included ([Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#what-gets-saved)). Training is unaffected: this transformers accepts those spellings, so the export reloads to exactly the config that was trained.

A source that itself declares no `auto_map` (a synthetic checkpoint) has no schema to hand on; the export says so and keeps the native one.

## Model loading

The checkpoint is a composite VLM with **no text-only CausalLM sibling** in transformers, so `text_only_model` is refused for this family — and the refusal sees through the hub config's poisoned `auto_map`, whose `AutoModelForCausalLM` entry is the remote-code *conditional-generation* class itself, not a text-only sibling (`resolve_auto_model_class` rejects a masquerading entry). The composite class loads through the shared VLM path; a text-only dataset still takes the text data path ([SFT — VLMs](https://whitecircle.com/halo/docs/agent-docs/training-methods/sft.md#vision-language-models)).

Train with `trust_remote_code: false` (the default). The repo's `auto_map` names remote-code config, processor and modeling classes for the serving engines; trusting it makes `AutoConfig` return the remote config, which the native class cannot build from (`StepRoboticsVisionEncoderConfig` has no `hidden_size`).

The repo ships no native processor config (`processor_config.json` / `preprocessor_config.json`; its only processor is remote code), so a text run takes the native `AutoTokenizer` as its processing class ([SFT — VLMs](https://whitecircle.com/halo/docs/agent-docs/training-methods/sft.md#vision-language-models)). It carries the chat template: `<|im_start|>`/`<|im_end|>` roles, a `reasoning_effort` template variable, and a generation prompt that forces `<think>`. An image-data run requires the native `Step3p7Processor` and stops at its load unless a processor config sits beside the checkpoint. `tests/cpu/model_loading/test_run_processing_class.py` pins that choice on a tiny checkpoint — the tokenizer for a text run without a processor config, the refusal for an image run without one, `Step3p7Processor` for either run with one — but no test trains on image data.

Two tokenizer facts to hold:

- `tokenizer_config.json` pins `model_max_length` 131072 while the model accepts 262144; set `max_length` explicitly for runs past 128k.
- The 128896-row embedding is deliberately wider than the 128815-entry tokenizer, so do not resize.

## EP layer

`EPStep3p7MoELayer` (`halo/distributed/expert_parallel/families/step3p7.py`) replaces `Step3p7SparseMoeBlock`.

- Routing: fp32 sigmoid scores plus `gate.e_score_correction_bias` on **selection only**, plain top-8 (no group limiting); the combine weights are the unbiased scores renormalized, with the block's `routed_scaling_factor` folded into them (equivalent — the combine is linear in the expert outputs).
- Experts: fused `Step3p7Experts.gate_up_proj [E, 2M, H]` / `down_proj [E, H, M]` in-module. Unclamped layers take the base fused-SiLU combine.

    The clamped layers run a fused Triton **post-activation** clamped SwiGLU: SiLU first, then clamp the activated gate (`GluCombine.clamped_silu(limit, clamp_activated=True)`, logged as `glu_combine=silu_then_clamp`, `halo/kernels/fused_glu.py`; the bound is a runtime kernel argument, so per-layer limits share one kernel). It is the mirror of DeepSeek-V4/GLM-5's pre-activation clamp and not interchangeable with it.

- Shared expert: the block's own `Step3p7MLP`, adopted by reference (it carries its own `swiglu_limits_shared` clamp), added after DeepEP combine. The 3 dense leading layers keep their plain MLP — EP swaps MoE blocks only.

ETP uses the shared fused-GLU helper: at `expert_tp_size > 1` the gate/up halves store as separate shards ([ETP](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-tensor-parallelism.md#limitations)).

EP loads take the lazy safetensors path. The hub checkpoint keeps a vendor namespace (the top-level prefix renames, `moe.*` → `mlp.*`, the two-source `moe.gate_proj + moe.up_proj → gate_up_proj` concatenate, the Step-3.5 vision tower's own chunked, RoPE-permuted `in_proj`).

The lazy loaders plan their keys on transformers' `step3p7` and `step3p5_vision` conversions, as `from_pretrained` does (the vision entries scoped under the tower's module path), while reading only this rank's expert slice through both fan-in sources, bit-exact against `from_pretrained` (`tests/gpu/parallelism/ep/test_lazy_load_converted.py --family step3p7`).

There is no full-model host residency at 198B ([host RAM under the lazy path](https://whitecircle.com/halo/docs/agent-docs/parallelism/large-scale-scenarios.md#other-300b-class-checkpoints)): ~398 GB of page cache node-wide at ep8 (the checkpoint minus the MTP tail, shared by the 8 ranks), up to ~65 GB of file-backed RSS per rank while its shard handles are open. The fan-in's 2× transient (~0.8 GB per layer) is on the GPU.

## Serving for GRPO (vLLM)

Online and async GRPO sync into vLLM 0.32.0, which serves the family from the **hub** namespace (`model.layers.N.moe.gate_proj` / `up_proj` / `down_proj` stacks, `moe.gate.weight`, `share_expert.*`, the vendor-namespace vision tower) and silently drops any language-model name it does not map (an unmapped vision name raises).

The sync forwards that namespace: the same save-side revert the gathered save runs rewrites every forwarded key. The fused `experts.gate_up_proj` gather is split into the two hub stacks, the vision q/k/v are fused back into the un-permuted `in_proj`, and the `model.language_model.*` / `mlp.*` spellings return to the hub's, so the served model receives exactly the tensors a gathered checkpoint would carry.

`tests/cpu/grpo/test_weight_sync_hub_namespace.py` pins the two against `save_pretrained`; `tests/gpu/trainers/grpo/test_step3p7_vllm_weight_sync_e2e.py` proves the server's logprobs follow the trainer's, per weight group, on a live server serving a tiny hub-layout checkpoint, with no training loop. On the release at EP=2 against a TP=2 server, an unchanged push forwards 1,378 tensors and moves no served log-prob on 2,010 positions. Serve the hub repo or a toolkit export directly, and note:

- At the release shape (288 experts × 1280 × 4096) one MoE layer gathers 8.44 GiB, its hub split copies 5.63 GiB beside that, and each 2.81 GiB stack rides alone through its snapshot and send buffer: about 19.7 GiB on the forwarding GPU at the peak (calculated). On the release a sync adds 18 GiB at EP=2 and 16 GiB at EP=4 on that GPU, read by a 2 s sampler. `fp32_non_ep_params` takes the held vision q/k/v past the 1 GiB bound on held dense tensors, so the first push raises at that bound, naming `HALO_WEIGHT_SYNC_CHUNK_MB`, and passes at `HALO_WEIGHT_SYNC_CHUNK_MB=1536`. At EP=4 on 4 GPUs the release takes one async step of expert LoRA at a 140 GiB peak on the forwarding GPU, and a full fine-tune runs out of memory at its first optimizer step.

- `--trust-remote-code` (`VLLM_TRUST_REMOTE_CODE=1` under compose) — vLLM cannot parse transformers' native `step3p7` config, so the composite config is the repo's own `configuration_step3p7.py` (the text tower resolves to vLLM's `step3p5`). A toolkit export ships that module and the ones it imports itself ([Checkpoint](#checkpoint)), so it needs the same flag and nothing else.
- `--moe-backend triton` and the layerwise-reload patch, as for every MoE family ([Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#weight-sync)); the experts are a `RoutedExperts`, which the patch already excludes from the reload lifecycle.
- The router's selection bias (`moe.router_bias`) is a buffer the sync does not ship (the family declares no `engine_loaded_buffers`): it stays at its checkpoint value on both sides, which is exact because weight-sync RL runs `moe_balancing: none` (bias-update balancing is downgraded before any bias state exists).
- vLLM's clamped fused-MoE kernel accepts `swiglu_limits` of exactly 7 (the release value); any other clamp fails at server start, not silently.
- SGLang is refused for the family at construction: `Step3p5ForCausalLM.load_weights` asserts full parameter coverage in each call, which the chunked online update cannot satisfy ([Rollout Servers](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#which-families-each-engine-serves)).

## Router balancing

The router carries a native `e_score_correction_bias` buffer that exports with every checkpoint (hub key `moe.router_bias`), and the modeling has no aux-loss machinery at all (no `router_aux_loss_coef`, no `output_router_logits` parameter). Under the EP layers `moe_balancing: auto` therefore resolves to `bias_update`: [BalancingBiasCallback](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#balancingbiascallback) sign-updates the gate's own slot (selection only) and a served copy routes on it.

On a tree with no EP layer (`ep_size=1` with `use_grouped_gemm: false`) nothing carries the bias and no aux term can reach the loss, so `auto` resolves to `none` with a warning.

## Attention

Upstream declares `_supports_flash_attn = False`; SDPA is the only fast backend — pin `attn_implementation: sdpa` (the resolver diverts any flash request there itself).

On CUDA the model is then built as `sdpa_flex_sliding` ([Flash Attention](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#sliding-window-and-wide-head-layers-on-sdpa)): the 33 sliding layers, and the 12 full layers on packed or padded rows, run compiled FlexAttention under a block mask of the dense mask, skipping the tiles outside the window and across documents that SDPA computes. A full layer over one unpadded document keeps SDPA's causal kernel. The load-time warm-up compiles each layer type's own head count (64 full, 96 sliding). Because it compiles without deterministic mode, a run at world size > 1 refuses `full_determinism` ([Troubleshooting](https://whitecircle.com/halo/docs/agent-docs/reference/troubleshooting.md)). `HALO_FLEX_SLIDING=0` keeps plain SDPA, which takes it.

## Limitations

- **CP** — `Step3p7Attention` has no Ulysses wrapper registered, so validation rejects the model as having no supported attention module ([Context Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/context-parallelism.md#supported-model-architectures)). Nothing architectural blocks a wrapper: both head counts (64 full / 96 sliding) and the 8 KV heads divide cp 2/4/8.
- **TP** — the per-layer head counts fit no uniform q/k/v shard plan, so `Step3p7Attention` is outside the selective-TP accept-list and `tensor_parallel_size > 1` is rejected (zero shardable layers).
- **Packing** — isolated: the layers are plain full/sliding attention (no conv or linear-attention mixers) and the forward feeds `position_ids` into both mask constructions, verified bit-exact through dense layers on SDPA and eager; through MoE layers doc-B drift is expert-summation reduction noise (~1e-7 fp32).

    The isolation holds on the training path only (`use_cache=False`; a live cache suppresses the packed mask, as DeepSeek-V4). See [Collators](https://whitecircle.com/halo/docs/agent-docs/data/collators.md#document-isolation-under-packing).

## Configs

`examples/sft/step3p7/step-3.7-flash-ultrachat-ep-lora.yaml` trains LoRA + expert-LoRA at EP=8 on one
8-GPU node, the single-node shape with headroom (~65 GB static of a 288 GB Blackwell card, expected).
Full fine-tuning at ep8 is ~190 GB of experts at 8 B/param, ~213 GB static: marginal and unmeasured.

The config points `model_name_or_path` at the hub repo directly (bf16 release). Full fine-tuning takes cross-node EP (`ep_scope: global`, e.g. EP=16 across 2×8), where the Gin dispatch ceiling caps `per_device_train_batch_size × max_length` at 8192 tokens/rank ([DeepEP](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md#expert-parallelism-over-aws-efa)).
