Step-3.7 Flash
Step3p7ForConditionalGeneration (stepfun-ai/Step-3.7-Flash) — 198B total / ~11B active composite VLM (196B language backbone + 1.8B vision encoder), native in transformers (model_type step3p7, text config step3p5; no remote code). 45 text layers interleave 12 full-attention and 33 sliding-window layers, every head carries a sigmoid attention gate, and a 288-expert sigmoid-routed MoE carries most of the parameters.
Per-axis support: Compatibility matrix. Pipeline parallelism takes the text tower only: a text-only run of the composite is admitted and re-emits the vision tower in every checkpoint, image data is refused, and stage boundaries land on the period-4 layer_types (Pipeline Parallelism).
Architecture
Attention interleave —
layer_typesrepeats full, sliding, sliding, sliding from index 0: 12 full-attention layers (64 heads, half-RoPE withpartial_rotary_factor0.5, at θ 5e6, llama3 scaling factor 2 over 131072 original positions → 262144 max) and 33 sliding-window layers (96 heads, window 512, full RoPE at θ 10000).All layers are GQA with 8 KV heads, with QK-norm and a per-head sigmoid gate on the attention output (
g_proj; applied after the attention kernel, so it rides every backend). The per-layer head count lives inper_layer_config, and a bareconfig.num_attention_headsread raises, so head counts resolve throughget_config_field's per-layer seam.Norms — Gemma-style RMSNorm: normalize in fp32, scale by
weight + 1.MoE — layers 3–44 (the first 3 are dense MLP): 288 routed experts, sigmoid top-8 with a selection-only correction bias (
gate.e_score_correction_bias, added to fp32 sigmoid scores for top-k; the combine weights are the unbiased scores, renormalized),routed_scaling_factor3.0, one shared expert (share_expert_dim1280),moe_intermediate_size1280.Layers 43–44 clamp the routed SwiGLU at 7 (
swiglu_limits, applied after the activation) and the shared expert at 16 (swiglu_limits_shared); every other layer runs unclamped.MTP — the checkpoint ships 3 MTP tail layers (indices 45–47); transformers never builds them (their keys are ignored on load), so every export ships without them. The config field
num_nextn_predict_layers: 3survives as metadata, so vLLM andgenerate(use_mtp=True)refuse an export, and each draft layer owns its head with attention widths that differ from the trunk's layers, somtp_loss_weightrefuses the family.
Checkpoint
The release is plain bf16 (~403 GB of shards) — train from the repo id directly, no conversion step. The sibling -FP8 / -NVFP4 repos are serving artifacts, not training inputs.
The hub spells the experts as per-layer fused 3D stacks — moe.gate_proj [E, M, H] / moe.up_proj / moe.down_proj — which transformers' weight converter concatenates into the native fused experts.gate_up_proj [E, 2M, H] on load and splits back on save, alongside the renames (moe.router_bias ↔ gate.e_score_correction_bias, share_expert.* ↔ mlp.shared_experts.*, the vendor-namespace vision tower). Embeddings are untied (a separate lm_head.weight).
An EP-gathered save writes its weights in the hub layout the serving engines read: like every family's, it reverts the load's conversions through transformers' save-side revert on every streamed chunk (Checkpoints) — here prefix renames back, moe.gate.weight / moe.router_bias / share_expert.*, the fused gate_up_proj split back into moe.gate_proj + moe.up_proj, the vendor-namespace vision tower.
The EP SFT round-trip test pins the on-disk keys against a plain save_pretrained of the same config, the expert halves bit-exact against the live gathered tensor, the fp32 moe.router_bias, and reload-loss equality.
Nothing restores that layout from a module-spelled file: scripts/after_training/unfuse_moe_experts.py refuses the family (its hub layout stores no per-expert tensors), and a transformers load + save_pretrained of a module-layout source reverts nothing (the load consumed no conversions). Sharded EP saves (save_sharded_ep: true) are refused for the family for the same reason: the offline merge streams key by key and cannot apply the revert.
The config.json beside those weights is the source checkpoint's own, not transformers'. vLLM 0.32.0 cannot serve the family through transformers' native step3p7 config class (its config parsing raises AmbiguousGlobalPerLayerAttributeError on it) and reads it only through the release's config.json and the modules its auto_map names, which spell it in the vendor's keys: moe_num_experts, moe_top_k, moe_layers_enum, attention_other_setting, per-layer rope_theta.
transformers 5.20 absorbs those at load (Step3p7TextConfig.attribute_map plus the kwargs.pop derivations in __post_init__) and re-emits only the native ones, so its own serialization is unservable and nothing on the save side can invert it.
The family spec therefore declares exports_source_config_schema, and every toolkit config write carries the source's config and remote-code modules forward with this run's changed values applied under the source's spellings, a patch_vocab.py vocab_size included (Checkpoints). Training is unaffected: this transformers accepts those spellings, so the export reloads to exactly the config that was trained.
A source that itself declares no auto_map (a synthetic checkpoint) has no schema to hand on; the export says so and keeps the native one.
Model loading
The checkpoint is a composite VLM with no text-only CausalLM sibling in transformers, so text_only_model is refused for this family — and the refusal sees through the hub config's poisoned auto_map, whose AutoModelForCausalLM entry is the remote-code conditional-generation class itself, not a text-only sibling (resolve_auto_model_class rejects a masquerading entry). The composite class loads through the shared VLM path; a text-only dataset still takes the text data path (SFT — VLMs).
Train with trust_remote_code: false (the default). The repo's auto_map names remote-code config, processor and modeling classes for the serving engines; trusting it makes AutoConfig return the remote config, which the native class cannot build from (StepRoboticsVisionEncoderConfig has no hidden_size).
The repo ships no native processor config (processor_config.json / preprocessor_config.json; its only processor is remote code), so a text run takes the native AutoTokenizer as its processing class (SFT — VLMs). It carries the chat template: <|im_start|>/<|im_end|> roles, a reasoning_effort template variable, and a generation prompt that forces <think>. An image-data run requires the native Step3p7Processor and stops at its load unless a processor config sits beside the checkpoint. tests/cpu/model_loading/test_run_processing_class.py pins that choice on a tiny checkpoint — the tokenizer for a text run without a processor config, the refusal for an image run without one, Step3p7Processor for either run with one — but no test trains on image data.
Two tokenizer facts to hold:
tokenizer_config.jsonpinsmodel_max_length131072 while the model accepts 262144; setmax_lengthexplicitly for runs past 128k.- The 128896-row embedding is deliberately wider than the 128815-entry tokenizer, so do not resize.
EP layer
EPStep3p7MoELayer (halo/distributed/expert_parallel/families/step3p7.py) replaces Step3p7SparseMoeBlock.
Routing: fp32 sigmoid scores plus
gate.e_score_correction_biason selection only, plain top-8 (no group limiting); the combine weights are the unbiased scores renormalized, with the block'srouted_scaling_factorfolded into them (equivalent — the combine is linear in the expert outputs).Experts: fused
Step3p7Experts.gate_up_proj [E, 2M, H]/down_proj [E, H, M]in-module. Unclamped layers take the base fused-SiLU combine.The clamped layers run a fused Triton post-activation clamped SwiGLU: SiLU first, then clamp the activated gate (
GluCombine.clamped_silu(limit, clamp_activated=True), logged asglu_combine=silu_then_clamp,halo/kernels/fused_glu.py; the bound is a runtime kernel argument, so per-layer limits share one kernel). It is the mirror of DeepSeek-V4/GLM-5's pre-activation clamp and not interchangeable with it.Shared expert: the block's own
Step3p7MLP, adopted by reference (it carries its ownswiglu_limits_sharedclamp), added after DeepEP combine. The 3 dense leading layers keep their plain MLP — EP swaps MoE blocks only.
ETP uses the shared fused-GLU helper: at expert_tp_size > 1 the gate/up halves store as separate shards (ETP).
EP loads take the lazy safetensors path. The hub checkpoint keeps a vendor namespace (the top-level prefix renames, moe.* → mlp.*, the two-source moe.gate_proj + moe.up_proj → gate_up_proj concatenate, the Step-3.5 vision tower's own chunked, RoPE-permuted in_proj).
The lazy loaders plan their keys on transformers' step3p7 and step3p5_vision conversions, as from_pretrained does (the vision entries scoped under the tower's module path), while reading only this rank's expert slice through both fan-in sources, bit-exact against from_pretrained (tests/gpu/parallelism/ep/test_lazy_load_converted.py --family step3p7).
There is no full-model host residency at 198B (host RAM under the lazy path): 398 GB of page cache node-wide at ep8 (the checkpoint minus the MTP tail, shared by the 8 ranks), up to ~65 GB of file-backed RSS per rank while its shard handles are open. The fan-in's 2× transient (0.8 GB per layer) is on the GPU.
Serving for GRPO (vLLM)
Online and async GRPO sync into vLLM 0.32.0, which serves the family from the hub namespace (model.layers.N.moe.gate_proj / up_proj / down_proj stacks, moe.gate.weight, share_expert.*, the vendor-namespace vision tower) and silently drops any language-model name it does not map (an unmapped vision name raises).
The sync forwards that namespace: the same save-side revert the gathered save runs rewrites every forwarded key. The fused experts.gate_up_proj gather is split into the two hub stacks, the vision q/k/v are fused back into the un-permuted in_proj, and the model.language_model.* / mlp.* spellings return to the hub's, so the served model receives exactly the tensors a gathered checkpoint would carry.
tests/cpu/grpo/test_weight_sync_hub_namespace.py pins the two against save_pretrained; tests/gpu/trainers/grpo/test_step3p7_vllm_weight_sync_e2e.py proves the server's logprobs follow the trainer's, per weight group, on a live server serving a tiny hub-layout checkpoint, with no training loop. On the release at EP=2 against a TP=2 server, an unchanged push forwards 1,378 tensors and moves no served log-prob on 2,010 positions. Serve the hub repo or a toolkit export directly, and note:
At the release shape (288 experts × 1280 × 4096) one MoE layer gathers 8.44 GiB, its hub split copies 5.63 GiB beside that, and each 2.81 GiB stack rides alone through its snapshot and send buffer: about 19.7 GiB on the forwarding GPU at the peak (calculated). On the release a sync adds 18 GiB at EP=2 and 16 GiB at EP=4 on that GPU, read by a 2 s sampler.
fp32_non_ep_paramstakes the held vision q/k/v past the 1 GiB bound on held dense tensors, so the first push raises at that bound, namingHALO_WEIGHT_SYNC_CHUNK_MB, and passes atHALO_WEIGHT_SYNC_CHUNK_MB=1536. At EP=4 on 4 GPUs the release takes one async step of expert LoRA at a 140 GiB peak on the forwarding GPU, and a full fine-tune runs out of memory at its first optimizer step.--trust-remote-code(VLLM_TRUST_REMOTE_CODE=1under compose) — vLLM cannot parse transformers' nativestep3p7config, so the composite config is the repo's ownconfiguration_step3p7.py(the text tower resolves to vLLM'sstep3p5). A toolkit export ships that module and the ones it imports itself (Checkpoint), so it needs the same flag and nothing else.--moe-backend tritonand the layerwise-reload patch, as for every MoE family (Rollout Servers); the experts are aRoutedExperts, which the patch already excludes from the reload lifecycle.The router's selection bias (
moe.router_bias) is a buffer the sync does not ship (the family declares noengine_loaded_buffers): it stays at its checkpoint value on both sides, which is exact because weight-sync RL runsmoe_balancing: none(bias-update balancing is downgraded before any bias state exists).vLLM's clamped fused-MoE kernel accepts
swiglu_limitsof exactly 7 (the release value); any other clamp fails at server start, not silently.SGLang is refused for the family at construction:
Step3p5ForCausalLM.load_weightsasserts full parameter coverage in each call, which the chunked online update cannot satisfy (Rollout Servers).
Router balancing
The router carries a native e_score_correction_bias buffer that exports with every checkpoint (hub key moe.router_bias), and the modeling has no aux-loss machinery at all (no router_aux_loss_coef, no output_router_logits parameter). Under the EP layers moe_balancing: auto therefore resolves to bias_update: BalancingBiasCallback sign-updates the gate's own slot (selection only) and a served copy routes on it.
On a tree with no EP layer (ep_size=1 with use_grouped_gemm: false) nothing carries the bias and no aux term can reach the loss, so auto resolves to none with a warning.
Attention
Upstream declares _supports_flash_attn = False; SDPA is the only fast backend — pin attn_implementation: sdpa (the resolver diverts any flash request there itself).
On CUDA the model is then built as sdpa_flex_sliding (Flash Attention): the 33 sliding layers, and the 12 full layers on packed or padded rows, run compiled FlexAttention under a block mask of the dense mask, skipping the tiles outside the window and across documents that SDPA computes. A full layer over one unpadded document keeps SDPA's causal kernel. The load-time warm-up compiles each layer type's own head count (64 full, 96 sliding). Because it compiles without deterministic mode, a run at world size > 1 refuses full_determinism (Troubleshooting). HALO_FLEX_SLIDING=0 keeps plain SDPA, which takes it.
Limitations
CP —
Step3p7Attentionhas no Ulysses wrapper registered, so validation rejects the model as having no supported attention module (Context Parallelism). Nothing architectural blocks a wrapper: both head counts (64 full / 96 sliding) and the 8 KV heads divide cp 2/4/8.TP — the per-layer head counts fit no uniform q/k/v shard plan, so
Step3p7Attentionis outside the selective-TP accept-list andtensor_parallel_size > 1is rejected (zero shardable layers).Packing — isolated: the layers are plain full/sliding attention (no conv or linear-attention mixers) and the forward feeds
position_idsinto both mask constructions, verified bit-exact through dense layers on SDPA and eager; through MoE layers doc-B drift is expert-summation reduction noise (~1e-7 fp32).The isolation holds on the training path only (
use_cache=False; a live cache suppresses the packed mask, as DeepSeek-V4). See Collators.
Configs
examples/sft/step3p7/step-3.7-flash-ultrachat-ep-lora.yaml trains LoRA + expert-LoRA at EP=8 on one
8-GPU node, the single-node shape with headroom (~65 GB static of a 288 GB Blackwell card, expected).
Full fine-tuning at ep8 is ~190 GB of experts at 8 B/param, ~213 GB static: marginal and unmeasured.
The config points model_name_or_path at the hub repo directly (bf16 release). Full fine-tuning takes cross-node EP (ep_scope: global, e.g. EP=16 across 2×8), where the Gin dispatch ceiling caps per_device_train_batch_size × max_length at 8192 tokens/rank (DeepEP).
Source on GitHubUpdated 18 November 2026