Qwen3.8-Flash-Next

model_type qwen4_exp (Qwen4ExpForConditionalGeneration, the release Qwen/Qwen3.8-Flash-Next) wrapping the text tower qwen4_exp_text (Qwen4ExpForCausalLM). text_only_model: true loads the text tower and drops the vision tower and the mtp.* head; the multimodal class loads too: its forward is tiny-model verified on image and packed text rows, and no test trains it. The release has 48 decoder layers: 36 GatedDeltaNet linear_attention layers and 12 sparse-attention (QSA) layers, which the hub spells full_attention and transformers maps to indexed_attention. Four hyper-connection streams carry the residual; every layer's MoE block routes 512 experts at top-10 beside a sigmoid-gated shared expert; the second layer adds a per-layer embedding (PLE) looked up in a hashed n-gram table of 320,001,536 rows × 160 (95 GiB in bf16, split_ngram_parts: 128 shards on the hub).

Experimental. GPU tests on tiny models cover EP and EP+ETP against the reference model, pure ETP, the gathered-save round trip, deterministic expert gradients, LoRA merged-save resume and DPO/KTO precompute resume. On the release under EP8, tests/gpu/models/test_qwen4_exp_real_checkpoint.py has run its load, export and resume modes: the cut to four layers loads lazily and matches the unsharded model, trains, and its gathered save reloads in stock transformers at the trained model's NLL; a full-depth export carries every release tensor unchanged; and on the cut, full fine-tuning resumed from a midway checkpoint replays the unbroken run's losses and final weights bit for bit. At full depth only a few steps of the EP8 LoRA recipe and 13 steps of pp2 × ep8 full fine-tuning on the text tower have run. Per-axis support: Compatibility matrix. Methods: SFT with full fine-tuning or LoRA (the recipes); DPO and KTO are tiny-model tested only, offline GRPO CPU-checked only. EP runs with lazy loading, and PP runs the text tower (Pipeline parallelism); TP and CP are refused, and RL weight sync into SGLang takes the toolkit's refreshed image only (RL weight sync). At ep_size: 1 (plain FSDP2) every GPU loads the whole model, about 246 GiB at full size: tiny and test models only.

EP layer

EPQwen4ExpMoELayer (halo/distributed/expert_parallel/families/qwen4_exp.py) subclasses Qwen3-Next's layer: Qwen3.5-MoE's fused gate_up_proj [E, 2M, H] / down_proj [E, H, M] experts and replicated gated shared expert, under a router that renormalizes the top-k weights only under norm_topk_prob. The release stores the experts fused under the block's own names, so the spec keeps the default FUSED layout and the gathered save writes experts.gate_up_proj / experts.down_proj as the hub does. Hidden 2560 is a multiple of 256, so DeepEP pads nothing.

N-gram table

The table never enters the module state. The family patches (halo/models/families/qwen4_exp/forwards.py, applied by apply_family_patches on every policy, trainer-path and frozen reference or teacher load) stamp the config the model builds on, and a stamped Qwen4ExpTextNGramEmbedding holds a HostNGramTable (halo/models/families/qwen4_exp/ngram_table.py) in place of its nn.Embedding: no parameter, buffer or state-dict entry, so FSDP2, the optimizer, the fp32 masters, PEFT, the coverage gate and the lazy planner never see it, and it trains frozen (the stock class trains it). A build without the stamp keeps the stock class and holds the table as a parameter: a checkpoint tool's from_pretrained. The sentence-transformers embedding load, which reads its backbone config itself, refuses a backbone with external tables.

A model built without the patches (from an unstamped config, or before apply_family_patches ran) would train the table as a parameter and run transformers' forwards. apply_run_model_policies therefore runs verify_qwen4_exp_patched, which raises on any n-gram embedding still holding an nn.Embedding or on forwards that are not the toolkit's, naming apply_family_patches as the remedy; it covers the policy and frozen reference or teacher loads (the embedding load refuses the family above, and reward scoring never runs the policies). A model handed to a trainer already built is not checked there. A checkpoint tool's stock load never reaches it.

finalize_loaded_model attaches the table from the local directory the weights were read from: the snapshot of the commit every rank agreed on, which the policy loaders, the trainer-path load and a frozen reference or teacher load that agreed its source name for a Hub source (the lazy EP loader reads it directly). A model loaded outside them whose config records a Hub id is refused, since transformers records no revision to pair the table with the weights. The parts are zero-copy views of the ngram_embedding.shard_* tensors, so every rank on a node shares one page-cache copy, and local rank 0 asks the kernel to read the table files ahead. A lookup gathers its rows on the CPU (one thread per rank) and copies them, about 5 KB per token, to the device through pinned memory; the reentrant recompute of the PLE layer reads the first pass's rows back. Part count, row count, width and dtype are checked against the config; a model whose config names no checkpoint, or one without the table, is refused, and so is a random-weight build (init_from_scratch, refused before the build), whose config still names the pretrained checkpoint.

The n-gram embedding copies eos_token_id when it is built and restarts its history after it. A run's eos_token that would record another id is refused (reject_stale_eos_copies, halo/model_loading/tokenizer_setup.py): training would restart at the copy while every export records the new id.

Checkpoints and serving

Every gathered save (EP gathered, FSDP2) writes the table as table-only files (model-table00-*.safetensors) under the hub's ngram_embedding.shard_{i}.weight keys, beside the streamed weights, and a PP save writes it from the stage holding the PLE layer (Pipeline parallelism); a model with the table always gets an index, and its exported config.json keeps split_ngram_parts and carries no run stamp (Checkpoints). One process or DDP at use_grouped_gemm: false, which no parallel saver claims, streams the same way rather than through transformers' save_pretrained, whose state dict holds no table (the family patch refuses a direct call); accelerate-managed FSDP, whose saves stay with save_pretrained, is refused at trainer construction unless the run trains adapters. The first save of a process writes the table once from the attached views; each later save hard-links the previous save's files, and a resume attached from a step checkpoint links on its first save. On per-node output storage every node's save rank keeps its own copy of the 95 GiB table. Full fine-tuning resumes from the step checkpoint and attaches the table from it; LoRA resumes onto the base. save_sharded_ep: true is refused: merge_ep_shards.py has no model to read the table from.

An export loads in stock transformers 5.20 and carries no mtp.* head, and mtp_loss_weight refuses the family: its draft layer reads hyper-connection streams the trunk's layers carry between them, which an embedding and a hidden state do not supply. vLLM 0.32 registers Qwen4ExpForCausalLM and reads the table as its shard_{i} keys, and an untrained full-depth export served at TP8 returns the release's prompt log-probs exactly (65,504 compared). SGLang 0.5.22 registers only the multimodal class, so run reattach_vision_tower.py first, which also streams the base's untrained MTP head back. Attention and GatedDeltaNet adapters merge with merge_peft_adapters.py, which loads the stock class (the table as a parameter, about 360 GB of host RAM); expert adapters fold with merge_expert_lora_on_save: true.

Attention and rows

The class declares no flash or flex support, so the load takes SDPA. transformers 5.20's QSA scores every query in a Python loop and its model forward builds a [B, 1, T, T] mask, which leaves rows past a few thousand tokens impractical; patch_qwen4_exp_document_forwards (halo/models/families/qwen4_exp/forwards.py, applied with the n-gram patch) replaces both for every uncached forward:

  • The model forward hands its layers the 2-D padding mask in place of the mask mapping, so no [T, T] mask exists.
  • A QSA layer selects each query's key blocks per 128-query tile against each document's pooled blocks, and re-scores a query whose k-th and next block tie in transformers' own order, so it attends the keys transformers' indexer gives it, up to a near-tie the two matmul shapes round apart. It attends either on the block-sparse kernel, which reads only the selected blocks and the tail, or on masked SDPA tiles over K/V from the document's first key to the tile end, which save only Q/K/V and the block ids and recompute each tile in the backward (halo/models/families/qwen4_exp/sparse_attention.py). HALO_QSA_KERNEL picks (auto by default: the kernel where it can run; off; on or a ValueError). The kernel accumulates dK and dV with fp32 atomics, so under full_determinism on raises ValueError and auto logs the reason once on rank 0 and takes the tiled path; deterministic runs use auto or off. Memory is linear in the row; the tiled path's compute is dense causal per document, not a flash kernel.
  • The PLE n-gram history restarts at every document start as well as after eos_token_id, and its dilated conv runs as masked shifted multiply-adds whose taps stop at the document start, off cuDNN (halo/models/families/qwen4_exp/documents.py). An adapter on the PLE conv1d is refused.

The family code reads the QSA indexer (heads, head width, budget, compress ratio), the hyper-connection count, the PLE dimensions and conv, the n-gram shape, split_ngram_parts and eos_token_id off the built modules and the config, never as constants: tests/cpu/models/test_qwen4_exp_config_driven.py runs a tiny model with all of them off the defaults against transformers' logits. QSA requires indexer_kv_heads: 1, as transformers does, and a table build without split_ngram_parts raises.

Documents come from seq_idx, which the packing collator emits for this family beside GatedDeltaNet's cu_seq_lens, so each document of a packed row computes what it computes alone. A row whose position_ids restart at 0 without seq_idx (and without a padding mask) is refused. A call with a cache or use_cache=True, as in generation, keeps transformers' forwards; one that leaves use_cache unset runs per document and returns no cache.

Under gradient checkpointing the recompute reuses the forward's QSA selection and n-gram rows; each QSA layer holds its selection, about 1 KB per token plus 8 B of spans, until its backward (about 400 MB at 32k tokens across the 12 layers). max_length must be set; the config's 262,144 exceeds the DeepEP token ceiling. The hyper-connections (Qwen4ExpTextGatedResidual) and the injection of a block's output into the streams run on Triton kernels by default (HALO_GATED_RESIDUAL_KERNELS=0 turns them off) (Flash-Next gated residual); they are layer-local, so EP, ETP, FSDP2, pipeline stages, LoRA and both checkpointing modes need nothing of them.

Kernel defaults

HALO_QSA_KERNEL=auto and HALO_GATED_RESIDUAL_KERNELS=1 are the defaults; HALO_FLA_L2NORM_FIXED_NB and HALO_TORCH_NUM_THREADS keep theirs. Measured on 8 B300 with the real checkpoint under EP8 (seed 42, 14 steps at gradient accumulation 4; medians of steps 3 to 14 per run, then over 3 interleaved runs; the bracket is the spread of the 3 run medians):

ArmRecipeStep timeTokens/s/GPUPeak allocated
offLoRA, 16k packed28.95 s (27.86 to 29.40)2,25498.71 GB
QSA kernelLoRA, 16k packed18.38 s (18.32 to 19.27)3,52598.70 GB
gated residualLoRA, 16k packed23.66 s (23.19 to 26.18)2,74296.52 GB
FLA l2norm fixed NBLoRA, 16k packed27.87 s (27.11 to 28.74)2,33698.69 GB
all threeLoRA, 16k packed15.93 s (14.93 to 16.45)4,10796.52 GB
4 CPU threadsLoRA, 16k packed24.86 s (24.58 to 25.05)2,62198.69 GB
offfull, 8k13.24 s (12.82 to 15.30)2,466155.6 GB
all threefull, 8k8.54 s (8.47 to 8.89)3,813154.1 GB
  • Loss. Every LoRA arm follows the same trajectory, 3.33 to 0.85 over the 14 steps, with no trend between arms. The kernel arms stay within 0.021 per step of the off arm of the same run, against 0.011 to 0.016 between two off runs (LoRA, three pairs) and 0.012 to 0.018 (full recipe); the four-thread arm reaches 0.031.
  • Defaults flipped. The QSA kernel (1.57×) and the gated residual (1.22×) clear a gain above the off arm's 5% spread, with no peak increase, the same loss trajectory and no compile after the warm-up.
  • l2norm stays off. Its 3.7% gain is below the 5% spread of the off runs, and the knob is global: it also applies to Qwen3.5 and Qwen3-Next.
  • Threads stay at 1. Four threads give 1.16× here, but HALO_TORCH_NUM_THREADS sets the CPU use of every run of every family and was measured on this recipe alone, without the kernels on.
  • Profile. Torch's mem-efficient attention backward is 22% of the off step's kernel time (10.4 s of 47.3 s over two steps) and about 470k small elementwise launches another 8.5%; with the kernels the total is 26.6 s, the block-sparse forward and backward take 9.2% and 6.1%, and the elementwise launches fall to 13.8k.

Recipes

examples/sft/qwen4_exp/qwen3.8-flash-next-ep8-lora.yaml (16k packed rows) and -ep8-full.yaml (8k, B300) train the text tower at ep8 on one node with the fused loss. LoRA sets moe_balancing: none (the router is frozen), full fine-tuning bias_update_transient. Both pin Halo's qwen3-multiturn.jinja (its first-step loss on Qwen3.6): the hub template is Qwen3.8-27B's, with its reasoning-effort caveats. Both leave eos_token unset: the PLE n-gram history restarts after the checkpoint's eos_token_id (<|endoftext|>), and a run's eos_token that changes it is refused (N-gram table).

LoRA

PEFT adapts attention (q_proj, k_proj, v_proj, o_proj) and GatedDeltaNet (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj); routed experts take Halo's native expert LoRA under EP (PEFT). An adapter on indexer.index_qk_proj never gets a gradient: the indexer's top-k is a boolean selection, in transformers as here. List targets explicitly, as the LoRA recipe does. The merged-save GPU rows adapt the attention projections and experts; the GatedDeltaNet projections ran through the EP8 recipe.

Balancing

The router has no bias slot. On the family's own head moe_balancing: auto resolves to aux_loss. The Liger spec declares the head's in-head aux loss, so under the fused loss output_router_logits is refused and auto resolves to none with a warning; pick bias_update_transient (trainer-only, no export carries the bias) or none explicitly. bias_update raises (Callbacks).

Pipeline parallelism

Qwen4ExpTextModel is registered for PP, tiny-model GPU tested at pp2 x ep2 and pp2 x etp2 with a PLE layer on each stage. On the release, pp2 × ep8 across two 8-GPU machines ran 13 steps with GC on at 16.89 s a step, 1,940 tok/s/GPU as logged (padding counted), at 115.7 / 78.2 GiB; with GC off the first step runs out of memory at about 260 GiB. Its stages carry the four-stream residual across each boundary at hc_count times the hidden width (20 KiB per token), and hyper_connection_mixer collapses it on the last stage alone. Every per-layer input hangs off the layer itself, so any split offset is exact.

  • Token ids. A stage holding a PLE layer looks its n-grams up from the token ids, which the runner hands every later stage beside the activation (Token ids on later stages). The release's one PLE layer (layer 1) sits on stage 0 whenever stage 0 holds at least two layers.
  • Packed rows. Each stage rebuilds the row's seq_idx and varlen set from its microbatch's positions, so the QSA indexer, the PLE history and the GatedDeltaNet layers keep each document apart (PP packing).
  • Checkpoints. The stage holding a PLE layer writes its table beside its parts, as model-ppSSSSS-of-PPPPP-table00-* under the unsplit model's keys; a resume attaches each stage's table from those files, and merge_pp_stages.py carries them like any part.
  • Limits. PP refuses adapters, so it trains full fine-tunes; with aux_loss severed by the split and no bias slot, balance with moe_balancing: none. Checkpointing runs non-reentrant, the QSA selection and n-gram rows replayed from each checkpoint frame. Load with text_only_model: true, or run the composite text-only, which re-emits its vision tower in every checkpoint; image data is refused.

RL weight sync

The weight sync into vLLM 0.32.0 and into SGLang 0.5.22 is exact on tiny composites at two ranks (the push-exactness and LoRA sync rows); SGLang only from the toolkit's image. No GRPO run on the real checkpoint exists yet, and the family stays experimental. Both need the composite: text_only_model is refused, since the sync sends model.layers.* names where the served class expects model.language_model.layers.*.

  • vLLM. tests/gpu/trainers/grpo/test_weight_push_exact.py --family qwen4_exp_text --backend vllm --reach passes on a tiny composite checkpoint the trainer's own gathered save writes (--write-checkpoint): an unchanged push leaves every served prompt log-prob bit-identical, a scaled attention weight moves them and the restoring push returns them exactly, every forwarded name but the indexer's reaches the served model (scaling an indexer weight leaves the key ranking where it was), and after a real optimizer step the pushed server sits as close to the trainer as the two were before it. vLLM's model reads each weight live and builds no copy of one at load or first use. The row runs at ep_size: 2; the server needs --moe-backend triton like every family, and the checkpoint's indexer_budget over indexer_compress_ratio must be 512, 1024 or 2048 (the release's is 512), which the server's top-k kernel enforces. A composite export served by vLLM needs deepstack_visual_indexes: [] in its vision_config, which transformers' vision config drops on save and the release's config.json carries.
  • SGLang. On SM100-family GPUs (B200, B300) each hyper-connection GatedResidual builds a padded, permuted copy of its input_mix_weight_up the first time it mixes a batch of at most 24 tokens, and stock SGLang never rebuilds it, so an update does not reach it. The toolkit's image rewrites the copy in place and reports hyperconnection_mix_up on /server_info (SGLang loader facts); the startup preflight refuses a server that does not. SGLang registers only the multimodal class, so a text-only export goes through reattach_vision_tower.py first. The same test passes with --backend sglang --reach on that image.

Offline GRPO trains on pre-collected completions with no server, so no refusal reaches it (Offline GRPO). The n-gram table is not a gap: it trains frozen, so the sync has nothing to send for it, and each server loads its own copy from the checkpoint at launch.

examples/grpo/environmental/qwen4_exp/vllm/qwen3.8-flash-next-exam-qa-full-ep8.yaml is the environmental recipe: full fine-tuning at EP=8 on one node and a vLLM server (qwen3_xml tool parser, qwen3 reasoning parser, the checkpoint's own template) on another. It is config-checked only; no GPU run has trained it.

Refused modes

  • CP: 36 of 48 layers are linear_attention, whose conv and recurrent scan run along the sequence axis.
  • TP: Qwen4ExpTextAttention is not in TP_SHARDABLE_ATTENTION_CLASSES, so a TP run would shard no layer.
  • padding_free, tree packing: SDPA reads no cu_seqlens, and tree packing admits only the families it is verified on.
  • Packed rows without seq_idx: the model forward refuses them (above).
  • init_from_scratch and an eos_token that changes eos_token_id (N-gram table), a direct save_pretrained and a full fine-tune under accelerate-managed FSDP (Checkpoints and serving).
  • RL weight sync into an SGLang server without the mix-up patch: the startup preflight (above). Offline GRPO needs no server.

Qwen4-lineage checklist

transformers' modular converter writes every class standalone: modular_qwen4_exp.py derives the attention from Qwen3.5's and the MoE block from Qwen3-Next's, but the generated Qwen4ExpTextAttention and Qwen4ExpTextSparseMoeBlock subclass nn.Module. A registry keyed on a class name carries nothing over by MRO, so a family built from qwen4_exp's blocks (GatedDeltaNet, QSA, hyper-connections, the PLE n-gram table, fused experts, MTP) needs its own entry in each:

  • EP: a MoEFamilySpec in MOE_FAMILY_SPECS (halo/models/moe/family_specs.py) claiming every model_type spelling, and an EP layer under halo/distributed/expert_parallel/families/ that names its block class in HF_MODULE_NAMES, imported in roster.py; it subclasses EPQwen4ExpMoELayer where the block matches (Add EP support).
  • Family patches: apply_family_patches (halo/model_loading/model_preparation.py) applies qwen4_exp's on a model_type starting with qwen4_exp, so a family whose model_type does not needs its own branch. The patches (halo/models/families/qwen4_exp/) replace methods by name on classes of transformers.models.qwen4_exp.modeling_qwen4_exp: the n-gram embedding's host table, the save_pretrained refusal, and the per-document model, decoder-layer and QSA forwards. verify_upstream_parameters (halo/models/upstream/upstream_guard.py) refuses a transformers release whose patched signatures moved.
  • Liger: a LigerFamilySpec in halo/model_loading/liger/family_specs.py. qwen4_exp's declares the head only (causal_lm, router_aux_loss_in_head): its RMSNorm class also runs the hyper-connection norms grouped per stream, and its rotary is interleaved mrope (Declare Liger coverage).
  • PP: a PPFamilySpec in PP_FAMILY_SPECS (halo/distributed/pipeline_parallel/families.py) claiming the backbone class, QWEN4_EXP_PP the template: final_norm_attrs names the stream collapse hyper_connection_mixer, stream_forward runs the stages that receive the widened [B, S, hc_count * D] stream, token_ids_kwarg hands the token ids to the PLE layers on later stages, and layer_types_rebase_safe holds because every per-layer input hangs off the layer itself. The per-stage segment markers and the PP save and attach of external tables take no entry. validated=True follows the family's PP GPU rows, which tests/cpu/conventions/test_pp_family_roster.py requires (test_pp_qwen4_exp_vs_reference.py the template).
  • Weight sync: qwen4_exp's spec carries no weight_sync_refusal; SGLang's client names the image patch it needs in REQUIRED_PATCHES (RL weight sync). A new spec starts without a refusal, so a family built from these blocks declares its own, or a per-engine loader fact in a client's UNSERVABLE_MODEL_TYPES (halo/weight_sync/clients/), until an engine run validates its sync. A family vLLM's sync admits needs a config-parity fixture under docker/vllm/parity/fixtures/, which the vLLM image build checks (tests/cpu/checkpoint/test_vllm_parity_fixtures.py fails without it); a family either engine admits needs its sync-exactness rows, _SYNC_EXACTNESS_SWEEP_FAMILIES in tests/gpu/manifest.py (test_lora_weight_sync_exact.py), which tests/cpu/conventions/test_tiny_family_roster.py holds to the syncable families, and its test_weight_push_exact.py rows.
  • External tables: a frozen table is an ExternalTable subclass (halo/models/external_tables.py; qwen4_exp's HostNGramTable), which the family's patched constructor builds when the config carries EXTERNAL_TABLES_STAMP. The loaders attach, and the gathered savers write, every table the module walk finds, so no saver takes a per-family entry; save_sharded_ep and a full fine-tune under accelerate-managed FSDP refuse a model holding one (Checkpoints).
  • Tiny test configs: the text tower in TINY_MOE_FAMILIES and the composite in TINY_MOE_VLM_FAMILIES (tests/common/tiny_models.py), with ulysses_cp=False; random init zeroes the PLE conv, which _with_live_ple_convs redraws on Qwen4ExpTextPLELayer by class. The text tower goes in _TINY_MOE_FAMILY_MARKERS (tests/gpu/manifest.py), its spellings in tests/cpu/parallelism/test_hf_model_types_completeness.py. The CPU EP-vs-reference comparison takes a fixture in FAMILIES (tests/cpu/parallelism/test_ep_vs_reference_equivalence_cpu.py), and tests/common/goldens/registry_order.json (test_registry_order.py) records the new EP layer, block claim, model_types and Liger spec.
  • GPU rows: in tests/gpu/manifest.py, the --expert-tp-size 2 EP+ETP row of test_ep_family_vs_reference.py (its EP2 row comes from the roster), the gathered round trip (an EPSftRoundTrip subclass in test_sft_ep_roundtrip.py, Qwen4ExpRoundTrip the template) and the packing-isolation row (a _FamilyRow in test_family_packing_isolation.py, --attn sdpa); the merged-resume, precompute-resume and determinism sweeps also generate their rows from the roster. tests/gpu/models/test_qwen4_exp_real_checkpoint.py is the template for a release-checkpoint row.

These carry over with no entry: GatedDeltaNet's segment markers and the CP refusal read linear_attention off layer_types; the attention cost reads the indexed_attention kind off it too (LAYER_TYPE_KINDS, halo/models/attention/layout.py, which raises on a spelling it does not list); reject_stale_eos_copies walks the modules; and transformers drops the mtp.* head on load, which reattach_vision_tower.py streams back. TP refuses a model none of whose attention classes is in TP_SHARDABLE_ATTENTION_CLASSES.

Source on GitHubUpdated 18 November 2026