Qwen3.8-Flash-Next
model_type qwen4_exp (Qwen4ExpForConditionalGeneration, the release Qwen/Qwen3.8-Flash-Next) wrapping the text tower qwen4_exp_text (Qwen4ExpForCausalLM). text_only_model: true loads the text tower and drops the vision tower and the mtp.* head; the multimodal class loads too: its forward is tiny-model verified on image and packed text rows, and no test trains it. The release has 48 decoder layers: 36 GatedDeltaNet linear_attention layers and 12 sparse-attention (QSA) layers, which the hub spells full_attention and transformers maps to indexed_attention. Four hyper-connection streams carry the residual; every layer's MoE block routes 512 experts at top-10 beside a sigmoid-gated shared expert; the second layer adds a per-layer embedding (PLE) looked up in a hashed n-gram table of 320,001,536 rows × 160 (95 GiB in bf16, split_ngram_parts: 128 shards on the hub).
Experimental. GPU tests on tiny models cover EP and EP+ETP against the reference model, pure ETP, the gathered-save round trip, deterministic expert gradients, LoRA merged-save resume and DPO/KTO precompute resume. On the release under EP8, tests/gpu/models/test_qwen4_exp_real_checkpoint.py has run its load, export and resume modes: the cut to four layers loads lazily and matches the unsharded model, trains, and its gathered save reloads in stock transformers at the trained model's NLL; a full-depth export carries every release tensor unchanged; and on the cut, full fine-tuning resumed from a midway checkpoint replays the unbroken run's losses and final weights bit for bit. At full depth only a few steps of the EP8 LoRA recipe and 13 steps of pp2 × ep8 full fine-tuning on the text tower have run. Per-axis support: Compatibility matrix. Methods: SFT with full fine-tuning or LoRA (the recipes); DPO and KTO are tiny-model tested only, offline GRPO CPU-checked only. EP runs with lazy loading, and PP runs the text tower (Pipeline parallelism); TP and CP are refused, and RL weight sync into SGLang takes the toolkit's refreshed image only (RL weight sync). At ep_size: 1 (plain FSDP2) every GPU loads the whole model, about 246 GiB at full size: tiny and test models only.
EP layer
EPQwen4ExpMoELayer (halo/distributed/expert_parallel/families/qwen4_exp.py) subclasses Qwen3-Next's layer: Qwen3.5-MoE's fused gate_up_proj [E, 2M, H] / down_proj [E, H, M] experts and replicated gated shared expert, under a router that renormalizes the top-k weights only under norm_topk_prob. The release stores the experts fused under the block's own names, so the spec keeps the default FUSED layout and the gathered save writes experts.gate_up_proj / experts.down_proj as the hub does. Hidden 2560 is a multiple of 256, so DeepEP pads nothing.
N-gram table
The table never enters the module state. The family patches (halo/models/families/qwen4_exp/forwards.py, applied by apply_family_patches on every policy, trainer-path and frozen reference or teacher load) stamp the config the model builds on, and a stamped Qwen4ExpTextNGramEmbedding holds a HostNGramTable (halo/models/families/qwen4_exp/ngram_table.py) in place of its nn.Embedding: no parameter, buffer or state-dict entry, so FSDP2, the optimizer, the fp32 masters, PEFT, the coverage gate and the lazy planner never see it, and it trains frozen (the stock class trains it). A build without the stamp keeps the stock class and holds the table as a parameter: a checkpoint tool's from_pretrained. The sentence-transformers embedding load, which reads its backbone config itself, refuses a backbone with external tables.
A model built without the patches (from an unstamped config, or before apply_family_patches ran) would train the table as a parameter and run transformers' forwards. apply_run_model_policies therefore runs verify_qwen4_exp_patched, which raises on any n-gram embedding still holding an nn.Embedding or on forwards that are not the toolkit's, naming apply_family_patches as the remedy; it covers the policy and frozen reference or teacher loads (the embedding load refuses the family above, and reward scoring never runs the policies). A model handed to a trainer already built is not checked there. A checkpoint tool's stock load never reaches it.
finalize_loaded_model attaches the table from the local directory the weights were read from: the snapshot of the commit every rank agreed on, which the policy loaders, the trainer-path load and a frozen reference or teacher load that agreed its source name for a Hub source (the lazy EP loader reads it directly). A model loaded outside them whose config records a Hub id is refused, since transformers records no revision to pair the table with the weights. The parts are zero-copy views of the ngram_embedding.shard_* tensors, so every rank on a node shares one page-cache copy, and local rank 0 asks the kernel to read the table files ahead. A lookup gathers its rows on the CPU (one thread per rank) and copies them, about 5 KB per token, to the device through pinned memory; the reentrant recompute of the PLE layer reads the first pass's rows back. Part count, row count, width and dtype are checked against the config; a model whose config names no checkpoint, or one without the table, is refused, and so is a random-weight build (init_from_scratch, refused before the build), whose config still names the pretrained checkpoint.
The n-gram embedding copies eos_token_id when it is built and restarts its history after it. A run's eos_token that would record another id is refused (reject_stale_eos_copies, halo/model_loading/tokenizer_setup.py): training would restart at the copy while every export records the new id.
Checkpoints and serving
Every gathered save (EP gathered, FSDP2) writes the table as table-only files (model-table00-*.safetensors) under the hub's ngram_embedding.shard_{i}.weight keys, beside the streamed weights, and a PP save writes it from the stage holding the PLE layer (Pipeline parallelism); a model with the table always gets an index, and its exported config.json keeps split_ngram_parts and carries no run stamp (Checkpoints). One process or DDP at use_grouped_gemm: false, which no parallel saver claims, streams the same way rather than through transformers' save_pretrained, whose state dict holds no table (the family patch refuses a direct call); accelerate-managed FSDP, whose saves stay with save_pretrained, is refused at trainer construction unless the run trains adapters. The first save of a process writes the table once from the attached views; each later save hard-links the previous save's files, and a resume attached from a step checkpoint links on its first save. On per-node output storage every node's save rank keeps its own copy of the 95 GiB table. Full fine-tuning resumes from the step checkpoint and attaches the table from it; LoRA resumes onto the base. save_sharded_ep: true is refused: merge_ep_shards.py has no model to read the table from.
An export loads in stock transformers 5.20 and carries no mtp.* head, and mtp_loss_weight refuses the family: its draft layer reads hyper-connection streams the trunk's layers carry between them, which an embedding and a hidden state do not supply. vLLM 0.32 registers Qwen4ExpForCausalLM and reads the table as its shard_{i} keys, and an untrained full-depth export served at TP8 returns the release's prompt log-probs exactly (65,504 compared). SGLang 0.5.22 registers only the multimodal class, so run reattach_vision_tower.py first, which also streams the base's untrained MTP head back. Attention and GatedDeltaNet adapters merge with merge_peft_adapters.py, which loads the stock class (the table as a parameter, about 360 GB of host RAM); expert adapters fold with merge_expert_lora_on_save: true.
Attention and rows
The class declares no flash or flex support, so the load takes SDPA. transformers 5.20's QSA scores every query in a Python loop and its model forward builds a [B, 1, T, T] mask, which leaves rows past a few thousand tokens impractical; patch_qwen4_exp_document_forwards (halo/models/families/qwen4_exp/forwards.py, applied with the n-gram patch) replaces both for every uncached forward:
- The model forward hands its layers the 2-D padding mask in place of the mask mapping, so no
[T, T]mask exists. - A QSA layer selects each query's key blocks per 128-query tile against each document's pooled blocks, and re-scores a query whose k-th and next block tie in transformers' own order, so it attends the keys transformers' indexer gives it, up to a near-tie the two matmul shapes round apart. It attends either on the block-sparse kernel, which reads only the selected blocks and the tail, or on masked SDPA tiles over K/V from the document's first key to the tile end, which save only Q/K/V and the block ids and recompute each tile in the backward (
halo/models/families/qwen4_exp/sparse_attention.py).HALO_QSA_KERNELpicks (autoby default: the kernel where it can run;off;onor aValueError). The kernel accumulates dK and dV with fp32 atomics, so underfull_determinismonraisesValueErrorandautologs the reason once on rank 0 and takes the tiled path; deterministic runs useautooroff. Memory is linear in the row; the tiled path's compute is dense causal per document, not a flash kernel. - The PLE n-gram history restarts at every document start as well as after
eos_token_id, and its dilated conv runs as masked shifted multiply-adds whose taps stop at the document start, off cuDNN (halo/models/families/qwen4_exp/documents.py). An adapter on the PLEconv1dis refused.
The family code reads the QSA indexer (heads, head width, budget, compress ratio), the hyper-connection count, the PLE dimensions and conv, the n-gram shape, split_ngram_parts and eos_token_id off the built modules and the config, never as constants: tests/cpu/models/test_qwen4_exp_config_driven.py runs a tiny model with all of them off the defaults against transformers' logits. QSA requires indexer_kv_heads: 1, as transformers does, and a table build without split_ngram_parts raises.
Documents come from seq_idx, which the packing collator emits for this family beside GatedDeltaNet's cu_seq_lens, so each document of a packed row computes what it computes alone. A row whose position_ids restart at 0 without seq_idx (and without a padding mask) is refused. A call with a cache or use_cache=True, as in generation, keeps transformers' forwards; one that leaves use_cache unset runs per document and returns no cache.
Under gradient checkpointing the recompute reuses the forward's QSA selection and n-gram rows; each QSA layer holds its selection, about 1 KB per token plus 8 B of spans, until its backward (about 400 MB at 32k tokens across the 12 layers). max_length must be set; the config's 262,144 exceeds the DeepEP token ceiling. The hyper-connections (Qwen4ExpTextGatedResidual) and the injection of a block's output into the streams run on Triton kernels by default (HALO_GATED_RESIDUAL_KERNELS=0 turns them off) (Flash-Next gated residual); they are layer-local, so EP, ETP, FSDP2, pipeline stages, LoRA and both checkpointing modes need nothing of them.
Kernel defaults
HALO_QSA_KERNEL=auto and HALO_GATED_RESIDUAL_KERNELS=1 are the defaults; HALO_FLA_L2NORM_FIXED_NB and HALO_TORCH_NUM_THREADS keep theirs. Measured on 8 B300 with the real checkpoint under EP8 (seed 42, 14 steps at gradient accumulation 4; medians of steps 3 to 14 per run, then over 3 interleaved runs; the bracket is the spread of the 3 run medians):
- Loss. Every LoRA arm follows the same trajectory, 3.33 to 0.85 over the 14 steps, with no trend between arms. The kernel arms stay within 0.021 per step of the off arm of the same run, against 0.011 to 0.016 between two off runs (LoRA, three pairs) and 0.012 to 0.018 (full recipe); the four-thread arm reaches 0.031.
- Defaults flipped. The QSA kernel (1.57×) and the gated residual (1.22×) clear a gain above the off arm's 5% spread, with no peak increase, the same loss trajectory and no compile after the warm-up.
- l2norm stays off. Its 3.7% gain is below the 5% spread of the off runs, and the knob is global: it also applies to Qwen3.5 and Qwen3-Next.
- Threads stay at 1. Four threads give 1.16× here, but
HALO_TORCH_NUM_THREADSsets the CPU use of every run of every family and was measured on this recipe alone, without the kernels on. - Profile. Torch's mem-efficient attention backward is 22% of the off step's kernel time (10.4 s of 47.3 s over two steps) and about 470k small elementwise launches another 8.5%; with the kernels the total is 26.6 s, the block-sparse forward and backward take 9.2% and 6.1%, and the elementwise launches fall to 13.8k.
Recipes
examples/sft/qwen4_exp/qwen3.8-flash-next-ep8-lora.yaml (16k packed rows) and -ep8-full.yaml (8k, B300) train the text tower at ep8 on one node with the fused loss. LoRA sets moe_balancing: none (the router is frozen), full fine-tuning bias_update_transient. Both pin Halo's qwen3-multiturn.jinja (its first-step loss on Qwen3.6): the hub template is Qwen3.8-27B's, with its reasoning-effort caveats. Both leave eos_token unset: the PLE n-gram history restarts after the checkpoint's eos_token_id (<|endoftext|>), and a run's eos_token that changes it is refused (N-gram table).
LoRA
PEFT adapts attention (q_proj, k_proj, v_proj, o_proj) and GatedDeltaNet (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj); routed experts take Halo's native expert LoRA under EP (PEFT). An adapter on indexer.index_qk_proj never gets a gradient: the indexer's top-k is a boolean selection, in transformers as here. List targets explicitly, as the LoRA recipe does. The merged-save GPU rows adapt the attention projections and experts; the GatedDeltaNet projections ran through the EP8 recipe.
Balancing
The router has no bias slot. On the family's own head moe_balancing: auto resolves to aux_loss. The Liger spec declares the head's in-head aux loss, so under the fused loss output_router_logits is refused and auto resolves to none with a warning; pick bias_update_transient (trainer-only, no export carries the bias) or none explicitly. bias_update raises (Callbacks).
Pipeline parallelism
Qwen4ExpTextModel is registered for PP, tiny-model GPU tested at pp2 x ep2 and pp2 x etp2 with a PLE layer on each stage. On the release, pp2 × ep8 across two 8-GPU machines ran 13 steps with GC on at 16.89 s a step, 1,940 tok/s/GPU as logged (padding counted), at 115.7 / 78.2 GiB; with GC off the first step runs out of memory at about 260 GiB. Its stages carry the four-stream residual across each boundary at hc_count times the hidden width (20 KiB per token), and hyper_connection_mixer collapses it on the last stage alone. Every per-layer input hangs off the layer itself, so any split offset is exact.
- Token ids. A stage holding a PLE layer looks its n-grams up from the token ids, which the runner hands every later stage beside the activation (Token ids on later stages). The release's one PLE layer (layer 1) sits on stage 0 whenever stage 0 holds at least two layers.
- Packed rows. Each stage rebuilds the row's
seq_idxand varlen set from its microbatch's positions, so the QSA indexer, the PLE history and the GatedDeltaNet layers keep each document apart (PP packing). - Checkpoints. The stage holding a PLE layer writes its table beside its parts, as
model-ppSSSSS-of-PPPPP-table00-*under the unsplit model's keys; a resume attaches each stage's table from those files, andmerge_pp_stages.pycarries them like any part. - Limits. PP refuses adapters, so it trains full fine-tunes; with
aux_losssevered by the split and no bias slot, balance withmoe_balancing: none. Checkpointing runs non-reentrant, the QSA selection and n-gram rows replayed from each checkpoint frame. Load withtext_only_model: true, or run the composite text-only, which re-emits its vision tower in every checkpoint; image data is refused.
RL weight sync
The weight sync into vLLM 0.32.0 and into SGLang 0.5.22 is exact on tiny composites at two ranks (the push-exactness and LoRA sync rows); SGLang only from the toolkit's image. No GRPO run on the real checkpoint exists yet, and the family stays experimental. Both need the composite: text_only_model is refused, since the sync sends model.layers.* names where the served class expects model.language_model.layers.*.
- vLLM.
tests/gpu/trainers/grpo/test_weight_push_exact.py --family qwen4_exp_text --backend vllm --reachpasses on a tiny composite checkpoint the trainer's own gathered save writes (--write-checkpoint): an unchanged push leaves every served prompt log-prob bit-identical, a scaled attention weight moves them and the restoring push returns them exactly, every forwarded name but the indexer's reaches the served model (scaling an indexer weight leaves the key ranking where it was), and after a real optimizer step the pushed server sits as close to the trainer as the two were before it. vLLM's model reads each weight live and builds no copy of one at load or first use. The row runs atep_size: 2; the server needs--moe-backend tritonlike every family, and the checkpoint'sindexer_budgetoverindexer_compress_ratiomust be 512, 1024 or 2048 (the release's is 512), which the server's top-k kernel enforces. A composite export served by vLLM needsdeepstack_visual_indexes: []in itsvision_config, which transformers' vision config drops on save and the release'sconfig.jsoncarries. - SGLang. On SM100-family GPUs (B200, B300) each hyper-connection
GatedResidualbuilds a padded, permuted copy of itsinput_mix_weight_upthe first time it mixes a batch of at most 24 tokens, and stock SGLang never rebuilds it, so an update does not reach it. The toolkit's image rewrites the copy in place and reportshyperconnection_mix_upon/server_info(SGLang loader facts); the startup preflight refuses a server that does not. SGLang registers only the multimodal class, so a text-only export goes throughreattach_vision_tower.pyfirst. The same test passes with--backend sglang --reachon that image.
Offline GRPO trains on pre-collected completions with no server, so no refusal reaches it (Offline GRPO). The n-gram table is not a gap: it trains frozen, so the sync has nothing to send for it, and each server loads its own copy from the checkpoint at launch.
examples/grpo/environmental/qwen4_exp/vllm/qwen3.8-flash-next-exam-qa-full-ep8.yaml is the environmental recipe: full fine-tuning at EP=8 on one node and a vLLM server (qwen3_xml tool parser, qwen3 reasoning parser, the checkpoint's own template) on another. It is config-checked only; no GPU run has trained it.
Refused modes
- CP: 36 of 48 layers are
linear_attention, whose conv and recurrent scan run along the sequence axis. - TP:
Qwen4ExpTextAttentionis not inTP_SHARDABLE_ATTENTION_CLASSES, so a TP run would shard no layer. padding_free, tree packing: SDPA reads nocu_seqlens, and tree packing admits only the families it is verified on.- Packed rows without
seq_idx: the model forward refuses them (above). init_from_scratchand aneos_tokenthat changeseos_token_id(N-gram table), a directsave_pretrainedand a full fine-tune under accelerate-managed FSDP (Checkpoints and serving).- RL weight sync into an SGLang server without the mix-up patch: the startup preflight (above). Offline GRPO needs no server.
Qwen4-lineage checklist
transformers' modular converter writes every class standalone: modular_qwen4_exp.py derives the attention from Qwen3.5's and the MoE block from Qwen3-Next's, but the generated Qwen4ExpTextAttention and Qwen4ExpTextSparseMoeBlock subclass nn.Module. A registry keyed on a class name carries nothing over by MRO, so a family built from qwen4_exp's blocks (GatedDeltaNet, QSA, hyper-connections, the PLE n-gram table, fused experts, MTP) needs its own entry in each:
- EP: a
MoEFamilySpecinMOE_FAMILY_SPECS(halo/models/moe/family_specs.py) claiming everymodel_typespelling, and an EP layer underhalo/distributed/expert_parallel/families/that names its block class inHF_MODULE_NAMES, imported inroster.py; it subclassesEPQwen4ExpMoELayerwhere the block matches (Add EP support). - Family patches:
apply_family_patches(halo/model_loading/model_preparation.py) applies qwen4_exp's on amodel_typestarting withqwen4_exp, so a family whosemodel_typedoes not needs its own branch. The patches (halo/models/families/qwen4_exp/) replace methods by name on classes oftransformers.models.qwen4_exp.modeling_qwen4_exp: the n-gram embedding's host table, thesave_pretrainedrefusal, and the per-document model, decoder-layer and QSA forwards.verify_upstream_parameters(halo/models/upstream/upstream_guard.py) refuses a transformers release whose patched signatures moved. - Liger: a
LigerFamilySpecinhalo/model_loading/liger/family_specs.py. qwen4_exp's declares the head only (causal_lm,router_aux_loss_in_head): its RMSNorm class also runs the hyper-connection norms grouped per stream, and its rotary is interleaved mrope (Declare Liger coverage). - PP: a
PPFamilySpecinPP_FAMILY_SPECS(halo/distributed/pipeline_parallel/families.py) claiming the backbone class,QWEN4_EXP_PPthe template:final_norm_attrsnames the stream collapsehyper_connection_mixer,stream_forwardruns the stages that receive the widened[B, S, hc_count * D]stream,token_ids_kwarghands the token ids to the PLE layers on later stages, andlayer_types_rebase_safeholds because every per-layer input hangs off the layer itself. The per-stage segment markers and the PP save and attach of external tables take no entry.validated=Truefollows the family's PP GPU rows, whichtests/cpu/conventions/test_pp_family_roster.pyrequires (test_pp_qwen4_exp_vs_reference.pythe template). - Weight sync: qwen4_exp's spec carries no
weight_sync_refusal; SGLang's client names the image patch it needs inREQUIRED_PATCHES(RL weight sync). A new spec starts without a refusal, so a family built from these blocks declares its own, or a per-engine loader fact in a client'sUNSERVABLE_MODEL_TYPES(halo/weight_sync/clients/), until an engine run validates its sync. A family vLLM's sync admits needs a config-parity fixture underdocker/vllm/parity/fixtures/, which the vLLM image build checks (tests/cpu/checkpoint/test_vllm_parity_fixtures.pyfails without it); a family either engine admits needs its sync-exactness rows,_SYNC_EXACTNESS_SWEEP_FAMILIESintests/gpu/manifest.py(test_lora_weight_sync_exact.py), whichtests/cpu/conventions/test_tiny_family_roster.pyholds to the syncable families, and itstest_weight_push_exact.pyrows. - External tables: a frozen table is an
ExternalTablesubclass (halo/models/external_tables.py; qwen4_exp'sHostNGramTable), which the family's patched constructor builds when the config carriesEXTERNAL_TABLES_STAMP. The loaders attach, and the gathered savers write, every table the module walk finds, so no saver takes a per-family entry;save_sharded_epand a full fine-tune under accelerate-managed FSDP refuse a model holding one (Checkpoints). - Tiny test configs: the text tower in
TINY_MOE_FAMILIESand the composite inTINY_MOE_VLM_FAMILIES(tests/common/tiny_models.py), withulysses_cp=False; random init zeroes the PLE conv, which_with_live_ple_convsredraws onQwen4ExpTextPLELayerby class. The text tower goes in_TINY_MOE_FAMILY_MARKERS(tests/gpu/manifest.py), its spellings intests/cpu/parallelism/test_hf_model_types_completeness.py. The CPU EP-vs-reference comparison takes a fixture inFAMILIES(tests/cpu/parallelism/test_ep_vs_reference_equivalence_cpu.py), andtests/common/goldens/registry_order.json(test_registry_order.py) records the new EP layer, block claim,model_types and Liger spec. - GPU rows: in
tests/gpu/manifest.py, the--expert-tp-size 2EP+ETP row oftest_ep_family_vs_reference.py(its EP2 row comes from the roster), the gathered round trip (anEPSftRoundTripsubclass intest_sft_ep_roundtrip.py,Qwen4ExpRoundTripthe template) and the packing-isolation row (a_FamilyRowintest_family_packing_isolation.py,--attn sdpa); the merged-resume, precompute-resume and determinism sweeps also generate their rows from the roster.tests/gpu/models/test_qwen4_exp_real_checkpoint.pyis the template for a release-checkpoint row.
These carry over with no entry: GatedDeltaNet's segment markers and the CP refusal read linear_attention off layer_types; the attention cost reads the indexed_attention kind off it too (LAYER_TYPE_KINDS, halo/models/attention/layout.py, which raises on a spelling it does not list); reject_stale_eos_copies walks the modules; and transformers drops the mtp.* head on load, which reattach_vision_tower.py streams back. TP refuses a model none of whose attention classes is in TP_SHARDABLE_ATTENTION_CLASSES.
Source on GitHubUpdated 18 November 2026