# Qwen3.8-Flash-Next

`model_type` `qwen4_exp` (`Qwen4ExpForConditionalGeneration`, the release `Qwen/Qwen3.8-Flash-Next`) wrapping the text tower `qwen4_exp_text` (`Qwen4ExpForCausalLM`). `text_only_model: true` loads the text tower and drops the vision tower and the `mtp.*` head; the multimodal class loads too: its forward is tiny-model verified on image and packed text rows, and no test trains it. The release has 48 decoder layers: 36 GatedDeltaNet `linear_attention` layers and 12 sparse-attention (QSA) layers, which the hub spells `full_attention` and transformers maps to `indexed_attention`. Four hyper-connection streams carry the residual; every layer's MoE block routes 512 experts at top-10 beside a sigmoid-gated shared expert; the second layer adds a per-layer embedding (PLE) looked up in a hashed n-gram table of 320,001,536 rows × 160 (95 GiB in bf16, `split_ngram_parts: 128` shards on the hub).

**Experimental.** GPU tests on tiny models cover EP and EP+ETP against the reference model, pure ETP, the gathered-save round trip, deterministic expert gradients, LoRA merged-save resume and DPO/KTO precompute resume. On the release under EP8, `tests/gpu/models/test_qwen4_exp_real_checkpoint.py` has run its load, export and resume modes: the cut to four layers loads lazily and matches the unsharded model, trains, and its gathered save reloads in stock transformers at the trained model's NLL; a full-depth export carries every release tensor unchanged; and on the cut, full fine-tuning resumed from a midway checkpoint replays the unbroken run's losses and final weights bit for bit. At full depth only a few steps of the EP8 LoRA recipe and 13 steps of `pp2 × ep8` full fine-tuning on the text tower have run. Per-axis support: [Compatibility matrix](https://whitecircle.com/halo/docs/agent-docs/models.md#compatibility-matrix). Methods: SFT with full fine-tuning or LoRA (the recipes); DPO and KTO are tiny-model tested only, offline GRPO CPU-checked only. EP runs with lazy loading, and PP runs the text tower ([Pipeline parallelism](#pipeline-parallelism)); TP and CP are refused, and RL weight sync into SGLang takes the toolkit's refreshed image only ([RL weight sync](#rl-weight-sync)). At `ep_size: 1` (plain FSDP2) every GPU loads the whole model, about 246 GiB at full size: tiny and test models only.

## EP layer

`EPQwen4ExpMoELayer` (`halo/distributed/expert_parallel/families/qwen4_exp.py`) subclasses [Qwen3-Next](https://whitecircle.com/halo/docs/agent-docs/models/qwen3-next.md#ep-layer)'s layer: Qwen3.5-MoE's fused `gate_up_proj [E, 2M, H]` / `down_proj [E, H, M]` experts and replicated gated shared expert, under a router that renormalizes the top-k weights only under `norm_topk_prob`. The release stores the experts fused under the block's own names, so the spec keeps the default `FUSED` layout and the gathered save writes `experts.gate_up_proj` / `experts.down_proj` as the hub does. Hidden 2560 is a multiple of 256, so DeepEP pads nothing.

## N-gram table

The table never enters the module state. The family patches (`halo/models/families/qwen4_exp/forwards.py`, applied by `apply_family_patches` on every policy, trainer-path and frozen reference or teacher load) stamp the config the model builds on, and a stamped `Qwen4ExpTextNGramEmbedding` holds a `HostNGramTable` (`halo/models/families/qwen4_exp/ngram_table.py`) in place of its `nn.Embedding`: no parameter, buffer or state-dict entry, so FSDP2, the optimizer, the fp32 masters, PEFT, the coverage gate and the lazy planner never see it, and it trains frozen (the stock class trains it). A build without the stamp keeps the stock class and holds the table as a parameter: a checkpoint tool's `from_pretrained`. The sentence-transformers embedding load, which reads its backbone config itself, refuses a backbone with external tables.

A model built without the patches (from an unstamped config, or before `apply_family_patches` ran) would train the table as a parameter and run transformers' forwards. `apply_run_model_policies` therefore runs `verify_qwen4_exp_patched`, which raises on any n-gram embedding still holding an `nn.Embedding` or on forwards that are not the toolkit's, naming `apply_family_patches` as the remedy; it covers the policy and frozen reference or teacher loads (the embedding load refuses the family above, and reward scoring never runs the policies). A model handed to a trainer already built is not checked there. A checkpoint tool's stock load never reaches it.

`finalize_loaded_model` attaches the table from the local directory the weights were read from: the snapshot of the commit every rank agreed on, which the policy loaders, the trainer-path load and a frozen reference or teacher load that agreed its source name for a Hub source (the lazy EP loader reads it directly). A model loaded outside them whose config records a Hub id is refused, since transformers records no revision to pair the table with the weights. The parts are zero-copy views of the `ngram_embedding.shard_*` tensors, so every rank on a node shares one page-cache copy, and local rank 0 asks the kernel to read the table files ahead. A lookup gathers its rows on the CPU (one thread per rank) and copies them, about 5 KB per token, to the device through pinned memory; the reentrant recompute of the PLE layer reads the first pass's rows back. Part count, row count, width and dtype are checked against the config; a model whose config names no checkpoint, or one without the table, is refused, and so is a random-weight build (`init_from_scratch`, refused before the build), whose config still names the pretrained checkpoint.

The n-gram embedding copies `eos_token_id` when it is built and restarts its history after it. A run's `eos_token` that would record another id is refused (`reject_stale_eos_copies`, `halo/model_loading/tokenizer_setup.py`): training would restart at the copy while every export records the new id.

## Checkpoints and serving

Every gathered save (EP gathered, FSDP2) writes the table as table-only files (`model-table00-*.safetensors`) under the hub's `ngram_embedding.shard_{i}.weight` keys, beside the streamed weights, and a PP save writes it from the stage holding the PLE layer ([Pipeline parallelism](#pipeline-parallelism)); a model with the table always gets an index, and its exported `config.json` keeps `split_ngram_parts` and carries no run stamp ([Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#external-tables)). One process or DDP at `use_grouped_gemm: false`, which no parallel saver claims, streams the same way rather than through transformers' `save_pretrained`, whose state dict holds no table (the family patch refuses a direct call); accelerate-managed FSDP, whose saves stay with `save_pretrained`, is refused at trainer construction unless the run trains adapters. The first save of a process writes the table once from the attached views; each later save hard-links the previous save's files, and a resume attached from a step checkpoint links on its first save. On per-node output storage every node's save rank keeps its own copy of the 95 GiB table. Full fine-tuning resumes from the step checkpoint and attaches the table from it; LoRA resumes onto the base. `save_sharded_ep: true` is refused: `merge_ep_shards.py` has no model to read the table from.

An export loads in stock transformers 5.20 and carries no `mtp.*` head, and `mtp_loss_weight` [refuses the family](https://whitecircle.com/halo/docs/agent-docs/optimization/mtp-heads.md): its draft layer reads hyper-connection streams the trunk's layers carry between them, which an embedding and a hidden state do not supply. vLLM 0.32 registers `Qwen4ExpForCausalLM` and reads the table as its `shard_{i}` keys, and an untrained full-depth export served at TP8 returns the release's prompt log-probs exactly (65,504 compared). SGLang 0.5.22 registers only the multimodal class, so run `reattach_vision_tower.py` first, which also streams the base's untrained MTP head back. Attention and GatedDeltaNet adapters merge with `merge_peft_adapters.py`, which loads the stock class (the table as a parameter, about 360 GB of host RAM); expert adapters fold with `merge_expert_lora_on_save: true`.

## Attention and rows

The class declares no flash or flex support, so the load takes SDPA. transformers 5.20's QSA scores every query in a Python loop and its model forward builds a `[B, 1, T, T]` mask, which leaves rows past a few thousand tokens impractical; `patch_qwen4_exp_document_forwards` (`halo/models/families/qwen4_exp/forwards.py`, applied with the n-gram patch) replaces both for every uncached forward:

- The model forward hands its layers the 2-D padding mask in place of the mask mapping, so no `[T, T]` mask exists.
- A QSA layer selects each query's key blocks per 128-query tile against each document's pooled blocks, and re-scores a query whose k-th and next block tie in transformers' own order, so it attends the keys transformers' indexer gives it, up to a near-tie the two matmul shapes round apart. It attends either on the [block-sparse kernel](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#block-sparse-attention-qwen38-flash-next), which reads only the selected blocks and the tail, or on masked SDPA tiles over K/V from the document's first key to the tile end, which save only Q/K/V and the block ids and recompute each tile in the backward (`halo/models/families/qwen4_exp/sparse_attention.py`). `HALO_QSA_KERNEL` picks (`auto` by default: the kernel where it can run; `off`; `on` or a `ValueError`). The kernel accumulates dK and dV with fp32 atomics, so under `full_determinism` `on` raises `ValueError` and `auto` logs the reason once on rank 0 and takes the tiled path; deterministic runs use `auto` or `off`. Memory is linear in the row; the tiled path's compute is dense causal per document, not a flash kernel.
- The PLE n-gram history restarts at every document start as well as after `eos_token_id`, and its dilated conv runs as masked shifted multiply-adds whose taps stop at the document start, off cuDNN (`halo/models/families/qwen4_exp/documents.py`). An adapter on the PLE `conv1d` is refused.

The family code reads the QSA indexer (heads, head width, budget, compress ratio), the hyper-connection count, the PLE dimensions and conv, the n-gram shape, `split_ngram_parts` and `eos_token_id` off the built modules and the config, never as constants: `tests/cpu/models/test_qwen4_exp_config_driven.py` runs a tiny model with all of them off the defaults against transformers' logits. QSA requires `indexer_kv_heads: 1`, as transformers does, and a table build without `split_ngram_parts` raises.

Documents come from `seq_idx`, which the packing collator emits for this family beside GatedDeltaNet's `cu_seq_lens`, so each document of a packed row computes what it computes alone. A row whose `position_ids` restart at 0 without `seq_idx` (and without a padding mask) is refused. A call with a cache or `use_cache=True`, as in generation, keeps transformers' forwards; one that leaves `use_cache` unset runs per document and returns no cache.

Under gradient checkpointing the recompute reuses the forward's QSA selection and n-gram rows; each QSA layer holds its selection, about 1 KB per token plus 8 B of spans, until its backward (about 400 MB at 32k tokens across the 12 layers). `max_length` must be set; the config's 262,144 exceeds the DeepEP token ceiling. The hyper-connections (`Qwen4ExpTextGatedResidual`) and the injection of a block's output into the streams run on Triton kernels by default (`HALO_GATED_RESIDUAL_KERNELS=0` turns them off) ([Flash-Next gated residual](https://whitecircle.com/halo/docs/agent-docs/optimization/hyper-connection-kernels.md#flash-next-gated-residual)); they are layer-local, so EP, ETP, FSDP2, pipeline stages, LoRA and both checkpointing modes need nothing of them.

## Kernel defaults

`HALO_QSA_KERNEL=auto` and `HALO_GATED_RESIDUAL_KERNELS=1` are the defaults; `HALO_FLA_L2NORM_FIXED_NB` and `HALO_TORCH_NUM_THREADS` keep theirs. Measured on 8 B300 with the real checkpoint under EP8 (seed 42, 14 steps at gradient accumulation 4; medians of steps 3 to 14 per run, then over 3 interleaved runs; the bracket is the spread of the 3 run medians):

| Arm | Recipe | Step time | Tokens/s/GPU | Peak allocated |
| --- | --- | --- | --- | --- |
| off | LoRA, 16k packed | 28.95 s (27.86 to 29.40) | 2,254 | 98.71 GB |
| QSA kernel | LoRA, 16k packed | 18.38 s (18.32 to 19.27) | 3,525 | 98.70 GB |
| gated residual | LoRA, 16k packed | 23.66 s (23.19 to 26.18) | 2,742 | 96.52 GB |
| FLA l2norm fixed NB | LoRA, 16k packed | 27.87 s (27.11 to 28.74) | 2,336 | 98.69 GB |
| all three | LoRA, 16k packed | 15.93 s (14.93 to 16.45) | 4,107 | 96.52 GB |
| 4 CPU threads | LoRA, 16k packed | 24.86 s (24.58 to 25.05) | 2,621 | 98.69 GB |
| off | full, 8k | 13.24 s (12.82 to 15.30) | 2,466 | 155.6 GB |
| all three | full, 8k | 8.54 s (8.47 to 8.89) | 3,813 | 154.1 GB |

- **Loss.** Every LoRA arm follows the same trajectory, 3.33 to 0.85 over the 14 steps, with no trend between arms. The kernel arms stay within 0.021 per step of the off arm of the same run, against 0.011 to 0.016 between two off runs (LoRA, three pairs) and 0.012 to 0.018 (full recipe); the four-thread arm reaches 0.031.
- **Defaults flipped.** The QSA kernel (1.57×) and the gated residual (1.22×) clear a gain above the off arm's 5% spread, with no peak increase, the same loss trajectory and no compile after the warm-up.
- **l2norm stays off.** Its 3.7% gain is below the 5% spread of the off runs, and the knob is global: it also applies to Qwen3.5 and Qwen3-Next.
- **Threads stay at 1.** Four threads give 1.16× here, but `HALO_TORCH_NUM_THREADS` sets the CPU use of every run of every family and was measured on this recipe alone, without the kernels on.
- **Profile.** Torch's mem-efficient attention backward is 22% of the off step's kernel time (10.4 s of 47.3 s over two steps) and about 470k small elementwise launches another 8.5%; with the kernels the total is 26.6 s, the block-sparse forward and backward take 9.2% and 6.1%, and the elementwise launches fall to 13.8k.

## Recipes

`examples/sft/qwen4_exp/qwen3.8-flash-next-ep8-lora.yaml` (16k packed rows) and `-ep8-full.yaml` (8k, B300) train the text tower at ep8 on one node with the fused loss. LoRA sets `moe_balancing: none` (the router is frozen), full fine-tuning `bias_update_transient`. Both pin Halo's `qwen3-multiturn.jinja` ([its first-step loss on Qwen3.6](https://whitecircle.com/halo/docs/agent-docs/models/qwen3_5.md#configs)): the hub template is Qwen3.8-27B's, with its [reasoning-effort caveats](https://whitecircle.com/halo/docs/agent-docs/models/qwen3_5.md#qwen38-27b). Both leave `eos_token` unset: the PLE n-gram history restarts after the checkpoint's `eos_token_id` (`<|endoftext|>`), and a run's `eos_token` that changes it is refused ([N-gram table](#n-gram-table)).

## LoRA

PEFT adapts attention (`q_proj`, `k_proj`, `v_proj`, `o_proj`) and GatedDeltaNet (`in_proj_qkv`, `in_proj_z`, `in_proj_a`, `in_proj_b`, `out_proj`); routed experts take Halo's native expert LoRA under EP ([PEFT](https://whitecircle.com/halo/docs/agent-docs/optimization/peft.md)). An adapter on `indexer.index_qk_proj` never gets a gradient: the indexer's top-k is a boolean selection, in transformers as here. List targets explicitly, as the LoRA recipe does. The merged-save GPU rows adapt the attention projections and experts; the GatedDeltaNet projections ran through the EP8 recipe.

## Balancing

The router has no bias slot. On the family's own head `moe_balancing: auto` resolves to `aux_loss`. The Liger spec declares the head's in-head aux loss, so under the fused loss `output_router_logits` is refused and `auto` resolves to `none` with a warning; pick `bias_update_transient` (trainer-only, no export carries the bias) or `none` explicitly. `bias_update` raises ([Callbacks](https://whitecircle.com/halo/docs/agent-docs/training-methods/callbacks.md#moe-balancing-modes)).

## Pipeline parallelism

`Qwen4ExpTextModel` is registered for PP, tiny-model GPU tested at pp2 x ep2 and pp2 x etp2 with a PLE layer on each stage. On the release, `pp2 × ep8` across two 8-GPU machines ran 13 steps with GC on at 16.89 s a step, 1,940 tok/s/GPU as logged (padding counted), at 115.7 / 78.2 GiB; with GC off the first step runs out of memory at about 260 GiB. Its stages carry the four-stream residual across each boundary at `hc_count` times the hidden width (20 KiB per token), and `hyper_connection_mixer` collapses it on the last stage alone. Every per-layer input hangs off the layer itself, so any split offset is exact.

- **Token ids.** A stage holding a PLE layer looks its n-grams up from the token ids, which the runner hands every later stage beside the activation ([Token ids on later stages](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#token-ids-on-later-stages)). The release's one PLE layer (layer 1) sits on stage 0 whenever stage 0 holds at least two layers.
- **Packed rows.** Each stage rebuilds the row's `seq_idx` and varlen set from its microbatch's positions, so the QSA indexer, the PLE history and the GatedDeltaNet layers keep each document apart ([PP packing](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#packing)).
- **Checkpoints.** The stage holding a PLE layer writes its table beside its parts, as `model-ppSSSSS-of-PPPPP-table00-*` under the unsplit model's keys; a resume attaches each stage's table from those files, and `merge_pp_stages.py` carries them like any part.
- **Limits.** PP refuses adapters, so it trains full fine-tunes; with `aux_loss` severed by the split and no bias slot, balance with `moe_balancing: none`. Checkpointing runs non-reentrant, the QSA selection and n-gram rows replayed from each checkpoint frame. Load with `text_only_model: true`, or run the composite text-only, which re-emits its vision tower in every checkpoint; image data is refused.

## RL weight sync

The weight sync into vLLM 0.32.0 and into SGLang 0.5.22 is exact on tiny composites at two ranks (the push-exactness and LoRA sync rows); SGLang only from the toolkit's image. No GRPO run on the real checkpoint exists yet, and the family stays experimental. Both need the composite: `text_only_model` is refused, since the sync sends `model.layers.*` names where the served class expects `model.language_model.layers.*`.

- **vLLM.** `tests/gpu/trainers/grpo/test_weight_push_exact.py --family qwen4_exp_text --backend vllm --reach` passes on a tiny composite checkpoint the trainer's own gathered save writes (`--write-checkpoint`): an unchanged push leaves every served prompt log-prob bit-identical, a scaled attention weight moves them and the restoring push returns them exactly, every forwarded name but the indexer's reaches the served model (scaling an indexer weight leaves the key ranking where it was), and after a real optimizer step the pushed server sits as close to the trainer as the two were before it. vLLM's model reads each weight live and builds no copy of one at load or first use. The row runs at `ep_size: 2`; the server needs `--moe-backend triton` like every family, and the checkpoint's `indexer_budget` over `indexer_compress_ratio` must be 512, 1024 or 2048 (the release's is 512), which the server's top-k kernel enforces. A composite export served by vLLM needs `deepstack_visual_indexes: []` in its `vision_config`, which transformers' vision config drops on save and the release's `config.json` carries.
- **SGLang.** On SM100-family GPUs (B200, B300) each hyper-connection `GatedResidual` builds a padded, permuted copy of its `input_mix_weight_up` the first time it mixes a batch of at most 24 tokens, and stock SGLang never rebuilds it, so an update does not reach it. The toolkit's image rewrites the copy in place and reports `hyperconnection_mix_up` on `/server_info` ([SGLang loader facts](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#which-families-each-engine-serves)); the startup preflight refuses a server that does not. SGLang registers only the multimodal class, so a text-only export goes through `reattach_vision_tower.py` first. The same test passes with `--backend sglang --reach` on that image.

Offline GRPO trains on pre-collected completions with no server, so no refusal reaches it ([Offline GRPO](https://whitecircle.com/halo/docs/agent-docs/training-methods/grpo/offline.md)). The n-gram table is not a gap: it trains frozen, so the sync has nothing to send for it, and each server loads its own copy from the checkpoint at launch.

`examples/grpo/environmental/qwen4_exp/vllm/qwen3.8-flash-next-exam-qa-full-ep8.yaml` is the environmental recipe: full fine-tuning at EP=8 on one node and a vLLM server (`qwen3_xml` tool parser, `qwen3` reasoning parser, the checkpoint's own template) on another. It is config-checked only; no GPU run has trained it.

## Refused modes

- **CP**: 36 of 48 layers are `linear_attention`, whose conv and recurrent scan run along the sequence axis.
- **TP**: `Qwen4ExpTextAttention` is not in `TP_SHARDABLE_ATTENTION_CLASSES`, so a TP run would shard no layer.
- **`padding_free`, tree packing**: SDPA reads no `cu_seqlens`, and tree packing admits only the families it is verified on.
- **Packed rows without `seq_idx`**: the model forward refuses them ([above](#attention-and-rows)).
- **`init_from_scratch`** and an **`eos_token`** that changes `eos_token_id` ([N-gram table](#n-gram-table)), a direct **`save_pretrained`** and a full fine-tune under **accelerate-managed FSDP** ([Checkpoints and serving](#checkpoints-and-serving)).
- **RL weight sync into an SGLang server without the mix-up patch**: the startup preflight ([above](#rl-weight-sync)). Offline GRPO needs no server.

## Qwen4-lineage checklist

transformers' modular converter writes every class standalone: `modular_qwen4_exp.py` derives the attention from Qwen3.5's and the MoE block from Qwen3-Next's, but the generated `Qwen4ExpTextAttention` and `Qwen4ExpTextSparseMoeBlock` subclass `nn.Module`. A registry keyed on a class name carries nothing over by MRO, so a family built from qwen4_exp's blocks (GatedDeltaNet, QSA, hyper-connections, the PLE n-gram table, fused experts, MTP) needs its own entry in each:

- **EP**: a `MoEFamilySpec` in `MOE_FAMILY_SPECS` (`halo/models/moe/family_specs.py`) claiming every `model_type` spelling, and an EP layer under `halo/distributed/expert_parallel/families/` that names its block class in `HF_MODULE_NAMES`, imported in `roster.py`; it subclasses `EPQwen4ExpMoELayer` where the block matches ([Add EP support](https://whitecircle.com/halo/docs/agent-docs/models/adding-a-model.md#add-ep-support)).
- **Family patches**: `apply_family_patches` (`halo/model_loading/model_preparation.py`) applies qwen4_exp's on a `model_type` starting with `qwen4_exp`, so a family whose `model_type` does not needs its own branch. The patches (`halo/models/families/qwen4_exp/`) replace methods by name on classes of `transformers.models.qwen4_exp.modeling_qwen4_exp`: the n-gram embedding's host table, the `save_pretrained` refusal, and the per-document model, decoder-layer and QSA forwards. `verify_upstream_parameters` (`halo/models/upstream/upstream_guard.py`) refuses a transformers release whose patched signatures moved.
- **Liger**: a `LigerFamilySpec` in `halo/model_loading/liger/family_specs.py`. qwen4_exp's declares the head only (`causal_lm`, `router_aux_loss_in_head`): its RMSNorm class also runs the hyper-connection norms grouped per stream, and its rotary is interleaved mrope ([Declare Liger coverage](https://whitecircle.com/halo/docs/agent-docs/models/adding-a-model.md#declare-liger-coverage)).
- **PP**: a `PPFamilySpec` in `PP_FAMILY_SPECS` (`halo/distributed/pipeline_parallel/families.py`) claiming the backbone class, `QWEN4_EXP_PP` the template: `final_norm_attrs` names the stream collapse `hyper_connection_mixer`, `stream_forward` runs the stages that receive the widened `[B, S, hc_count * D]` stream, `token_ids_kwarg` hands the token ids to the PLE layers on later stages, and `layer_types_rebase_safe` holds because every per-layer input hangs off the layer itself. The per-stage segment markers and the PP save and attach of external tables take no entry. `validated=True` follows the family's PP GPU rows, which `tests/cpu/conventions/test_pp_family_roster.py` requires (`test_pp_qwen4_exp_vs_reference.py` the template).
- **Weight sync**: qwen4_exp's spec carries no `weight_sync_refusal`; SGLang's client names the image patch it needs in `REQUIRED_PATCHES` ([RL weight sync](#rl-weight-sync)). A new spec starts without a refusal, so a family built from these blocks declares its own, or a per-engine loader fact in a client's `UNSERVABLE_MODEL_TYPES` (`halo/weight_sync/clients/`), until an engine run validates its sync. A family vLLM's sync admits needs a [config-parity fixture](https://whitecircle.com/halo/docs/agent-docs/infrastructure/rollout-servers.md#config-schema-parity) under `docker/vllm/parity/fixtures/`, which the vLLM image build checks (`tests/cpu/checkpoint/test_vllm_parity_fixtures.py` fails without it); a family either engine admits needs its sync-exactness rows, `_SYNC_EXACTNESS_SWEEP_FAMILIES` in `tests/gpu/manifest.py` (`test_lora_weight_sync_exact.py`), which `tests/cpu/conventions/test_tiny_family_roster.py` holds to the syncable families, and its `test_weight_push_exact.py` rows.
- **External tables**: a frozen table is an `ExternalTable` subclass (`halo/models/external_tables.py`; qwen4_exp's `HostNGramTable`), which the family's patched constructor builds when the config carries `EXTERNAL_TABLES_STAMP`. The loaders attach, and the gathered savers write, every table the module walk finds, so no saver takes a per-family entry; `save_sharded_ep` and a full fine-tune under accelerate-managed FSDP refuse a model holding one ([Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#external-tables)).
- **Tiny test configs**: the text tower in `TINY_MOE_FAMILIES` and the composite in `TINY_MOE_VLM_FAMILIES` (`tests/common/tiny_models.py`), with `ulysses_cp=False`; random init zeroes the PLE conv, which `_with_live_ple_convs` redraws on `Qwen4ExpTextPLELayer` by class. The text tower goes in `_TINY_MOE_FAMILY_MARKERS` (`tests/gpu/manifest.py`), its spellings in `tests/cpu/parallelism/test_hf_model_types_completeness.py`. The CPU EP-vs-reference comparison takes a fixture in `FAMILIES` (`tests/cpu/parallelism/test_ep_vs_reference_equivalence_cpu.py`), and `tests/common/goldens/registry_order.json` (`test_registry_order.py`) records the new EP layer, block claim, `model_type`s and Liger spec.
- **GPU rows**: in `tests/gpu/manifest.py`, the `--expert-tp-size 2` EP+ETP row of `test_ep_family_vs_reference.py` (its EP2 row comes from the roster), the gathered round trip (an `EPSftRoundTrip` subclass in `test_sft_ep_roundtrip.py`, `Qwen4ExpRoundTrip` the template) and the packing-isolation row (a `_FamilyRow` in `test_family_packing_isolation.py`, `--attn sdpa`); the merged-resume, precompute-resume and determinism sweeps also generate their rows from the roster. `tests/gpu/models/test_qwen4_exp_real_checkpoint.py` is the template for a release-checkpoint row.

These carry over with no entry: GatedDeltaNet's segment markers and the CP refusal read `linear_attention` off `layer_types`; the attention cost reads the `indexed_attention` kind off it too (`LAYER_TYPE_KINDS`, `halo/models/attention/layout.py`, which raises on a spelling it does not list); `reject_stale_eos_copies` walks the modules; and transformers drops the `mtp.*` head on load, which `reattach_vision_tower.py` streams back. TP refuses a model none of whose attention classes is in `TP_SHARDABLE_ATTENTION_CLASSES`.
