Inkling
Thinking Machines' InklingForConditionalGeneration (thinkingmachines/Inkling-Small, 266B total / 12B active) — a multimodal MoE routing to 256 experts with top-6 selection, 2 always-on shared experts, and 40 MoE layers (of 42 — dense_mlp_idx: 2 keeps the first two dense) at hidden 4096.
Attention is hybrid sliding (5 sliding : 1 full, period 6) with no RoPE — position enters as an additive relative-logits bias — plus depthwise causal short convolutions (InklingShortConvolution) on the projected K/V and around each decoder sublayer.
Transformers ships transformers.models.inkling natively (the image pins 5.20.2), so the family loads with no remote code. The composite config's model_type is inkling_mm_model, the text config's inkling_text (there is no bare inkling spelling).
Per-axis support: Compatibility matrix. CP and TP are out for architectural reasons, not a missing registration (why). Pipeline parallelism splits the text tower on whole six-layer layer_types periods, with packing refused whatever allow_cross_document_linear_attention says (Pipeline Parallelism). LoRA is tiny-model verified only (test_lora_merged_save_resume.py --family inkling_text: expert and mixed adapters at ep2 and ep1 through a merged save and an exact resume).
EP layer
EPInklingMoELayer (halo/distributed/expert_parallel/families/inkling.py) claims InklingMoE / model_type: inkling_mm_model, inkling_text. Fused gate_up_proj/down_proj routed experts in the GLM-4 layout, so the base fused-GLU storage, gather, and sharded merge apply unchanged.
That includes ETP, whose split-shard path and token-space partial-sum reduce are validated by tests/gpu/parallelism/ep/test_ep_family_vs_reference.py --family inkling_text --expert-tp-size 2 (registered as EP+ETP on 4 GPUs; launched by hand at --nproc_per_node=2 it runs pure ETP; the 2 shared experts stay replicated on every ETP rank). Two family quirks force a standalone EP layer rather than a GLM-4 subclass:
Joint routed+shared normalization.
InklingTopkRouteremitsn_routed_experts + n_shared_expertslogits from one projection and normalizes the routed top-k and the shared experts jointly (logsumexpovertop_k + n_sharedlogits), so the shared experts compete for probability mass.The shared FFN is scaled by its resulting share (
gammas), an extra argumentEPSharedExpertsMoELayerBase's shared-expert call cannot pass, so the EP layer runs the shared leg itself.Selection vs gating split. Selection runs on
sigmoid(routed_logits) + e_score_correction_bias(a learned routing bias the family ships);moe_balancing: bias_updateadds its balancing bias on top for selection only, while the returned weights stay derived from the unbiased logits.The causal-LM loss never reads
router_logits, so there is no aux-loss path andmoe_balancing: autoresolves tobias_update.
One contract fails closed: the spec's weight_sync_refusal refuses online/async GRPO — no sync into either pinned engine's Inkling loader has been validated. An EP-gathered save of the composite writes the hub's own namespace (model.llm.*, the interleaved fused w13_weight) (Checkpoints).
Loading
The release's MTP tail (model.mtp.*, eight layers) drops at load and exports carry none, so method=mtp refuses them; mtp_loss_weight refuses the family.
The hub config spells the expert width intermediate_size: 2048 beside dense_intermediate_size: 16384. transformers 5.20 reads those as moe_intermediate_size 2048 and intermediate_size 16384, so a run needs no override: the built expert tensors match the hub's shard shapes.
The checkpoint ships Thinking Machines' original namespace (model.llm.*, attn.wq_du, interleaved experts.w13_weight), which transformers converts inside from_pretrained (transformers/conversion_mapping.py, keyed inkling_mm_model).
The lazy loaders plan their keys on the same declarative entries, as from_pretrained does, and translate the converters into a closed op vocabulary (de-interleave, chunk) applied at materialization, after the ranged per-expert read. A text-only artifact (an InklingTextConfig over the composite's weights), for which transformers maps no conversions, takes the composite entries the family spec declares in hub_conversion_keys. The default lazy path therefore loads the hub checkpoint directly, each rank reading only its expert slice, pinned bitwise against from_pretrained by tests/gpu/parallelism/ep/test_lazy_load_inkling.py.
The from_pretrained fallback still works but materializes the full checkpoint per concurrently-loading rank: 532 GB for Inkling-Small, so max_concurrent_loading: 4 ≈ 2.1 TB peak host RAM.
The family's fp32 pin covers the short convolutions (_keep_in_fp32_modules_strict: k_sconv/v_sconv/attn_sconv/mlp_sconv). Every training loader casts them to the run dtype (Load precision), so the sconvs train in bf16 unless the run keeps fp32 masters — a deliberate trade validated by the multi-node runs below.
Why CP and TP are out
Packing — the attention layers isolate packed documents, but the four depthwise convolutions per layer cross boundaries: the modeling reads
seq_idxyet its conv call sites never forward kwargs, so no collator emission can reach them.packingandpadding_freeare refused unlessallow_cross_document_linear_attention: trueaccepts that mixing (Document isolation).CP —
InklingShortConvolutionruns over the sequence axis (a CP shard severs its receptive field at every chunk boundary, the LFM-2/Zaya blocker), and the additive relative-logits bias cannot pass throughflash_attn_func, which the Ulysses path is built on (_supports_flash_attn = Falseupstream).Validation rejects the conv class with a precise error.
TP — the selective-TP planner shards q/k/v/o structurally; Inkling's attention carries per-layer head geometry (
swa_*on sliding layers), sequence convolutions on the projected K/V, and a per-headrel_logits_proj, none of which the planner can shard. The zero-shard raise names the class.
Multimodal training
The composite class trains under EP: patching finds InklingMoE under model.language_model, the
vision/audio towers survive as replicated (FSDP-managed) modules, and an image-carrying batch (pixel
features masked-scattered at image_token_id placeholders) backpropagates into the vision tower
alongside the expert shards (tests/gpu/parallelism/ep/test_ep_vlm_inkling.py, EP=2 vs the
undistributed composite reference).
Image-text SFT rides the VLM data path (VLMDataCollator + processing_inkling); text-only data
through the same class is what the multi-node runs below validated. CP rejects the class outright, so
multimodal is EP/ETP-only.
Multi-node EP
532 GB of bf16 weights do not fit node-local EP=8 alongside full-FT optimizer states — Inkling-Small trains at ep_size: 16 / ep_scope: global on 2×8 B300. Measured on that topology on Halo 1.0.0 or earlier:
- Keep
per_device_train_batch_size × max_lengthwithin the cross-node Gin dispatch cap (HALO_DEEPEP_GIN_MAX_TOKENS_PER_RANK, 8192 — DeepEP); the shipped config runs mbs 2 × 4096 = 8192, validated exactly at the cap. - Grouped GEMM and the auto SM count (the 24-SM zero-division fallback) run clean at Inkling's shapes. If the first cross-node combine times out intermittently, pin
HALO_DEEPEP_NUM_SMS=16— fewer channels shrink the barrier race window. - Measured: 3.7-4.1 s/step at mbs 2 × 4096 (2×8 B300, per-batch padding),
9,400 attended tok/s global (590/GPU), peak ~246 GB of the 288 GB card; DeepEP arena grows are exercised and stable.
Configs
examples/sft/inkling/inkling-small-multinode-ep16.yaml (EP=16 across 2 nodes, 256 experts → 16/rank). Two settings are load-bearing:
attn_implementation: sdpa— no Inkling FA dispatch; the Blackwell auto-select would fall back per-rank. Every attention call carries the dense relative bias, which the class declares by not being backend-compatible, so the sliding layers stay on SDPA rather thansdpa_flex_sliding.pad_token: "<|unused|>"— the tokenizer ships no pad token; id 199998 is a dedicated special that never appears in text.
Liger covers Inkling's RMSNorm and cross-entropy (the config runs it). Its MLP and head stay eager: InklingMLP scales its output by a trained global_scale, and the head divides by logits_mup_width_multiplier and truncates to unpadded_vocab_size before the loss. There is no rotary to fuse: position enters as a learned relative-logit bias.
That division and cut are the family's declared head transform (halo/models/head_transform.py), so offline GRPO's use_chunked_grpo_logprobs sweep applies them too.
Inkling loads as a ConditionalGeneration class, but the data path follows the run: text-only SFT rows take the text pipeline, so packing is available here under allow_cross_document_linear_attention: true (SFT — VLMs). The binding limit at EP=16 is the cross-node dispatch cap above — per_device_train_batch_size × max_length ≤ 8192 tokens/rank, however the rows are formed.
The shipped config sets packing: false: packed rows would attend the full 8192 tokens/rank/step, against the ~2.3k real tokens the measurement above ran at with a 246 of 288 GB peak, and the depthwise convs would cross packed documents (refused without the opt-in).
CPU coverage: tests/cpu/models/test_inkling_support.py (registration and router parity, joint normalization included). GPU gates: tests/gpu/parallelism/ep/test_ep_family_vs_reference.py --family inkling_text (with and without --expert-tp-size 2), tests/gpu/parallelism/ep/test_ep_vlm_inkling.py, tests/gpu/trainers/lora/test_lora_merged_save_resume.py (--family inkling_text).
Source on GitHubUpdated 18 November 2026