Inkling

Thinking Machines' InklingForConditionalGeneration (thinkingmachines/Inkling-Small, 266B total / 12B active) — a multimodal MoE routing to 256 experts with top-6 selection, 2 always-on shared experts, and 40 MoE layers (of 42 — dense_mlp_idx: 2 keeps the first two dense) at hidden 4096.

Attention is hybrid sliding (5 sliding : 1 full, period 6) with no RoPE — position enters as an additive relative-logits bias — plus depthwise causal short convolutions (InklingShortConvolution) on the projected K/V and around each decoder sublayer.

Transformers ships transformers.models.inkling natively (the image pins 5.20.2), so the family loads with no remote code. The composite config's model_type is inkling_mm_model, the text config's inkling_text (there is no bare inkling spelling).

Per-axis support: Compatibility matrix. CP and TP are out for architectural reasons, not a missing registration (why). Pipeline parallelism splits the text tower on whole six-layer layer_types periods, with packing refused whatever allow_cross_document_linear_attention says (Pipeline Parallelism). LoRA is tiny-model verified only (test_lora_merged_save_resume.py --family inkling_text: expert and mixed adapters at ep2 and ep1 through a merged save and an exact resume).

EP layer

EPInklingMoELayer (halo/distributed/expert_parallel/families/inkling.py) claims InklingMoE / model_type: inkling_mm_model, inkling_text. Fused gate_up_proj/down_proj routed experts in the GLM-4 layout, so the base fused-GLU storage, gather, and sharded merge apply unchanged.

That includes ETP, whose split-shard path and token-space partial-sum reduce are validated by tests/gpu/parallelism/ep/test_ep_family_vs_reference.py --family inkling_text --expert-tp-size 2 (registered as EP+ETP on 4 GPUs; launched by hand at --nproc_per_node=2 it runs pure ETP; the 2 shared experts stay replicated on every ETP rank). Two family quirks force a standalone EP layer rather than a GLM-4 subclass:

  • Joint routed+shared normalization. InklingTopkRouter emits n_routed_experts + n_shared_experts logits from one projection and normalizes the routed top-k and the shared experts jointly (logsumexp over top_k + n_shared logits), so the shared experts compete for probability mass.

    The shared FFN is scaled by its resulting share (gammas), an extra argument EPSharedExpertsMoELayerBase's shared-expert call cannot pass, so the EP layer runs the shared leg itself.

  • Selection vs gating split. Selection runs on sigmoid(routed_logits) + e_score_correction_bias (a learned routing bias the family ships); moe_balancing: bias_update adds its balancing bias on top for selection only, while the returned weights stay derived from the unbiased logits.

    The causal-LM loss never reads router_logits, so there is no aux-loss path and moe_balancing: auto resolves to bias_update.

One contract fails closed: the spec's weight_sync_refusal refuses online/async GRPO — no sync into either pinned engine's Inkling loader has been validated. An EP-gathered save of the composite writes the hub's own namespace (model.llm.*, the interleaved fused w13_weight) (Checkpoints).

Loading

The release's MTP tail (model.mtp.*, eight layers) drops at load and exports carry none, so method=mtp refuses them; mtp_loss_weight refuses the family.

The hub config spells the expert width intermediate_size: 2048 beside dense_intermediate_size: 16384. transformers 5.20 reads those as moe_intermediate_size 2048 and intermediate_size 16384, so a run needs no override: the built expert tensors match the hub's shard shapes.

The checkpoint ships Thinking Machines' original namespace (model.llm.*, attn.wq_du, interleaved experts.w13_weight), which transformers converts inside from_pretrained (transformers/conversion_mapping.py, keyed inkling_mm_model).

The lazy loaders plan their keys on the same declarative entries, as from_pretrained does, and translate the converters into a closed op vocabulary (de-interleave, chunk) applied at materialization, after the ranged per-expert read. A text-only artifact (an InklingTextConfig over the composite's weights), for which transformers maps no conversions, takes the composite entries the family spec declares in hub_conversion_keys. The default lazy path therefore loads the hub checkpoint directly, each rank reading only its expert slice, pinned bitwise against from_pretrained by tests/gpu/parallelism/ep/test_lazy_load_inkling.py.

The from_pretrained fallback still works but materializes the full checkpoint per concurrently-loading rank: 532 GB for Inkling-Small, so max_concurrent_loading: 4 ≈ 2.1 TB peak host RAM.

The family's fp32 pin covers the short convolutions (_keep_in_fp32_modules_strict: k_sconv/v_sconv/attn_sconv/mlp_sconv). Every training loader casts them to the run dtype (Load precision), so the sconvs train in bf16 unless the run keeps fp32 masters — a deliberate trade validated by the multi-node runs below.

Why CP and TP are out

  • Packing — the attention layers isolate packed documents, but the four depthwise convolutions per layer cross boundaries: the modeling reads seq_idx yet its conv call sites never forward kwargs, so no collator emission can reach them. packing and padding_free are refused unless allow_cross_document_linear_attention: true accepts that mixing (Document isolation).

  • CP — InklingShortConvolution runs over the sequence axis (a CP shard severs its receptive field at every chunk boundary, the LFM-2/Zaya blocker), and the additive relative-logits bias cannot pass through flash_attn_func, which the Ulysses path is built on (_supports_flash_attn = False upstream).

    Validation rejects the conv class with a precise error.

  • TP — the selective-TP planner shards q/k/v/o structurally; Inkling's attention carries per-layer head geometry (swa_* on sliding layers), sequence convolutions on the projected K/V, and a per-head rel_logits_proj, none of which the planner can shard. The zero-shard raise names the class.

Multimodal training

The composite class trains under EP: patching finds InklingMoE under model.language_model, the vision/audio towers survive as replicated (FSDP-managed) modules, and an image-carrying batch (pixel features masked-scattered at image_token_id placeholders) backpropagates into the vision tower alongside the expert shards (tests/gpu/parallelism/ep/test_ep_vlm_inkling.py, EP=2 vs the undistributed composite reference).

Image-text SFT rides the VLM data path (VLMDataCollator + processing_inkling); text-only data through the same class is what the multi-node runs below validated. CP rejects the class outright, so multimodal is EP/ETP-only.

Multi-node EP

532 GB of bf16 weights do not fit node-local EP=8 alongside full-FT optimizer states — Inkling-Small trains at ep_size: 16 / ep_scope: global on 2×8 B300. Measured on that topology on Halo 1.0.0 or earlier:

  • Keep per_device_train_batch_size × max_length within the cross-node Gin dispatch cap (HALO_DEEPEP_GIN_MAX_TOKENS_PER_RANK, 8192 — DeepEP); the shipped config runs mbs 2 × 4096 = 8192, validated exactly at the cap.
  • Grouped GEMM and the auto SM count (the 24-SM zero-division fallback) run clean at Inkling's shapes. If the first cross-node combine times out intermittently, pin HALO_DEEPEP_NUM_SMS=16 — fewer channels shrink the barrier race window.
  • Measured: 3.7-4.1 s/step at mbs 2 × 4096 (2×8 B300, per-batch padding), 9,400 attended tok/s global (590/GPU), peak ~246 GB of the 288 GB card; DeepEP arena grows are exercised and stable.

Configs

examples/sft/inkling/inkling-small-multinode-ep16.yaml (EP=16 across 2 nodes, 256 experts → 16/rank). Two settings are load-bearing:

  • attn_implementation: sdpa — no Inkling FA dispatch; the Blackwell auto-select would fall back per-rank. Every attention call carries the dense relative bias, which the class declares by not being backend-compatible, so the sliding layers stay on SDPA rather than sdpa_flex_sliding.
  • pad_token: "<|unused|>" — the tokenizer ships no pad token; id 199998 is a dedicated special that never appears in text.

Liger covers Inkling's RMSNorm and cross-entropy (the config runs it). Its MLP and head stay eager: InklingMLP scales its output by a trained global_scale, and the head divides by logits_mup_width_multiplier and truncates to unpadded_vocab_size before the loss. There is no rotary to fuse: position enters as a learned relative-logit bias.

That division and cut are the family's declared head transform (halo/models/head_transform.py), so offline GRPO's use_chunked_grpo_logprobs sweep applies them too.

Inkling loads as a ConditionalGeneration class, but the data path follows the run: text-only SFT rows take the text pipeline, so packing is available here under allow_cross_document_linear_attention: true (SFT — VLMs). The binding limit at EP=16 is the cross-node dispatch cap above — per_device_train_batch_size × max_length ≤ 8192 tokens/rank, however the rows are formed.

The shipped config sets packing: false: packed rows would attend the full 8192 tokens/rank/step, against the ~2.3k real tokens the measurement above ran at with a 246 of 288 GB peak, and the depthwise convs would cross packed documents (refused without the opt-in).

CPU coverage: tests/cpu/models/test_inkling_support.py (registration and router parity, joint normalization included). GPU gates: tests/gpu/parallelism/ep/test_ep_family_vs_reference.py --family inkling_text (with and without --expert-tp-size 2), tests/gpu/parallelism/ep/test_ep_vlm_inkling.py, tests/gpu/trainers/lora/test_lora_merged_save_resume.py (--family inkling_text).

Source on GitHubUpdated 18 November 2026