# Inkling

Thinking Machines' `InklingForConditionalGeneration` (`thinkingmachines/Inkling-Small`, 266B total / 12B active) — a multimodal MoE routing to 256 experts with top-6 selection, 2 always-on shared experts, and 40 MoE layers (of 42 — `dense_mlp_idx: 2` keeps the first two dense) at hidden 4096.

Attention is hybrid sliding (5 sliding : 1 full, period 6) with **no RoPE** — position enters as an additive relative-logits bias — plus depthwise causal short convolutions (`InklingShortConvolution`) on the projected K/V and around each decoder sublayer.

Transformers ships `transformers.models.inkling` natively (the image pins 5.20.2), so the family loads with no remote code. The composite config's `model_type` is `inkling_mm_model`, the text config's `inkling_text` (there is no bare `inkling` spelling).

Per-axis support: [Compatibility matrix](https://whitecircle.com/halo/docs/agent-docs/models.md#compatibility-matrix). CP and TP are out for architectural reasons, not a missing registration ([why](#why-cp-and-tp-are-out)). Pipeline parallelism splits the text tower on whole six-layer `layer_types` periods, with packing refused whatever `allow_cross_document_linear_attention` says ([Pipeline Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md#model-support)). LoRA is tiny-model verified only (`test_lora_merged_save_resume.py --family inkling_text`: expert and mixed adapters at ep2 and ep1 through a merged save and an exact resume).

## EP layer

`EPInklingMoELayer` (`halo/distributed/expert_parallel/families/inkling.py`) claims `InklingMoE` / `model_type: inkling_mm_model`, `inkling_text`. Fused `gate_up_proj`/`down_proj` routed experts in the GLM-4 layout, so the base fused-GLU storage, gather, and sharded merge apply unchanged.

That includes ETP, whose split-shard path and token-space partial-sum reduce are validated by `tests/gpu/parallelism/ep/test_ep_family_vs_reference.py --family inkling_text --expert-tp-size 2` (registered as EP+ETP on 4 GPUs; launched by hand at `--nproc_per_node=2` it runs pure ETP; the 2 shared experts stay replicated on every ETP rank). Two family quirks force a standalone EP layer rather than a GLM-4 subclass:

- **Joint routed+shared normalization.** `InklingTopkRouter` emits `n_routed_experts + n_shared_experts` logits from one projection and normalizes the routed top-k and the shared experts **jointly** (`logsumexp` over `top_k + n_shared` logits), so the shared experts compete for probability mass.

    The shared FFN is scaled by its resulting share (`gammas`), an extra argument `EPSharedExpertsMoELayerBase`'s shared-expert call cannot pass, so the EP layer runs the shared leg itself.

- **Selection vs gating split.** Selection runs on `sigmoid(routed_logits) + e_score_correction_bias` (a learned routing bias the family ships); `moe_balancing: bias_update` adds its balancing bias on top for selection only, while the returned weights stay derived from the unbiased logits.

    The causal-LM loss never reads `router_logits`, so there is no aux-loss path and `moe_balancing: auto` resolves to `bias_update`.

One contract fails closed: the spec's `weight_sync_refusal` refuses online/async GRPO — no sync into either pinned engine's Inkling loader has been validated. An EP-gathered save of the composite writes the hub's own namespace (`model.llm.*`, the interleaved fused `w13_weight`) ([Checkpoints](https://whitecircle.com/halo/docs/agent-docs/reference/checkpoints.md#serving-on-vllm--sglang)).

## Loading

The release's MTP tail (`model.mtp.*`, eight layers) drops at load and exports carry none, so `method=mtp` refuses them; `mtp_loss_weight` [refuses the family](https://whitecircle.com/halo/docs/agent-docs/optimization/mtp-heads.md).

The hub config spells the expert width `intermediate_size: 2048` beside `dense_intermediate_size: 16384`. transformers 5.20 reads those as `moe_intermediate_size` 2048 and `intermediate_size` 16384, so a run needs no override: the built expert tensors match the hub's shard shapes.

The checkpoint ships Thinking Machines' original namespace (`model.llm.*`, `attn.wq_du`, interleaved `experts.w13_weight`), which transformers converts inside `from_pretrained` (`transformers/conversion_mapping.py`, keyed `inkling_mm_model`).

The lazy loaders plan their keys on the same declarative entries, as `from_pretrained` does, and translate the converters into a closed op vocabulary (de-interleave, chunk) applied at materialization, after the ranged per-expert read. A text-only artifact (an `InklingTextConfig` over the composite's weights), for which transformers maps no conversions, takes the composite entries the family spec declares in `hub_conversion_keys`. The default lazy path therefore loads the hub checkpoint directly, each rank reading only its expert slice, pinned bitwise against `from_pretrained` by `tests/gpu/parallelism/ep/test_lazy_load_inkling.py`.

The `from_pretrained` fallback still works but materializes the full checkpoint per concurrently-loading rank: 532 GB for Inkling-Small, so `max_concurrent_loading: 4` ≈ 2.1 TB peak host RAM.

The family's fp32 pin covers the **short convolutions** (`_keep_in_fp32_modules_strict`: `k_sconv`/`v_sconv`/`attn_sconv`/`mlp_sconv`). Every training loader casts them to the run dtype ([Load precision](https://whitecircle.com/halo/docs/agent-docs/models.md#load-precision)), so the sconvs train in bf16 unless the run keeps fp32 masters — a deliberate trade validated by the multi-node runs below.

## Why CP and TP are out

- **Packing** — the attention layers isolate packed documents, but the four depthwise convolutions per layer cross boundaries: the modeling reads `seq_idx` yet its conv call sites never forward kwargs, so no collator emission can reach them. `packing` and `padding_free` are refused unless `allow_cross_document_linear_attention: true` accepts that mixing ([Document isolation](https://whitecircle.com/halo/docs/agent-docs/data/collators.md#document-isolation-under-packing)).
- **CP** — `InklingShortConvolution` runs over the sequence axis (a CP shard severs its receptive field at every chunk boundary, the LFM-2/Zaya blocker), and the additive relative-logits bias cannot pass through `flash_attn_func`, which the Ulysses path is built on (`_supports_flash_attn = False` upstream).

    Validation rejects the conv class with a precise error.

- **TP** — the selective-TP planner shards q/k/v/o structurally; Inkling's attention carries per-layer head geometry (`swa_*` on sliding layers), sequence convolutions on the projected K/V, and a per-head `rel_logits_proj`, none of which the planner can shard. The zero-shard raise names the class.

## Multimodal training

The composite class trains under EP: patching finds `InklingMoE` under `model.language_model`, the
vision/audio towers survive as replicated (FSDP-managed) modules, and an image-carrying batch (pixel
features masked-scattered at `image_token_id` placeholders) backpropagates into the vision tower
alongside the expert shards (`tests/gpu/parallelism/ep/test_ep_vlm_inkling.py`, EP=2 vs the
undistributed composite reference).

Image-text SFT rides the VLM data path (`VLMDataCollator` + `processing_inkling`); text-only data
through the same class is what the multi-node runs below validated. CP rejects the class outright, so
multimodal is EP/ETP-only.

## Multi-node EP

532 GB of bf16 weights do not fit node-local EP=8 alongside full-FT optimizer states — Inkling-Small trains at `ep_size: 16` / `ep_scope: global` on 2×8 B300. Measured on that topology on Halo 1.0.0 or earlier:

- Keep `per_device_train_batch_size × max_length` within the cross-node Gin dispatch cap (`HALO_DEEPEP_GIN_MAX_TOKENS_PER_RANK`, 8192 — [DeepEP](https://whitecircle.com/halo/docs/agent-docs/infrastructure/deepep.md)); the shipped config runs mbs 2 × 4096 = 8192, validated exactly at the cap.
- Grouped GEMM and the auto SM count (the 24-SM zero-division fallback) run clean at Inkling's shapes. If the first cross-node combine times out intermittently, pin `HALO_DEEPEP_NUM_SMS=16` — fewer channels shrink the barrier race window.
- Measured: 3.7-4.1 s/step at mbs 2 × 4096 (2×8 B300, per-batch padding), ~9,400 attended tok/s global (~590/GPU), peak ~246 GB of the 288 GB card; DeepEP arena grows are exercised and stable.

## Configs

`examples/sft/inkling/inkling-small-multinode-ep16.yaml` (EP=16 across 2 nodes, 256 experts → 16/rank). Two settings are load-bearing:

- `attn_implementation: sdpa` — no Inkling FA dispatch; the Blackwell auto-select would fall back per-rank. Every attention call carries the dense relative bias, which the class declares by not being backend-compatible, so the sliding layers stay on SDPA rather than `sdpa_flex_sliding`.
- `pad_token: "<|unused|>"` — the tokenizer ships no pad token; id 199998 is a dedicated special that never appears in text.

Liger covers Inkling's RMSNorm and cross-entropy (the config runs it). Its MLP and head stay eager: `InklingMLP` scales its output by a trained `global_scale`, and the head divides by `logits_mup_width_multiplier` and truncates to `unpadded_vocab_size` before the loss. There is no rotary to fuse: position enters as a learned relative-logit bias.

That division and cut are the family's declared head transform (`halo/models/head_transform.py`), so offline GRPO's `use_chunked_grpo_logprobs` sweep applies them too.

Inkling loads as a `ConditionalGeneration` class, but the data path follows the run: text-only SFT rows take the text pipeline, so `packing` is available here under `allow_cross_document_linear_attention: true` ([SFT — VLMs](https://whitecircle.com/halo/docs/agent-docs/training-methods/sft.md#vision-language-models)). The binding limit at EP=16 is the cross-node dispatch cap above — `per_device_train_batch_size × max_length ≤ 8192` tokens/rank, however the rows are formed.

The shipped config sets `packing: false`: packed rows would attend the full 8192 tokens/rank/step, against the ~2.3k real tokens the measurement above ran at with a 246 of 288 GB peak, and the depthwise convs would cross packed documents (refused without the opt-in).

CPU coverage: `tests/cpu/models/test_inkling_support.py` (registration and router parity, joint normalization included). GPU gates: `tests/gpu/parallelism/ep/test_ep_family_vs_reference.py --family inkling_text` (with and without `--expert-tp-size 2`), `tests/gpu/parallelism/ep/test_ep_vlm_inkling.py`, `tests/gpu/trainers/lora/test_lora_merged_save_resume.py` (`--family inkling_text`).
