Tensor Parallelism (TP)

TP shards large weight matrices across GPUs. Use it when a dense model does not fit on one GPU: its HF-native load places each rank's shard as it reads, while plain FSDP2 and DDP load the whole model onto every GPU first (Data Parallelism). data_parallel_size = world_size / tp_size; all ranks participate in gradient sync (ParallelismConfig, halo/distributed/parallelism_config.py). Why its all-reduce has to stay on NVLink: GPU Training Theory §9.

Two sharding mechanisms are in play — HF-native TP (tp_plan="auto", dense models) and the toolkit's selective attention TP — and both place their sharded params as DTensors on the TP mesh, so one full_tensor() walk reconstructs either. The exception is the params sliced by hand (GptOss sinks), which stay plain tensors and are load-bearing for gradients and checkpoints. See Gradient sync. Both run their linears on the local shards rather than through DTensor dispatch: Local-shard linears.

Supported models

Native HF TP (dense models with base_model_tp_plan, via tp_plan="auto" — shards attention, MLP, and the vocab dim of a tied embedding/head pair): Llama, Mistral, Qwen3, Qwen3.5/3.6 dense, any dense HF model declaring the plan.

Selective TP (attention only, apply_tp_to_attention_only(); MoE handled by EP, embedding/lm_head replicated):

ModelNotesPage
GPT-OSSAttention sinks sharded as non-DTensor, all-gathered at savegpt-oss.md
Qwen3 MoEcolwise Q/K/V + rowwise Oqwen3.md
Qwen3.5 / Qwen3.6 MoEDouble-width q_proj shards uniformlyqwen3_5.md
GLM-4 MoE LiteMLA — of the colwise projections only q_b_proj / kv_b_proj shard (o_proj is still rowwise)glm4.md
LFM-2 MoEStandard GQA patternlfm2.md
Mistral4MLA — same as GLM-4mistral4.md
Cohere2 MoEStandard GQA pattern; the tied lm_head stays replicatedcohere2-moe.md

Every MoE under pure TP routes to load_tp_moe_model rather than tp_plan="auto", because HF's auto plan wrongly shards MoE expert biases. Supported attention classes: TP_SHARDABLE_ATTENTION_CLASSES in halo/distributed/tensor_parallel/parallelize_attention.py.

Not supported: every family absent from the table above. Their attention class is in no registry, so a tp_size > 1 run shards zero layers and raises rather than leaving every weight replicated; a dense checkpoint shipping no base_model_tp_plan (Qwen3-VL) raises at the HF-native load. Per family, with the reason on its page: Compatibility matrix.

Load paths

Three paths in load_distributed_model (halo/model_loading/parallel/loader.py):

PathWhenMechanism
load_tp_modelPure TP, dense, ep_size=1from_pretrained(distributed_config=DistributedConfig(tp_plan="auto")) — HF-native
load_tp_moe_modelPure TP, MoE, ep_size=1apply_tp_to_attention_only(); experts replicated
load_ep_tp_modelEP+TPthe EP loader slices each rank's experts, lazy or eager, then apply_tp_to_attention_only()

The HF-native path validates the load's materialization, not just its plan (finish_hf_tp_load): a plan that shards zero parameters raises (the run would be tp_size replicas), and a plan-sharded param that materialized as a plain tensor raises too (reject_plan_sharded_plain_params, halo/distributed/tensor_parallel/hf_plan.py) — such a bare slice is indistinguishable from a replica downstream, and the replicated-grad AVG would mix disjoint slices across the TP group. Hand-sliced params (register_hand_sliced, halo/distributed/tensor_parallel/plain_params.py) are exempt.

Tied embeddings under HF-native TP

load_tp_model shards a tied embedding/head pair as one vocab-parallel weight. A tied config injects embed_tokens: embedding_rowwise into the backbone base_model_tp_plan, the ForCausalLM's lm_head: colwise_gather_output agrees with it on Shard(0) of the vocab dim, and transformers shards module by module and ties afterwards — so both ends land on the same DTensor: one weight, one gradient, one exported key, and half the embedding + head memory per rank at tp_size=2.

Sharding the pair makes the vocabulary a TP dimension: colwise_gather_output refuses a vocab_size that tp_size does not divide (transformers' ColwiseParallel.validate_param, loud at load, naming the layer), where a replicated pair did not care. Every tied dense model in the roster divides evenly (Qwen3 151936, Qwen3.5 248320); a vocab-patched checkpoint that does not must drop tp_size to a divisor.

An architecture planning only one end of the pair (a multimodal wrapper class declaring no lm_head entry; a backbone shipping no base_model_tp_plan) cannot do that: the tie would hand the unplanned end a weight of the other kind and the first forward dies on mixed plain/DTensor operands.

consistent_tied_tp_plan (halo/distributed/tensor_parallel/hf_plan.py) drops that lone entry for the load so the pair stays replicated instead, warning that those weights cost a full copy per rank. validate_tied_pair_consistent then fails the load loud if the loaded pair is untied, or if its sharding disagrees with the applied plan.

For a checkpoint carrying both tied keys on disk (the Qwen3-0.6B/1.7B export shape) transformers compares them before tying, which on two DTensors is an all-reduce on the default process group: every rank must reach it.

load_tp_model therefore does not throttle the load with max_concurrent_loading. A rank-serialized load blocks the loading rank inside that collective while its peers wait their turn, and the job hangs until the NCCL watchdog fires.

Nothing is lost: transformers streams the checkpoint key by key and places each rank's shard straight on its GPU, so host RAM never holds the model. The FP32-master restore that follows (fp32_non_ep_params) stages each full FP32 tensor in host memory, so it does take the throttle (Multi-Node → Model loading).

The selective-TP plan

apply_tp_to_attention_only (halo/distributed/tensor_parallel/parallelize_attention.py) walks the model and shards each attention projection under its style (shard_linear, rank 0's values distributed into DTensor placements):

ModuleStrategyNotes
self_attn.{q,k,v}_projcolwiseper-head
self_attn.{q_b,kv_b}_projcolwiseMLA: only the head-dim expansion shards
self_attn.{o,out}_projrowwisethe all-reduce point
self_attn.sinkssliced (shard_sinks_param)GptOss, one slice per head shard
per-head norms (q_norm/k_norm, LFM2 q_layernorm/k_layernorm)replicated + grad all-reducesee below
embed_tokens / lm_headreplicatedthe plan never visits them: DTensor-sharding them faults cuBLAS under EP+TP

Head divisibility is validated before sharding: num_attention_heads % tp_size == 0 always, and num_key_value_heads % tp_size == 0 for standard GQA. The colwise split of Q/K/V knows nothing of heads, so an indivisible count would split within a head; the guard raises instead. MLA models skip the KV-head check — they shard by query head and keep the KV compression replicated.

Per-head norm gradients. A norm applied after a colwise projection is a replicated (head_dim,) parameter shared across heads. A colwise projection returns this rank's heads only, so each rank's gradient covers only its own heads: the true gradient is the SUM over the TP group, not the average a replica gets.

apply_tp_to_attention_only finds those norms structurally (any unplanned attention child with trainable parameters, never a name list), records their names through register_per_head_norm_params, and _sync_tp_replicated_grads sums them once per optimizer step. A name missing from that set is AVG-reduced as a plain replica, i.e. 1/tp_size of its true gradient.

MLA families are exempt: their q_a/kv_a norms sit before the colwise expansion, whose input gradient is already summed over the TP group. The rope half of their kv_a_proj_with_mqa output bypasses kv_b_proj, so a forward hook (register_mla_rope_grad_reduction) sums its gradient over the TP group.

Transformers' own ReplicatedWithGradAllReduce is deliberately not used: its full_backward_hook re-reduces whatever sits in .grad on every backward, so under gradient accumulation each earlier micro-batch's contribution is multiplied by tp_size again per micro-step. On the HF-native tp_plan="auto" path retarget_hf_replicated_grad_hooks strips those hooks into the same step-time SUM, and raises if the plan asks for that reduction and no hook is found. Guarded by tests/gpu/parallelism/tp/test_tp_attention_norm_grad.py.

MoE layers are never in this plan. In EP+TP the experts belong to EP; in pure-TP MoE they stay replicated.

Identical-input requirement

Every TP rank must call model.forward with identical input_ids / attention_mask shapes and values — column/row-parallel collectives assume each rank computes a partial result for the same hidden states. Mismatched inputs do not error early; they deadlock at the first o_proj all-reduce (visible in py-spy as ranks stuck at different sub-modules of one forward).

This affects trainers pulling per-rank data from an external service. DistributedOnlineGRPOTrainer overrides _generate_single_turn to broadcast the vLLM rollout tuple from the TP-group leader. For custom trainers, add an assert_rank_uniform call (halo.runtime.consensus; pass tuple(t.shape) to compare a shape, strict=True to raise instead of warn) at the suspect tensor — the toolkit ships no call sites.

Local-shard linears

Every TP linear of either mechanism computes F.linear on its local shards (weight.to_local()) between explicit c10d collectives instead of dispatching through DTensor, which costs about ten dispatches per linear and copies the activation inside every functional all-reduce (halo/distributed/tensor_parallel/local_linears.py). The parameters stay DTensors in the same placements, so FSDP2, checkpoints, export, the optimizer, the clip and the weight sync see the same layout. Each style runs the collectives its DTensor redistributes ran, on the same values and group:

StyleWeight / biasForwardBackward
colwiseShard(0) / Shard(0)local output shardinput gradient SUM
colwise_gather_outputShard(0) / Shard(0)output all-gathered on the last dimthis rank's gradient slice; input gradient SUM
rowwiseShard(1) / replicatedin-place output SUM, then the biasidentity
rowwise_split_inputShard(1) / replicatedthis rank's input slice; in-place output SUM, then the biasinput gradient all-gathered

The HF-native load installs this forward on every planned nn.Linear at the load finish (install_planned_local_forwards, called from finish_hf_tp_load); the selective plan shards and installs it in one step, and apply_lowp_compute re-installs it around the dense MLP projections lowp_precision retypes, computing their fake-quant matmul on the local shards. Embeddings (embedding_rowwise), the MoE and activation styles, and a Linear subclass with a forward of its own keep transformers' forward. A projection called outside its FSDP unit's forward, with FSDP2's (dp, tp) shard still registered, raises rather than computing part of its output.

The results are bit-identical to DTensor dispatch with one exception: a rowwise bias is added once after the SUM, in the matmul's dtype, where DTensor added bias / tp_size to each rank's partial, so GPT-OSS's biased o_proj differs in the last bit (tests/cpu/parallelism/test_tp_local_linears.py: every TP family of the tiny roster at TP 2, both plans under FSDP2 at dp 2 × tp 2 and under activation checkpointing).

Against DTensor dispatch on 8× B300 at s4096 (1.2.0, three rounds), Qwen3-8B TP2 runs 57.5% faster with the step-1 loss bitwise and gpt-oss-20b ep8 × tp2 11.0% faster.

Quick start

Bash

# Pure TP=4 on 4 GPUs (no data parallelism)
torchrun --nproc_per_node=4 scripts/training/sft.py \
    examples/sft/qwen3/qwen3-4b-ultrachat.yaml \
    --tensor_parallel_size=4

--tensor_parallel_size defaults to 1 (TP off) and is the only TP flag — there is no --tensor_parallel_plan. Dense TP always passes distributed_config=DistributedConfig(tp_plan="auto"); tp_plan and device_map are mutually exclusive, so do not set device_map.

TP + data parallelism

2D (dp, tp) mesh: TP within a node, FSDP2 across replicas. DTensor gradients cannot be reduced with plain dist.all_reduce or DDP — FSDP2's fully_shard understands DTensor semantics. The loader builds the mesh and attaches it to the model; the trainer wraps over the DP dimension via apply_fsdp2_for_tp (halo/distributed/fsdp.py), reusing that mesh. EP+TP takes the same entry point with its EP modules as ignored_params. Pure TP (DP=1) skips FSDP. Prefer tp_size = world_size when the model fits the TP group's memory.

Gradient sync and the grad norm

A TP run holds four kinds of parameter, and each needs a different reduction. Getting the kind wrong is silent — the loss stays finite and the objective is just different:

KindProduced byGrad syncGrad-norm bucket
DTensor shardeither mechanism — HF's tp_plan styles, selective attention TPDTensor / FSDP2summed over the axes its mesh names
Plain slicehand-sliced GptOss sinks (register_hand_sliced)none — the slices are already disjointsummed over the TP group
Replicaembed_tokens/lm_head under selective TP, router, norms outside attention_sync_tp_replicated_grads averages over the TP group (the local dp shards once FSDP2 wraps them — TP siblings hold the same shard)counted once, no TP reduce
Per-head attention norma norm after a colwise projection (q_norm/k_norm)_sync_tp_replicated_grads sums over the TP groupcounted once, no TP reduce

The trap is the plain slice. It is indistinguishable from a replica by tensor type, and once FSDP2 wraps the run for DP both become 1-D dp DTensors, so the mesh cannot separate them either.

_tp_sharded_plain_param_ids() (halo/trainers/parallelism/grad_sync.py) is the single source of truth: it reads the hand-sliced registry (hand_sliced_suffixes: the suffixes register_hand_sliced records, each with its shard dim, when a param is sliced by hand), and both _sync_tp_replicated_grads and _sharded_grad_bucket key off it. Everything a TP plan shards is a DTensor whose gradient reduces itself, so the plan contributes no entries here.

Classifying a slice as a replica costs twice: the TP-group average mixes another rank's heads into every slice's gradient, and the norm counts only the local slice, making the global norm rank-dependent so TP peers clip the same parameter by different coefficients. Guarded by tests/gpu/parallelism/tp/test_tp_correctness.py and test_tp_dp_correctness.py, both against an unsharded single-GPU grad norm.

EP+TP mode

EP+TP shards attention with TP and distributes MoE experts with EP on the same GPU groups (load_ep_tp_model). Router, embed_tokens/lm_head, and norms stay replicated. ep_size must be a multiple of tp_size, TP stays NVLink-local, and DP = world_size / tp_size. Its topology rules: Topology rules; example shapes and the gradient-sync groups: multi-node.md.

For MoE where only expert memory is the bottleneck, shard the expert FFN with ETP instead. Attention TP and expert TP are mutually exclusive.

Numerics. EP+TP in bf16 carries a deterministic offset against an unsplit reference: TP reorders the attention reductions feeding the router, which flips a handful of near-tied top-k expert picks and moves whole-expert mass. Gradient scale and direction still match; this is bf16 reduction order, not a sync bug.

Measured 0.060 loss offset on gpt-oss-20b (top-4-of-32, before 1.2.0), so the correctness gate uses TOL.router_pick_flip_loss_abs (0.1, shared with the EP+ETP gate) instead of the generic parallel-vs-baseline bound.

Limitations

Trainers. No trainer rejects TP; its trainer-level gates reject knobs (LoRA and load_best_model_at_end below, online GRPO's rollout_func). Matrix: Trainer Compatibility.

Models. There is no per-family "supports TP" flag. Selective TP shards exactly the attention classes in TP_SHARDABLE_ATTENTION_CLASSES (halo/distributed/tensor_parallel/parallelize_attention.py), and a tp_size > 1 run that shards zero layers raises, naming the model's attention classes; that is how the unsupported families are rejected. The check runs on a meta-device shell before any weight is read (reject_unsupported_structure), and again when the plan is applied.

A run that shards some layers (hybrid stacks — Qwen3.5/3.6 GatedDeltaNet, LFM-2 short-conv) only warns: numerics stay correct, but the unsharded layers keep a full replica per rank, so per-rank memory falls by far less than 1/tp_size.

Head divisibility is checked first: num_attention_heads % tp_size == 0 always, num_key_value_heads % tp_size == 0 for non-MLA GQA. The dense HF-native tp_plan="auto" path gets the same gate: reject_unsupported_model runs validate_tp_head_divisibility on every load path, off config.json before any weight is read.

Transformers validates no head count of its own, so without it the run dies on the first forward's reshape, after the whole checkpoint has been pulled and placed on every rank.

Axis combinations. TP composes with EP only; the allowlist and the EP+TP topology rules: Supported combinations.

Knobs. Everything below raises unless the verdict says otherwise.

KnobUnder TPGate
use_peft / LoRA (incl. EP+TP)rejected — PEFT adapters are plain tensors outside the TP graph, so the replicated half diverges and the sharded half is corrupted by the replicated-grad sync. Use LoRA with FSDP/DP, CP, or pure ETPreject_lora_under_tp
native EP expert LoRArejected — the grouped expert adapters ride the EP-distributed expert weights, which both TP gates skip by param identity, so no adapter shape under EP+TP is gradient- or save/merge-covered. Train expert LoRA under EP without TPsame
QLoRA / load_in_4bitrejected — the TP loaders materialize plain de-quantized weightsmodel_loading/parallel/load_rules.py
fsdp_reshard_after_forwardrejected at data_parallel_size > 1 — a plain all-gather on TP-sharded DTensor params has no registered sharding strategy_reject_fsdp_settings
fsdp_reshard_after_backward: falserejected — the TP setup shards through its own fully_shard callssame
fsdp_defer_grad_syncrejected at data_parallel_size == 1 — no FSDP2 wrap, so no reduce to defersame
fsdp_shard_ep1_experts: falserejected — the TP path shards ep1 experts unconditionallysame
use_hsdprejected — TP builds its own (dp, tp) mesh_reject_hsdp
init_from_scratchrejected — no sharded random initmodel_loading/parallel/load_rules.py
accelerate launchrejected — TP requires torchrunreject_unsupported_run, reject_custom_parallelism_under_accelerate
ref_model (explicit)warned — the reference is never parallelized, so every rank holds a whole dense replica; its log-probs match the policy's up to kernel numericswarn_unparallelized_reference
load_best_model_at_endsupported under pure TP — the reload distributes each checkpoint tensor into the live placements; rejected as a full fine-tune under TP+DP and under a MoE carrying EP layersreject_unreloadable_best_model
use_liger_kernelsupported; cross_entropy and fused_linear_cross_entropy are forced off (warned when explicitly enabled)halo/model_loading/liger/dispatch.py
added_special_tokens that grow the vocabrejected on a dense tied model, whose embedding is a vocab-sharded DTensor that resize_token_embeddings cannot re-shard. Patch the vocab offline (scripts/before_training/patch_vocab.py) insteadconfigure_run_tokenizer (is_input_embedding_tp_sharded)
gradient_checkpointing, packing, padding_free, lowp_precision, moe_balancingnot gated under TP—
torch_compilenot gated and unmeasured under TP: the local-shard linears and their collectives trace without a graph break (Graph breaks), but FA4 still breaks it at every attention call. Use Liger—

Multi-node

TP requires an NVLink-bandwidth all-reduce after every layer, so it must stay within a domain; cross-node TP over InfiniBand would bottleneck every pass. The pattern is TP within nodes, FSDP across nodes. nvlink_domain_size % tp_size == 0 is enforced at config time.

Bash

# Each node (set --node_rank per node)
torchrun --nnodes=2 --node_rank=0 --nproc_per_node=4 \
    --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT \
    scripts/training/sft.py <config>.yaml --tensor_parallel_size=4

The NCCL watchdog (DIST_NCCL_TIMEOUT_MINUTES) covers the TP mesh too: Launch Recipes → Environment variables.

Checkpoint saving

save_tp_checkpoint (halo/checkpoint/parallel/save.py), the ladder's TP rung, always gathers: all TP ranks reconstruct full tensors and one save rank writes a standard HF checkpoint, put through normalize_gathered_state_dict (hub expert layout, save dtype) like the FSDP2 and CP writers. There is no per-rank TP save. The write streams through stream_gathered_checkpoint, so the save rank's host RAM is bounded by one chunk plus the shards it is filling and writing (Checkpoints) rather than the whole model.

chunked_saveable_tensors reconstructs every sharded param with DTensor.full_tensor() — one walk covers both mechanisms — and gather_hand_sliced_params (halo/checkpoint/parallel/writing.py) all-gathers the hand-sliced GptOss sinks after it. Both are collective: every TP-mesh rank must drive them or the gather hangs.

On resume every TP shape is Path B: the training scripts repoint model_name_or_path at the checkpoint, so the weights load at construction and load_resume_weights skips the re-read. Where it does read (a best-model reload, or a model built from elsewhere), each rank streams the checkpoint's full tensors and distribute_tensors them into the live DTensor placements.

Configured FP32 masters are restored at construction, before the DP wrap (Load precision), so the constructed-from-checkpoint skip keeps their precision without a reload into a 2-D packed FSDP2 layout.

TP+DP is the exception: FSDP2 over TP stacks a strided dp shard on the tp shard, a 2-D placement distribute_tensor does not invert for packed projections, so it refuses every reload but the constructed-from-checkpoint skip. See Checkpoints.

Troubleshooting

  • "tp_plan and device_map are mutually exclusive" — drop device_map.
  • Quiet non-main ranks — expected: transformers drops to ERROR off the main rank, and the script logger follows log_level_replica (HF default warning).
  • DTensor all_reduce error with DDP — DTensors cannot be used with DDP; use FSDP2.
  • OOM with TP + FSDP — enable gradient checkpointing.

Source on GitHubUpdated 18 November 2026