Tensor Parallelism (TP)
TP shards large weight matrices across GPUs. Use it when a dense model does not fit on one GPU: its
HF-native load places each rank's shard as it reads, while plain FSDP2 and DDP load the whole model
onto every GPU first (Data Parallelism).
data_parallel_size = world_size / tp_size; all ranks participate in gradient sync
(ParallelismConfig, halo/distributed/parallelism_config.py). Why its all-reduce has to stay on
NVLink: GPU Training Theory §9.
Two sharding mechanisms are in play — HF-native TP (tp_plan="auto", dense models) and the
toolkit's selective attention TP — and both place their sharded params as DTensors on the TP
mesh, so one full_tensor() walk reconstructs either. The exception is the params sliced by hand
(GptOss sinks), which stay plain tensors and are load-bearing for gradients and checkpoints. See
Gradient sync. Both run their linears on the local shards
rather than through DTensor dispatch: Local-shard linears.
Supported models
Native HF TP (dense models with base_model_tp_plan, via tp_plan="auto" — shards attention,
MLP, and the vocab dim of a tied embedding/head pair): Llama,
Mistral, Qwen3, Qwen3.5/3.6 dense, any dense HF model declaring the plan.
Selective TP (attention only, apply_tp_to_attention_only(); MoE handled by EP,
embedding/lm_head replicated):
Every MoE under pure TP routes to load_tp_moe_model rather than tp_plan="auto", because
HF's auto plan wrongly shards MoE expert biases. Supported attention classes:
TP_SHARDABLE_ATTENTION_CLASSES in halo/distributed/tensor_parallel/parallelize_attention.py.
Not supported: every family absent from the table above. Their attention class is in no
registry, so a tp_size > 1 run shards zero layers and raises rather than leaving every weight
replicated; a dense checkpoint shipping no base_model_tp_plan (Qwen3-VL) raises at the HF-native
load. Per family, with the reason on its page:
Compatibility matrix.
Load paths
Three paths in load_distributed_model (halo/model_loading/parallel/loader.py):
The HF-native path validates the load's materialization, not just its plan (finish_hf_tp_load): a plan
that shards zero parameters raises (the run would be tp_size replicas), and a plan-sharded param that materialized
as a plain tensor raises too (reject_plan_sharded_plain_params,
halo/distributed/tensor_parallel/hf_plan.py) — such a bare slice is indistinguishable from a
replica downstream, and the replicated-grad AVG would mix disjoint slices across the TP group.
Hand-sliced params (register_hand_sliced, halo/distributed/tensor_parallel/plain_params.py) are exempt.
Tied embeddings under HF-native TP
load_tp_model shards a tied embedding/head pair as one vocab-parallel weight. A tied config
injects embed_tokens: embedding_rowwise into the backbone base_model_tp_plan, the ForCausalLM's
lm_head: colwise_gather_output agrees with it on Shard(0) of the vocab dim, and transformers
shards module by module and ties afterwards — so both ends land on the same DTensor: one weight,
one gradient, one exported key, and half the embedding + head memory per rank at tp_size=2.
Sharding the pair makes the vocabulary a TP dimension: colwise_gather_output refuses a
vocab_size that tp_size does not divide (transformers' ColwiseParallel.validate_param, loud at
load, naming the layer), where a replicated pair did not care. Every tied dense model in the roster divides
evenly (Qwen3 151936, Qwen3.5 248320); a vocab-patched checkpoint that does not must drop tp_size
to a divisor.
An architecture planning only one end of the pair (a multimodal wrapper class declaring no
lm_head entry; a backbone shipping no base_model_tp_plan) cannot do that: the tie would hand the
unplanned end a weight of the other kind and the first forward dies on mixed plain/DTensor operands.
consistent_tied_tp_plan (halo/distributed/tensor_parallel/hf_plan.py) drops that lone entry for
the load so the pair stays replicated instead, warning that those weights cost a full copy per rank.
validate_tied_pair_consistent then fails the
load loud if the loaded pair is untied, or if its sharding disagrees with the applied plan.
For a checkpoint carrying both tied keys on disk (the Qwen3-0.6B/1.7B export shape) transformers compares them before tying, which on two DTensors is an all-reduce on the default process group: every rank must reach it.
load_tp_model therefore does not throttle the load with max_concurrent_loading. A
rank-serialized load blocks the loading rank inside that collective while its peers wait their turn,
and the job hangs until the NCCL watchdog fires.
Nothing is lost: transformers streams the checkpoint key by key and places each rank's shard
straight on its GPU, so host RAM never holds the model. The FP32-master restore that follows
(fp32_non_ep_params) stages each full FP32 tensor in host memory, so it does take the throttle
(Multi-Node → Model loading).
The selective-TP plan
apply_tp_to_attention_only (halo/distributed/tensor_parallel/parallelize_attention.py) walks the
model and shards each attention projection under its style (shard_linear, rank 0's values
distributed into DTensor placements):
Head divisibility is validated before sharding: num_attention_heads % tp_size == 0 always, and
num_key_value_heads % tp_size == 0 for standard GQA. The colwise split of Q/K/V knows nothing of
heads, so an indivisible count would split within a head; the guard raises instead.
MLA models skip the KV-head check — they shard by query head and keep the KV compression replicated.
Per-head norm gradients. A norm applied after a colwise projection is a replicated
(head_dim,) parameter shared across heads. A colwise projection returns this rank's heads only, so
each rank's gradient covers only its own heads: the true gradient is the SUM over the TP group,
not the average a replica gets.
apply_tp_to_attention_only finds those norms structurally (any unplanned attention child with
trainable parameters, never a name list), records their names through register_per_head_norm_params,
and _sync_tp_replicated_grads sums them once per optimizer step. A name missing from that set is
AVG-reduced as a plain replica, i.e. 1/tp_size of its true gradient.
MLA families are exempt: their q_a/kv_a norms sit before the colwise expansion, whose input
gradient is already summed over the TP group. The rope half of their kv_a_proj_with_mqa
output bypasses kv_b_proj, so a forward hook (register_mla_rope_grad_reduction) sums its
gradient over the TP group.
Transformers' own ReplicatedWithGradAllReduce is deliberately not used: its
full_backward_hook re-reduces whatever sits in .grad on every backward, so under gradient
accumulation each earlier micro-batch's contribution is multiplied by tp_size again per
micro-step. On the HF-native tp_plan="auto" path retarget_hf_replicated_grad_hooks strips those
hooks into the same step-time SUM, and raises if the plan asks for that reduction and no hook is
found. Guarded by tests/gpu/parallelism/tp/test_tp_attention_norm_grad.py.
MoE layers are never in this plan. In EP+TP the experts belong to EP; in pure-TP MoE they stay replicated.
Identical-input requirement
Every TP rank must call model.forward with identical input_ids / attention_mask shapes
and values — column/row-parallel collectives assume each rank computes a partial result for the
same hidden states. Mismatched inputs do not error early; they deadlock at the first o_proj
all-reduce (visible in py-spy as ranks stuck at different sub-modules of one forward).
This affects trainers pulling per-rank data from an external service. DistributedOnlineGRPOTrainer
overrides _generate_single_turn to broadcast the vLLM rollout tuple from the TP-group leader. For
custom trainers, add an assert_rank_uniform call (halo.runtime.consensus; pass
tuple(t.shape) to compare a shape, strict=True to raise instead of warn) at the suspect tensor —
the toolkit ships no call sites.
Local-shard linears
Every TP linear of either mechanism computes F.linear on its local shards (weight.to_local())
between explicit c10d collectives instead of dispatching through DTensor, which costs about ten
dispatches per linear and copies the activation inside every functional all-reduce
(halo/distributed/tensor_parallel/local_linears.py). The parameters stay DTensors in the same
placements, so FSDP2, checkpoints, export, the optimizer, the clip and the weight sync see the same
layout. Each style runs the collectives its DTensor redistributes ran, on the same values and group:
The HF-native load installs this forward on every planned nn.Linear at the load finish
(install_planned_local_forwards, called from finish_hf_tp_load); the selective plan shards and installs
it in one step, and apply_lowp_compute re-installs it around the dense MLP projections lowp_precision
retypes, computing their fake-quant matmul on the local shards. Embeddings (embedding_rowwise), the MoE
and activation styles, and a Linear subclass with a forward of its own keep transformers' forward. A
projection called outside its FSDP unit's forward, with FSDP2's (dp, tp) shard still registered, raises
rather than computing part of its output.
The results are bit-identical to DTensor dispatch with one exception: a rowwise bias is added once after
the SUM, in the matmul's dtype, where DTensor added bias / tp_size to each rank's partial, so GPT-OSS's
biased o_proj differs in the last bit (tests/cpu/parallelism/test_tp_local_linears.py: every TP family
of the tiny roster at TP 2, both plans under FSDP2 at dp 2 × tp 2 and under activation checkpointing).
Against DTensor dispatch on 8× B300 at s4096 (1.2.0, three rounds), Qwen3-8B TP2 runs 57.5% faster with the step-1 loss bitwise and gpt-oss-20b ep8 × tp2 11.0% faster.
Quick start
Bash
# Pure TP=4 on 4 GPUs (no data parallelism)
torchrun --nproc_per_node=4 scripts/training/sft.py \
examples/sft/qwen3/qwen3-4b-ultrachat.yaml \
--tensor_parallel_size=4--tensor_parallel_size defaults to 1 (TP off) and is the only TP flag — there is no
--tensor_parallel_plan. Dense TP always passes distributed_config=DistributedConfig(tp_plan="auto");
tp_plan and device_map are mutually exclusive, so do not set device_map.
TP + data parallelism
2D (dp, tp) mesh: TP within a node, FSDP2 across replicas. DTensor gradients cannot be reduced
with plain dist.all_reduce or DDP — FSDP2's fully_shard understands DTensor semantics. The
loader builds the mesh and attaches it to the model; the trainer wraps over the DP dimension via
apply_fsdp2_for_tp (halo/distributed/fsdp.py), reusing that mesh. EP+TP takes the same entry
point with its EP modules as ignored_params. Pure TP (DP=1)
skips FSDP. Prefer tp_size = world_size when the model fits the TP group's memory.
Gradient sync and the grad norm
A TP run holds four kinds of parameter, and each needs a different reduction. Getting the kind wrong is silent — the loss stays finite and the objective is just different:
The trap is the plain slice. It is indistinguishable from a replica by tensor type, and once FSDP2
wraps the run for DP both become 1-D dp DTensors, so the mesh cannot separate them either.
_tp_sharded_plain_param_ids() (halo/trainers/parallelism/grad_sync.py) is the single source of truth:
it reads the hand-sliced registry (hand_sliced_suffixes: the suffixes register_hand_sliced
records, each with its shard dim, when a param is sliced by hand), and both
_sync_tp_replicated_grads and _sharded_grad_bucket key off it. Everything a TP plan shards is a DTensor whose gradient reduces itself, so the plan contributes
no entries here.
Classifying a slice as a replica costs twice: the TP-group average mixes another rank's heads into
every slice's gradient, and the norm counts only the local slice, making the global norm
rank-dependent so TP peers clip the same parameter by different coefficients. Guarded by
tests/gpu/parallelism/tp/test_tp_correctness.py and test_tp_dp_correctness.py, both against an
unsharded single-GPU grad norm.
EP+TP mode
EP+TP shards attention with TP and distributes MoE experts with EP on the same GPU groups
(load_ep_tp_model). Router, embed_tokens/lm_head, and norms stay replicated. ep_size must be
a multiple of tp_size, TP stays NVLink-local, and DP = world_size / tp_size. Its topology rules:
Topology rules; example shapes and the gradient-sync groups:
multi-node.md.
For MoE where only expert memory is the bottleneck, shard the expert FFN with ETP instead. Attention TP and expert TP are mutually exclusive.
Numerics. EP+TP in bf16 carries a deterministic offset against an unsplit reference: TP reorders the attention reductions feeding the router, which flips a handful of near-tied top-k expert picks and moves whole-expert mass. Gradient scale and direction still match; this is bf16 reduction order, not a sync bug.
Measured 0.060 loss offset on gpt-oss-20b (top-4-of-32, before 1.2.0), so the correctness gate uses
TOL.router_pick_flip_loss_abs (0.1, shared with the EP+ETP gate) instead of the generic
parallel-vs-baseline bound.
Limitations
Trainers. No trainer rejects TP; its trainer-level gates reject knobs (LoRA and
load_best_model_at_end below, online GRPO's rollout_func). Matrix:
Trainer Compatibility.
Models. There is no per-family "supports TP" flag. Selective TP shards exactly the attention
classes in TP_SHARDABLE_ATTENTION_CLASSES (halo/distributed/tensor_parallel/parallelize_attention.py), and
a tp_size > 1 run that shards zero layers raises, naming the model's attention classes; that
is how the unsupported families are rejected. The check runs on a meta-device shell
before any weight is read (reject_unsupported_structure), and again when the plan is applied.
A run that shards some layers (hybrid stacks — Qwen3.5/3.6 GatedDeltaNet, LFM-2 short-conv)
only warns: numerics stay correct, but the unsharded layers keep a full replica per rank, so
per-rank memory falls by far less than 1/tp_size.
Head divisibility is checked first: num_attention_heads % tp_size == 0 always,
num_key_value_heads % tp_size == 0 for non-MLA GQA. The dense HF-native tp_plan="auto" path gets
the same gate: reject_unsupported_model runs validate_tp_head_divisibility
on every load path, off config.json before any weight is read.
Transformers validates no head count of its own, so without it the run dies on the first forward's reshape, after the whole checkpoint has been pulled and placed on every rank.
Axis combinations. TP composes with EP only; the allowlist and the EP+TP topology rules: Supported combinations.
Knobs. Everything below raises unless the verdict says otherwise.
Multi-node
TP requires an NVLink-bandwidth all-reduce after every layer, so it must stay within a domain;
cross-node TP over InfiniBand would bottleneck every pass. The pattern is TP within nodes, FSDP
across nodes. nvlink_domain_size % tp_size == 0 is enforced at config time.
Bash
# Each node (set --node_rank per node)
torchrun --nnodes=2 --node_rank=0 --nproc_per_node=4 \
--master_addr=$MASTER_ADDR --master_port=$MASTER_PORT \
scripts/training/sft.py <config>.yaml --tensor_parallel_size=4The NCCL watchdog (DIST_NCCL_TIMEOUT_MINUTES) covers the TP mesh too:
Launch Recipes → Environment variables.
Checkpoint saving
save_tp_checkpoint (halo/checkpoint/parallel/save.py), the ladder's TP rung, always
gathers: all TP ranks reconstruct full tensors and one save rank writes a standard HF checkpoint, put
through normalize_gathered_state_dict (hub expert layout, save dtype) like the FSDP2 and CP
writers. There is no per-rank TP save. The write streams through stream_gathered_checkpoint, so
the save rank's host RAM is bounded by one chunk plus the shards it is filling and writing
(Checkpoints) rather than the whole model.
chunked_saveable_tensors reconstructs every sharded param with DTensor.full_tensor() — one walk
covers both mechanisms — and gather_hand_sliced_params (halo/checkpoint/parallel/writing.py)
all-gathers the hand-sliced GptOss sinks after it. Both are collective: every TP-mesh rank
must drive them or the gather hangs.
On resume every TP shape is Path B: the training scripts repoint model_name_or_path at the
checkpoint, so the weights load at construction and load_resume_weights skips the re-read.
Where it does read (a best-model reload, or a model built from elsewhere), each rank streams the
checkpoint's full tensors and distribute_tensors them into the live DTensor placements.
Configured FP32 masters are restored at construction, before the DP wrap (Load precision), so the constructed-from-checkpoint skip keeps their precision without a reload into a 2-D packed FSDP2 layout.
TP+DP is the exception: FSDP2 over TP stacks a strided dp shard on the tp shard, a 2-D placement
distribute_tensor does not invert for packed projections, so it refuses every reload but the
constructed-from-checkpoint skip. See
Checkpoints.
Troubleshooting
- "tp_plan and device_map are mutually exclusive" — drop
device_map. - Quiet non-main ranks — expected: transformers drops to ERROR off the main rank, and the script logger follows
log_level_replica(HF defaultwarning). - DTensor all_reduce error with DDP — DTensors cannot be used with DDP; use FSDP2.
- OOM with TP + FSDP — enable gradient checkpointing.
Source on GitHubUpdated 18 November 2026