Nemotron 3.5 Super VL Cookbook
NVIDIA's Nemotron 3.5 Super VL interleaves 40 Mamba-2, 40 MoE and 8 attention layers, 120B parameters in all, with a C-RADIO vision tower. Each MoE layer routes 512 experts at top-22 on a 1024-wide latent.
Support
Modes: the Nemotron-H row of the Supported Matrix; online RL: Rollout engines.
- Checkpoint:
nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16, loaded through transformers' own classes with no conversion step. - GPUs: EP8 on eight B300s, 64 experts per GPU.
- Lightning:
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16is the same text class with no vision tower. Its recipe,examples/sft/nemotron_h/nemotron-3.5-lightning-ep8-full.yaml, ran 20 steps at EP8 and 8,192 tokens at 9,986 tok/s per GPU and 55–60 GB reserved. A full-depth export serves as saved (Lightning ↗). The rest of this page is Super VL. - What ran. All four recipes and the image recipe ran on the release on eight B300s, 5 to 20 steps each, plus an exact resume at a 9-layer cut and exports byte for byte at full depth. No run has trained to a quality metric. Both engines have served an untrained reattached export, on one image prompt and one text prompt each.
A B300 reports 288 GB, and the top GPU reached 99.4% at 32,000 and 99% at 50,000, so treat those as the limit. Each
max_length is a starting point: watch the first steps' peak.
Recipes
Run these in the training container. All four load the text tower
with text_only_model: true.
Full fine-tune
Bash
halo launch sft examples/sft/nemotron_h/nemotron-3.5-super-vl-ep8-full.yaml -n 8 \
--output_dir=/data/checkpoints/nemotron-sft--max_length=32000 also ran. Freezing the experts with freeze_layers_patterns on *.mixer.up_proj and
*.mixer.down_proj reached --max_length=50000. Each save writes about 241 GB, and a full resume checkpoint comes to
about 725 GB by arithmetic.
LoRA
Bash
halo launch sft examples/sft/nemotron_h/nemotron-3.5-super-vl-ep8-lora.yaml -n 8 \
--output_dir=/data/checkpoints/nemotron-loraThese targets adapt attention, the Mamba projections and the experts. Every save folds the adapters into a full
checkpoint, 251 GB with its adapter state, so raise save_steps when disk is tight.
SMPO
Bash
halo launch smpo examples/preference/nemotron_h/smpo-nemotron-3.5-super-vl-tulu3-prefmix-ep8.yaml -n 8 \
--output_dir=/data/checkpoints/nemotron-smpoOffline GRPO
Copy the recipe and point dataset at your scored completions
(Offline GRPO):
Bash
cp examples/grpo/offline/nemotron_h/offline-grpo-nemotron-3.5-super-vl-gsm8k-ep8.yaml nemotron-grpo.yaml
halo launch grpo/offline nemotron-grpo.yaml -n 8 --output_dir=/data/checkpoints/nemotron-grpoImages
Copy the full fine-tune recipe, delete text_only_model and packing, and point dataset at rows with images
(SFT). The composite computes its loss on the plain head, which holds more
per token.
Image and text rows can mix under torchrun: half-image rows ran at EP8 on the release with no hang. Training the
vision tower on mixed rows rests on tiny-model tests only.
With Liger on, a training step logs mean_token_accuracy 0 and entropy ln(vocab), because Liger writes its gradient
into the fp32 logits the forward returns. The loss, the gradients and eval metrics that mask label -100 are right; a
compute_metrics that reads logits at ignored positions sees zeros in eval too.
Settings that matter
- Gradient checkpointing stays on. By arithmetic a token holds about 66 MB without it, so a GPU fits about 1,650 tokens per rank. It costs 37% on a Mamba layer.
- Speed. The Mamba scan dominates, at 64.8% of the tower's forward and backward at EP8, and throughput stays flat with length. Rows of many short documents scan slower: 64 × 128 costs 1.9× one 8,192-token document on a Mamba layer.
- Attention. Leave
attn_implementationunset: FA4 runs both towers on B300, with FA2 as the fallback. - Pad token.
pad_token: <unk>keeps the export's config equal to the release's. The tokenizer's own pad is its eos. - Chat template. The hub template renders an assistant turn with no reasoning as
<think></think>plus its text, so the empty block trains as part of the completion. - Router balancing.
autopicksbias_update, which trains the router's own bias and ships it in every checkpoint. The bias stays fp32 and routes with balancing off as well. - LoRA.
lora_rmust be a multiple of 8. An adapter onconv1dis refused.
Limits
- No CP, TP or HSDP: the Mamba scan runs along the sequence axis, and attention is 0.24% of the parameters.
- No tree packing.
- No online RL. Both engines serve the release, but no update into either is validated, so Halo refuses weight sync at startup. Offline GRPO needs no server: score completions from a served export and train them with the recipe above.
- DPO and KTO take text rows, a full fine-tune with
precompute_ref_log_probs: true, and refuse images. - No video or audio rows. They raise.
trust_remote_code: trueon the composite is refused at the load: the hub's modeling file hands its text tower no document markers, so leave it unset.text_only_model: truebuilds transformers' class either way.- Pipeline parallelism takes the text tower only and passes tiny-model GPU tests only.
save_sharded_epandmerge-ep-shardsare refused. Gathered saves stream one layer at a time.merge-peft-adaptersstreams the merge of expert adapters;merge_expert_lora_on_savefolds them at training time instead.
Export and serve
Smoke-test a text-only export with AutoModelForCausalLM (snippet). Stock
transformers clamps Mamba's time steps at a floor neither the run nor the engines use, so its loss on real text reads
0.05–0.09 nats higher than under the run's limit. To serve the export, rebuild the composite, which also streams back the
vision tower and, unless the run trained one with mtp_loss_weight, the mtp.* head at the release's weights:
Bash
halo run reattach-vision-tower --input_dir /data/checkpoints/nemotron-sft \
--model_id nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 \
--output_dir /data/checkpoints/nemotron-sft-vlServe from the host. vLLM started this way needs --trust-remote-code and
--mamba-ssm-cache-dtype float32, which docker-compose.vllm.yml takes as VLLM_TRUST_REMOTE_CODE=1 and
VLLM_EXTRA_ARGS:
Bash
docker run -d --rm --name nemotron-vllm --gpus '"device=0,1"' --ipc=host --network host \
-v "$HF_HOME":/root/.cache/huggingface -v "$HALO_SCRATCH":/data:ro "$VLLM_IMAGE" \
/data/checkpoints/nemotron-sft-vl --port 8000 --trust-remote-code --tensor-parallel-size 2 \
--mamba-ssm-cache-dtype float32 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
--reasoning-parser nemotron_v3SGLang's compose file on port 30000 takes:
Bash
SGLANG_MODEL="$HALO_SCRATCH/checkpoints/nemotron-sft-vl" SGLANG_CUDA_DEVICES=0,1 SGLANG_TP=2 \
SGLANG_TRUST_REMOTE_CODE=1 SGLANG_TOOL_PARSER=qwen3_coder SGLANG_REASONING_PARSER=nemotron_3 \
docker compose -f docker-compose.sglang.yml up -d sglang-serverDon't copy the model card's --mamba-ssm-dtype flags onto this image: they are for the card's dedicated image.
Reference
- Nemotron-H model notes ↗: the Mamba scan, the memory model, the export layouts
- Offline GRPO, Preference
- Model card
Source on GitHubUpdated 18 November 2026