To fine-tune Gemma 4 with Halo, train google/gemma-4-26B-A4B-it on eight GPUs with expert parallelism of size 8 (EP8), the SDPA attention backend and no router balancing. Context and tensor parallelism are not supported for this family.
Steps
- Start the training container. Set up the host as installation describes, then start the cookbook container. Run the commands below inside it. On H100 or H200, use the Hopper image. The training config does not change.
- Pick the checkpoint.
google/gemma-4-26B-A4B-itis a mixture-of-experts model with 128 routed experts, eight selected for each token. It can process text and images. The recipe uses text data. - Write the config. Create
gemma4-sft.yamlfor supervised fine-tuning of all weights, or start from the shippedexamples/sft/gemma4/gemma4-26b-a4b-ultrachat-ep.yaml. Among its keys:YAML
model_name_or_path: google/gemma-4-26B-A4B-it moe_balancing: none expert_parallel_size: 8 attn_implementation: sdpa packing: true max_length: 32768 per_device_train_batch_size: 1 - Launch eight processes. The recipe uses eight NVIDIA B300 GPUs. With expert parallelism of size 8 (EP8), each GPU holds 16 experts.
Bash
halo launch sft gemma4-sft.yaml -n 8 - Load the result. Use
AutoModelForImageTextToTextandAutoProcessor. The same model class and processor handle image-and-text inference.
Common mistakes
- Another attention backend. Keep
attn_implementation: sdpa(scaled dot-product attention). It is the supported backend for this checkpoint. - Router balancing. Gemma 4 has no supported auxiliary-loss or bias-update balancing path, so keep
moe_balancing: none. - A larger batch. Keep
per_device_train_batch_sizeat 1 with packing and scale throughgradient_accumulation_steps. Packing builds a dense mask here, and its memory grows with the square of batch size timesmax_length.
With Halo
Halo does not support context or tensor parallelism for Gemma 4. The Gemma 4 cookbook has the full config, expert-tensor parallelism, a LoRA adapter for the language model, and reinforcement learning with GRPO.
