How to fine-tune Gemma 4

To fine-tune Gemma 4 with Halo, train google/gemma-4-26B-A4B-it on eight GPUs with expert parallelism of size 8 (EP8), the SDPA attention backend and no router balancing. Context and tensor parallelism are not supported for this family.

Steps

  1. Start the training container. Set up the host as installation describes, then start the cookbook container. Run the commands below inside it. On H100 or H200, use the Hopper image. The training config does not change.
  2. Pick the checkpoint. google/gemma-4-26B-A4B-it is a mixture-of-experts model with 128 routed experts, eight selected for each token. It can process text and images. The recipe uses text data.
  3. Write the config. Create gemma4-sft.yaml for supervised fine-tuning of all weights, or start from the shipped examples/sft/gemma4/gemma4-26b-a4b-ultrachat-ep.yaml. Among its keys:

    YAML

    model_name_or_path: google/gemma-4-26B-A4B-it
    moe_balancing: none
    expert_parallel_size: 8
    attn_implementation: sdpa
    packing: true
    max_length: 32768
    per_device_train_batch_size: 1
  4. Launch eight processes. The recipe uses eight NVIDIA B300 GPUs. With expert parallelism of size 8 (EP8), each GPU holds 16 experts.

    Bash

    halo launch sft gemma4-sft.yaml -n 8
  5. Load the result. Use AutoModelForImageTextToText and AutoProcessor. The same model class and processor handle image-and-text inference.

Common mistakes

  • Another attention backend. Keep attn_implementation: sdpa (scaled dot-product attention). It is the supported backend for this checkpoint.
  • Router balancing. Gemma 4 has no supported auxiliary-loss or bias-update balancing path, so keep moe_balancing: none.
  • A larger batch. Keep per_device_train_batch_size at 1 with packing and scale through gradient_accumulation_steps. Packing builds a dense mask here, and its memory grows with the square of batch size times max_length.

With Halo

Halo does not support context or tensor parallelism for Gemma 4. The Gemma 4 cookbook has the full config, expert-tensor parallelism, a LoRA adapter for the language model, and reinforcement learning with GRPO.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL