# How to fine-tune Gemma 4

To fine-tune Gemma 4 with Halo, train google/gemma-4-26B-A4B-it on eight GPUs with expert parallelism of size 8 (EP8), the SDPA attention backend and no router balancing. Context and tensor parallelism are not supported for this family.

## Steps

1. **Start the training container.** Set up the host as [installation](/halo/docs/installation) describes, then start the [cookbook container](/halo/docs/cookbooks#start-the-training-container). Run the commands below inside it. On H100 or H200, use the Hopper image. The training config does not change.
2. **Pick the checkpoint.** `google/gemma-4-26B-A4B-it` is a [mixture-of-experts](/glossary/mixture-of-experts) model with 128 routed experts, eight selected for each token. It can process text and images. The recipe uses text data.
3. **Write the config.** Create `gemma4-sft.yaml` for supervised fine-tuning of all weights, or start from the shipped `examples/sft/gemma4/gemma4-26b-a4b-ultrachat-ep.yaml`. Among its keys:
   ```yaml
   model_name_or_path: google/gemma-4-26B-A4B-it
   moe_balancing: none
   expert_parallel_size: 8
   attn_implementation: sdpa
   packing: true
   max_length: 32768
   per_device_train_batch_size: 1
   ```
4. **Launch eight processes.** The recipe uses eight NVIDIA B300 GPUs. With [expert parallelism](/glossary/expert-parallelism) of size 8 (EP8), each GPU holds 16 experts.
   ```bash
   halo launch sft gemma4-sft.yaml -n 8
   ```
5. **Load the result.** Use `AutoModelForImageTextToText` and `AutoProcessor`. The same model class and processor handle image-and-text inference.

## Common mistakes

- **Another attention backend.** Keep `attn_implementation: sdpa` (scaled dot-product attention). It is the supported backend for this checkpoint.
- **Router balancing.** Gemma 4 has no supported auxiliary-loss or bias-update [balancing](/glossary/moe-load-balancing) path, so keep `moe_balancing: none`.
- **A larger batch.** Keep `per_device_train_batch_size` at 1 with [packing](/glossary/sequence-packing) and scale through `gradient_accumulation_steps`. Packing builds a dense mask here, and its memory grows with the square of batch size times `max_length`.

## With Halo

Halo does not support context or tensor parallelism for Gemma 4. The [Gemma 4 cookbook](/halo/docs/cookbooks/halo-gemma4-moe-cookbook) has the full config, expert-tensor parallelism, a [LoRA](/glossary/lora) adapter for the language model, and reinforcement learning with GRPO.

