# How to fine-tune Laguna 2.1

To fine-tune Laguna 2.1 with Halo, train poolside/Laguna-S-2.1 on exactly four GPUs with expert parallelism of size 4 (EP4) and the SDPA attention backend. Laguna XS 2.1 trains on one GPU as a single process.

## Steps

1. **Start the training container.** Set up the host as [installation](/halo/docs/installation) describes, then start the [cookbook container](/halo/docs/cookbooks#start-the-training-container). Run the commands below inside it.
2. **Pick the checkpoint.** `poolside/Laguna-S-2.1` starts on four GPUs with [expert parallelism](/glossary/expert-parallelism) of size 4 (EP4). `poolside/Laguna-XS-2.1` starts on one B300 as a single process.
3. **Write the config.** Create `laguna-s-2.1-sft.yaml` for supervised fine-tuning of all weights, or use the shipped `examples/sft/laguna/laguna-s-2.1-ultrachat-ep.yaml`. Among its keys:
   ```yaml
   model_name_or_path: poolside/Laguna-S-2.1
   trust_remote_code: true
   pad_token: "〈|PAD|〉"
   eos_token: "〈|EOS|〉"
   expert_parallel_size: 4
   attn_implementation: sdpa
   ```
4. **Launch four processes.** On four NVIDIA B300 GPUs, EP4 places 64 of the 256 routed experts on each.
   ```bash
   halo launch sft laguna-s-2.1-sft.yaml -n 4
   ```
5. **Train Laguna XS instead.** Copy the config, change the model and output directory, drop `expert_parallel_size`, and launch with `-n 1`.

## Common mistakes

- **Retyping the tokens in ASCII.** `pad_token` and `eos_token` use the CJK angle brackets U+3008 and U+3009. ASCII `<` and `>` silently add new tokens instead of resolving the existing ones.
- **Eight GPUs.** The same config on eight GPUs is rejected at config time. Run it on exactly four.
- **Another attention backend.** `attn_implementation: sdpa` (scaled dot-product attention) is required. The hub revision the cookbook pins has no [Flash Attention](/glossary/flash-attention) path.

## With Halo

Halo does not support context or tensor parallelism for Laguna, a [mixture-of-experts](/glossary/mixture-of-experts) family. Expert-tensor parallelism is tested only on a tiny model, so validate a short run first. The [Laguna 2.1 cookbook](/halo/docs/cookbooks/halo-laguna-2.1-cookbook) has the full config, the Laguna XS run, an expert [LoRA](/glossary/lora) adapter, and [GRPO](/glossary/grpo).

