# Classification

Sequence classification — single-label (multi-class) and multi-label — on `AutoModelForSequenceClassification`. Trainer `ClassificationTrainer` (`halo/trainers/reward/classification.py`), script `scripts/training/classification.py`. Text-only. For a scalar preference score on the same architecture use [Reward Modeling](https://whitecircle.com/halo/docs/agent-docs/training-methods/reward-modeling.md).

EP, TP, ETP, EP+TP, EP+ETP, PP, PP+EP and PP+ETP run, as does plain FSDP2 data parallelism; under PP `compute_metrics` is served. CP does not — the head pools the whole sequence ([matrix](https://whitecircle.com/halo/docs/agent-docs/reference/trainer-architecture.md#trainer-compatibility)).

## Dataset

A `prompt` conversation plus a `label` column: a string for single-label, a list for multi-label. `text_field` names a raw text column instead, wrapped as one user turn. A dataset with neither is rejected right after the dataset load. Row shapes: [Dataset Formats](https://whitecircle.com/halo/docs/agent-docs/data/dataset-formats.md#classification).

Rows are chat-templated and **truncated** to `max_length` — a label describes the whole document, so a shortened document keeps a valid label. A dataset handed to `ClassificationTrainer` directly (no `input_ids` column) is instead tokenized untruncated and its over-length rows dropped, with the fraction logged.

The label set comes from the training split: stringified, sorted for deterministic ids, and unioned across ranks on a pre-sharded dataset so every rank builds an identically shaped head. `num_labels`, `label2id` and `id2label` follow from it, never from YAML; single- vs multi-label follows whether `label` is a list. Labels seen only in validation or test are added with a warning.

`ds["test"]` is the eval split; a source with no `test` split evaluates on its `validation` split, and next to a `test` split `validation` contributes labels only.

`-1` marks an unlabeled row. A multi-label row treats it as absence and it is dropped from the label set. A single-label split carrying it is refused before the model load when the run reads that split (train always; the eval split when the run evaluates: `eval_strategy` other than `no`, or `eval_on_start`), since the row has no class and the loss would read `-1` as an out-of-range index. A split the run reads with no `label` column at all is refused the same way, by name; one it does not read adds no labels. A split the run does not read is not tokenized. Filter unlabeled rows out of train; for an unlabeled eval split (a GLUE-style `test`), set `dataset: <id>@train` and cut the eval split with `test_size`, or set `eval_strategy: no`.

## Configuration

```yaml
model_name_or_path: Qwen/Qwen3.5-9B
dataset: yaful/MAGE
text_field: text
test_size: 0.1
attn_implementation: sdpa
max_length: 2048
per_device_train_batch_size: 4
gradient_accumulation_steps: 8
learning_rate: 2.0e-05
num_train_epochs: 1
gradient_checkpointing: true
output_dir: checkpoints/clf-qwen3.5-9b-mage
```

`loss_type` picks `cross_entropy`, `focal` or `label_smoothing_ce`, and refuses the other losses' knobs; every field's type and default, and which apply to single- or multi-label heads, is in [ClassificationConfig](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#classificationconfig).

`class_weights` and `derive_class_weights` are mutually exclusive — setting both raises, on the CLI as well as in YAML. Every objective, the default cross-entropy included, is the trainer's own `ClassificationLoss` (`halo/trainers/reward/classification_loss.py`), evaluated in fp32; the head's built-in loss would run in the logits' bf16. A single-label head needs `num_labels >= 2` — one logit is transformers' regression convention — and is refused at construction otherwise. HF's own `label_smoothing_factor` is rejected at construction: the trainer builds its own loss and never reaches HF's `label_smoother`.

Batches are right-padded, so an unpinned `attn_implementation` takes the padded-batch default: the loader's own pick on Blackwell (SDPA for a remote-code checkpoint, FA2 for a softcapped policy), SDPA elsewhere ([Flash Attention](https://whitecircle.com/halo/docs/agent-docs/optimization/flash-attention.md#model-specific-handling)). Under `reset_sinks: false` it requests nothing and lets the resolver auto-select, which there accepts only a sink-carrying backend.

## Launch

```bash
# MoE, expert parallel
torchrun --nproc_per_node=8 scripts/training/classification.py \
    examples/classification/gptoss/clf-gptoss-20b-mage-ep.yaml --expert_parallel_size=8

# dense, plain FSDP2 data parallel
halo launch classification examples/classification/qwen3_5/clf-qwen3.5-9b-mage.yaml --nproc 8
```

LoRA takes `lora_task_type: SEQ_CLS` and is rejected under TP, EP+TP and PP ([PEFT](https://whitecircle.com/halo/docs/agent-docs/optimization/peft.md#parallelism-compatibility)).

## Vision-language

Not supported. The script loads through the text-only `load_model_and_tokenizer`, builds no processor, and refuses `text_only_model` as well as any image column.

A multimodal checkpoint is also checked for a sequence-classification head before the load and refused without one. transformers 5.20 ships that head for a few families and the toolkit registers two more ([roster](https://whitecircle.com/halo/docs/agent-docs/training-methods/reward-modeling.md#vision-language)).

## Testing a setup

```bash
torchrun --nproc_per_node=2 scripts/training/classification.py <config> \
    --max_steps=5 --save_strategy=no --report_to=none
```

Covering tests: `pytest tests/cpu/trainers tests/cpu/config -m cpu` and `tests/gpu/trainers/other/test_classification.py`; under EP, TP and ETP with save, exact resume and export, `tests/gpu/parallelism/ep/test_ep_pooled_head_trainers.py --trainer classification`; under PP, `tests/gpu/parallelism/pp/test_pp_classification.py` and `test_pp_trainer_checkpoints.py --trainer classification`.

## What to watch

| Signal | Reading |
|---|---|
| `accuracy` | Single-label; multi-label reports `exact_match_accuracy` instead |
| `f1` | Binary averaging at 2 labels, weighted for multi-class, macro for multi-label |
| `mcc` | Single-label (`compute_mcc`); stays honest under class imbalance |

Failure signatures:

- `metric_for_best_model: auc_roc` raises — either `compute_auc_roc` is off, or an eval slice holds a class with no samples and the metric is undefined there. Rank on `accuracy`, `f1` or `mcc`.
- Documents silently shortened: `max_length` truncates rather than drops. Raise it when the label depends on the tail.

## Related pages

- [Reward Modeling](https://whitecircle.com/halo/docs/agent-docs/training-methods/reward-modeling.md) · [SFT](https://whitecircle.com/halo/docs/agent-docs/training-methods/sft.md)
- [Expert Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/expert-parallelism.md) · [Pipeline Parallelism](https://whitecircle.com/halo/docs/agent-docs/parallelism/pipeline-parallelism.md)
- [ClassificationConfig Reference](https://whitecircle.com/halo/docs/agent-docs/reference/configuration-reference.md#classificationconfig) · [Scripts Reference](https://whitecircle.com/halo/docs/agent-docs/reference/scripts-reference.md)
