Embedding Models
Fine-tune a text encoder for retrieval, semantic similarity, deduplication or clustering, using
sentence-transformers losses under Halo's parallelism. The output loads with
SentenceTransformer(path).
Data
The collator reads columns by position, not by name: the label is the first of label, labels,
score, scores that exists, and every other column is one text input. The table's shape picks the
loss: anchor-positive pairs train mnrl (the default) or cached_mnrl, a third negative column adds
triplet, a float score trains cosent or angle, and a 0/1 label the contrastive losses
(shapes and losses ↗). Embedding
training is text-only.
Run
Bash
halo launch embedding examples/embedding/qwen3/embedding-qwen3-4b-nq.yaml -n 8Recipes for Qwen3-Embedding, Qwen3.5, GPT-OSS and Gemma 4 ship under examples/embedding/.
mnrltakes its negatives from the other rows of the same micro-batch, soper_device_train_batch_sizeis the dominant hyperparameter; gradient accumulation and extra GPUs do not enlarge the pool. Switch tocached_mnrlwhen a larger batch runs out of memory.pooling_modemust match the backbone:lasttokenfor decoder-based embedders like Qwen3-Embedding,meanfor most encoders.batch_sampler: no_duplicateskeeps two copies of one text out of a batch, where they would become each other's false negatives. Tensor and expert-tensor parallelism take only the plainbatch_sampler.
sentence-transformers owns tokenization and the pipeline around the backbone, so the chat-template and
special-token knobs, the freeze patterns, tools_field, log_decoded_samples, text_only_model and
tokenizer_backend: gigatoken raise at startup. Context and pipeline parallelism are refused, and LoRA
runs on plain data parallelism only, on a MoE at use_grouped_gemm: false. Under expert or tensor
parallelism, and on a MoE at the default use_grouped_gemm, a checkpoint whose pipeline carries a module
past pooling and normalization (a Dense head) is refused.
What to watch
The embed/* metrics read the sentence embeddings of the loss forward, on logging steps and on every
eval batch.
embed/cos_simshould rise, andembed/neg_cos_simfall on a dataset with a third text column.embed/recall@1andembed/mrrare the ranking view of the same thing.- Collapse shows up two ways:
embed/stdtrending to zero, or a highembed/cos_simwith poorembed/recall@1. embed/mrrsaturating at 1.0 within a few steps usually means the batch is too small to be a real ranking task.
Every knob, the save and resume rules and the limits per parallelism mode: Embedding reference ↗.
Source on GitHubUpdated 18 November 2026