AI content moderation

Content moderation in an AI system detects unsafe content in what users send and in what the model writes, such as hate, harassment, sexual content, violence and self-harm, and stops or flags it.

What it checks

Moderation runs on both sides of the model: on user messages before the model sees them, and on replies before the user does. Common categories:

  • Hate and harassment
  • Sexual content, including anything involving minors
  • Violence and threats
  • Self-harm
  • Instructions for illegal or dangerous acts

The categories and their limits depend on the product. A medical assistant has to discuss what a children's tutor must refuse, so each limit is written down as a policy.

Why keyword lists fail

A block list matches strings, not meaning. It misses misspellings, slang, code words and paraphrase, and it stops harmless uses of a listed word, such as a clinical term or a place name. Whether a sentence is a threat or a line from a novel depends on context. So moderation uses classifiers: models trained to judge what a text means (guard models).

Multilingual content

A classifier trained mostly on English is weaker in other languages, and translating a request is a known way around a filter. Test moderation in every language your users write.

With White Circle

Control enforces your policies, built-in or custom, on every input and output in real time. We support 150+ languages and multimodal content.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL