Evaluate AI guardrails on your own traffic for four things: how much harm they catch, how often they block honest requests, how much latency they add, and whether they hold up against adversarial prompts.
What to measure
- Recall. Of the requests that break a policy, how many are caught.
- Precision. Of the requests blocked, how many deserved it. The rest are false positives.
- Latency. Added time at the median and at the 95th percentile, on your payload sizes: see guardrail latency.
- Robustness. Whether recall survives jailbreaks, paraphrase and other languages.
Report recall at a fixed false-positive rate. A guard that blocks everything has perfect recall.
Steps
- Collect a few hundred real conversations from your product, good and bad.
- Label them against your own policies, not a vendor's categories.
- Add adversarial versions of the bad ones: see AI red teaming.
- Run each candidate with the same policies and the same data.
- Measure latency at your real traffic shape, not on single short prompts.
- Read the mistakes. Twenty wrong verdicts teach more than the headline score.
What public benchmarks tell you
A public benchmark compares guard models on shared categories and says nothing about your policies. Use it to shortlist. CircleGuardBench scores moderation models on accuracy, on resistance to jailbreaks and on speed together, since a guard that is accurate but slow cannot sit in the request path.
With White Circle
Control lets you test a policy against past traffic before it is live, and Red Team attacks your setup to find what gets through.
