Openai Moderation Scores Safety Routing Poc 2026

Slug: openai-moderation-scores-safety-routing-poc-2026 Run date: 2026-07-03 Track: api-backed-poc Evidence level: openai-api-backed-lab Runner: Effloow Lab (Claude runtime)

What we tested

OpenAI added inline moderation scores to the Responses API and Chat Completions API on 2026-06-04 (pass a moderation object, get response.moderation.input / response.moderation.output back in the same call). The underlying classifier is the same one behind the free /v1/moderations endpoint (omni-moderation-latest).

We built a tiny safety-routing gate: score a message, then route it to allow or block based on (a) the boolean flagged field and (b) a 0.50 score threshold on the top category. We measured routing precision/recall against a synthetic labeled set.

Environment

• Endpoint: POST https://api.openai.com/v1/moderations
• Model: omni-moderation-latest
• Cost: free — the moderation endpoint does not count toward usage limits (OpenAI Help Center: "Is the Moderation endpoint free to use?" — yes).
• Script: scripts/moderation-lab-run.py (stdlib only, no SDK, never prints the key)
• Artifact: data/lab-runs/openai-moderation-scores-safety-routing-poc-2026.openai.json

Commands

python3 scripts/moderation-lab-run.py

Labeled set (10 synthetic items)

7 safe, 3 unsafe. All invented; no real personal data. The unsafe items are mild synthetic examples covering harassment, threatening violence, and self-harm intent. Two of the "safe" items are deliberate false-positive traps:

• A message complaining that a customer was "difficult and rude" (should NOT block).
• A figurative-violence idiom ("I could kill for a good coffee") (should NOT block).

Results (measured on THIS set only)

Routing metrics (both flagged boolean and 0.50 threshold):

• TP 3, FP 0, FN 0, TN 7
• Precision 1.00, Recall 1.00 (on this 10-item set)

Separation margin: highest safe score = 0.1842; lowest unsafe score = 0.8666. On this set any threshold in roughly (0.19, 0.86) separates the two classes perfectly. Both traps stayed well under threshold (figurative violence 0.1842, rude-customer 0.0095).

Limitations

• 10 items is a demonstration, not a benchmark. Precision/recall here do not generalize to real user traffic. Real false-positive and false-negative rates are [DATA NOT AVAILABLE] until measured on production data.
• The clean margin is a property of these easy examples. Adversarial or subtly-worded content would narrow it; do not read "1.00" as a production accuracy claim.
• We tested the standalone /v1/moderations endpoint. The inline moderation object on the Responses/Chat APIs uses the same classifier per OpenAI docs, but we did not separately measure any latency or cost delta from enabling it inline — the changelog states no cost/latency impact, which we treat as vendor-stated, not measured.
• We did not test image moderation (omni-moderation supports images; out of scope here).