Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
A small classifier reads each message and replies 'safe', or 'unsafe' plus category codes. Your code decides what those codes mean for this product, and what happens if the classifier itself fails.
Moderates input and output with a dedicated classifier, and fails closed whenever actions are at stake.
1.2Key concepts
One idea per card, in plain words.
A safety model that labels text using a harm taxonomy such as S1–S14.
Check the user's message before the model, and the reply after it.
A medical product may allow health topics that a children's product blocks. The product, not the classifier, decides.
If the classifier errors, block anything that takes an action.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Add a moderation gate in front of your assistant. Parse the classifier's output, apply a per-product allow list, and define behaviour when the classifier is down.
2.2Clarify first
Each answer removes a guess.
| You ask | Answer |
|---|---|
| Can the assistant take actions? | Yes, it books and cancels. That decides the failure mode. |
| What latency budget? | About 50 ms on input. Stream output through a lighter check. |
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Classify the input
Send the message in the model's expected prompt format.
- Parse strictly
First line 'safe' or 'unsafe'; second line holds category codes. Treat anything else as an error.
- Apply product policy
Remove categories this product allows, then block on what remains.
- Choose failure modes
Fail closed for actions. A read-only FAQ may fail open with logging.
- Measure both errors
Track missed harms and over-blocking, per category.
3.2Code
Working code for the core of it.
A safety gate before the model
It runs before the expensive model, and has to decide what happens if it fails.
1export interface Verdict {2 allow: boolean;3 categories: string[];4 reason: "safe" | "unsafe" | "blocked-category" | "classifier-error";5}6 7/** Llama Guard replies "safe" or "unsafe" followed by category codes such as "S1,S10". */8export function parseGuardOutput(text: string): { safe: boolean; categories: string[] } {9 const [first, second = ""] = text.trim().split("\n");10 const safe = first.trim().toLowerCase() === "safe";11 return { safe, categories: safe ? [] : second.split(",").map((c) => c.trim()).filter(Boolean) };12}13 14/** Runs before the main model. allowed lists categories this product accepts despite the flag. */15export async function gate(input: string, classify: (text: string) => Promise<string>, allowed: string[] = [], failOpen = false): Promise<Verdict> {16 try {17 const { safe, categories } = parseGuardOutput(await classify(input));18 if (safe) return { allow: true, categories: [], reason: "safe" };19 const blocked = categories.filter((c) => !allowed.includes(c));20 if (categories.length > 0 && blocked.length === 0) return { allow: true, categories, reason: "safe" };21 return { allow: false, categories: blocked, reason: blocked.length ? "blocked-category" : "unsafe" };22 } catch {23 // Actions fail closed. A read-only product can choose to fail open.24 return { allow: failOpen, categories: [], reason: "classifier-error" };25 }26}parseGuardOutput- Llama Guard replies safe, or unsafe plus category codes.
allowed- A medical product may accept a category that a children's product blocks.
failOpen = false- If the classifier errors, products that take actions should refuse.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Does this stop prompt injection?
Why a separate model?
How do you handle false positives?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Output streaming
- Check complete sentences as they stream, or buffer sensitive flows.
- Counterpoint
- Classifiers miss new attacks and block innocent text. Treat them as one layer, never the whole defence.