Home / Question bank / AI System Design

AI System DesignSeniorQuestion 6 of 12

Design a content moderation service that inspects LLM outputs in real time.

30-second answer

Requirements: categories and per-tenant policies, latency budget, streaming support. Architecture: a policy service holding configurable thresholds, a fast tier of lightweight classifiers inline, a slower LLM-judge tier for borderline cases, and streaming inspection on chunk/sentence boundaries with the ability to halt and replace the response. Log decisions with reasons for audit, measure precision/recall per category on labeled sets, support appeals and human review queues, and fail safely (decide fail-open vs. fail-closed per surface).

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

Architecture diagram

Sketch this on the whiteboard as you talk through the answer. See all AI architecture diagrams →

Scroll sideways to see the whole diagram.

Real-time moderation of streaming LLM outputComponents: Audit log (every decision); Human review (samples · appeals); Policy service (per-tenant thresholds); LLM (token stream); LLM gateway (proxies the stream); Segmenter (sentence windows); Tier 1 classifiers (~5 ms per window); Decision engine (allow · redact · halt); Tier 2 LLM judge (borderline windows); User (receives safe stream). Flow: LLM to LLM gateway; LLM gateway to Segmenter; Segmenter to Tier 1 classifiers; Tier 1 classifiers to Decision engine; Tier 1 classifiers to Tier 2 LLM judge (uncertain); Tier 2 LLM judge to Decision engine (score); Policy service to Decision engine (thresholds); Decision engine to User (release); Audit log to Human review.Audit logevery decisionHuman reviewsamples · appealsPolicy serviceper-tenant thresholdsLLMtoken streamLLM gatewayproxies the streamSegmentersentence windowsTier 1 classifiers~5 ms per windowDecision engineallow · redact · haltTier 2 LLM judgeborderline windowsUserreceives safe streamuncertainscorethresholdsreleaselog decision
Real-time moderation of streaming LLM output. The stream is cut into sentence windows. Fast Tier 1 classifiers score every window; only borderline windows go to the Tier 2 judge. The decision engine applies per-tenant policy, releases or replaces text, and logs every decision.

0–5 min Clarify and frame

Questions:

  • Policies: which categories (hate, harassment, sexual content, self-harm, violence, PII, regulated advice)? Per-tenant configurable?
  • Surfaces: streaming chat, batch generation, images?
  • Latency budget: e.g. ≤50 ms added p95 for non-streaming; minimal delay to first token for streaming.
  • Actions: block, redact, rewrite, warn, log only?
  • Scale: e.g. 5k requests/sec peak, multi-region.
  • Audit and appeals requirements.

5–15 min Architecture

LLM gateway ──(prompt, response stream)──▶ Moderation service
   ├ Policy service: per-tenant config (categories, thresholds, actions), versioned
   ├ Stream segmenter: buffers tokens into sentence/chunk windows
   ├ Tier 1: fast classifiers (small multi-label model, regex/PII detectors) – inline
   ├ Tier 2: LLM judge with policy rubric – for borderline scores, async or inline by policy
   ├ Decision engine: combine scores + policy → allow / redact / halt / replace message
   └ Audit log: decision, scores, policy version, request id (PII-minimized)
Human review queue ← sampled + borderline + appealed items
Metrics: precision/recall per category, latency, block rate by tenant

15–25 min Deep dives

Streaming inspection.

  • Buffer tokens until a boundary (sentence end or N tokens), classify the window plus a little preceding context.
  • Release the window to the user only after it passes (adds small, steady delay) or release immediately and retract on violation (lower latency, but the user may glimpse content). Choose per surface risk.
  • On violation: stop generation upstream (cancel request to save cost), replace with a safe message.

Tiering for latency and cost.

  • Tier 1 runs on every chunk: tiny classifiers on CPU/GPU, single-digit ms.
  • Tier 2 only for scores in an uncertainty band, bounded by a concurrency budget.

Policy configurability.

  • Thresholds per category per tenant (a children's education app vs. a security research tool).
  • Policy versions recorded with each decision for audit and reproducibility.

Failure modes. Decide fail-open vs. fail-closed per surface; circuit breakers; degrade to Tier 1 only under load.

25–32 min Trade-offs and quality

  • Precision vs. recall per category; over-blocking frustrates users, under-blocking causes harm.
  • Context matters (quoting vs. endorsing, fiction vs. instructions); include conversation context in classification windows.
  • Multilingual coverage; adversarial obfuscation (leetspeak, spacing).
  • Measurement: labelled eval sets per category and language; shadow mode before enforcing new policies; drift monitoring.

32–37 min Production example

A platform rolled out streaming moderation: 40-token windows, Tier 1 classifier at ~5 ms per window, Tier 2 judge for ~3% of windows. Shadow mode for two weeks surfaced false positives on medical content in a health-tech tenant, fixed by a tenant-specific threshold and an allowlist of clinical terms. Enforcement then launched with per-category dashboards and a weekly human review sample.

37–40 min Likely follow-ups

  • How do you moderate inputs too? Same service on prompts; different policies (e.g. jailbreak detection).
  • How do you handle appeals? Store decision context; reviewers relabel; feed into training data.
  • How do you update classifiers safely? Offline eval, shadow, canary, per-category comparisons.

What interviewers listen for

  • Streaming-specific design (windowing, release strategy, cancellation).
  • Tiered classification for latency and cost.
  • Configurable policies, auditability and measurement.

Red flags

  • Only checking the full response after it's been streamed to the user.