AI System DesignSeniorQuestion 1 of 12

Design a customer-support chatbot that answers from a company knowledge base.

30-second answer

Clarify scale, latency, languages and escalation requirements. Metrics: resolution/deflection rate, groundedness, CSAT, p95 latency, cost per conversation. Architecture: ingestion pipeline (parse, chunk by section, embed, index with metadata), orchestrator with input guardrails, hybrid retrieval + rerank, an LLM prompted to answer only from cited context and hand off when unsure, output checks, streaming. Deep dives: evals (golden set with unanswerable cases), human handoff, freshness of docs, caching common questions, and monitoring with feedback loops.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

Architecture diagram

Sketch this on the whiteboard as you talk through the answer. See all AI architecture diagrams →

Scroll sideways to see the whole diagram.

Support chatbot over a knowledge base. Online path: guardrails, intent routing, hybrid retrieval and a grounded, streaming answer, with human handoff. Offline: CMS webhooks keep the index fresh.

0–5 min Clarify and frame

Questions to ask:

Users and channels: customers on web/mobile chat? Logged in (account-aware) or anonymous?
Scope: answer from the knowledge base only, or also take actions (refunds, order status)?
Scale: conversations per day, peak concurrency, languages.
Quality and risk: cost of a wrong answer; regulated domain? When must it hand off to a human?
Latency: first token under ~1s, streaming.
Constraints: data residency, PII handling, budget per conversation.

State assumptions: 50k conversations/day, English plus 3 languages, KB of ~20k articles updated daily, read-only at launch with human handoff.

5–15 min Metrics and high-level design

Success metrics: resolution (deflection) rate without escalation, CSAT, groundedness on a golden set, escalation precision (escalates when it should), p95 latency, cost per conversation, safety violation rate.

Chat widget → API gateway (auth, rate limit) → Conversation service (state, history)
  → Orchestrator
      ├ Input guardrails: PII redaction, abuse/injection classifier, language detect
      ├ Intent router (small model): FAQ / account question / complaint / out-of-scope / handoff
      ├ RAG: query rewrite → hybrid search (filtered by product, locale) → rerank
      ├ LLM (streaming) with grounded-answer prompt and citations
      └ Output guardrails: policy check, groundedness check, tone
  → Response stream → client
Human handoff queue (ticket with transcript + summary)
Logging/tracing → eval pipeline, analytics, feedback
KB ingestion pipeline: CMS webhooks → parse → chunk → embed → index

15–25 min Deep dives

Retrieval quality. KB articles chunked by heading with breadcrumbs, hybrid search for product names and error codes, reranker to top 4, locale filtering. Freshness via CMS webhooks.

Answer policy. Prompt: answer only from sources, cite them, ask a clarifying question when the request is ambiguous, offer handoff when sources don't cover it. Keep answers short with links.

Handoff. Trigger on explicit request, low retrieval confidence, negative sentiment, repeated rephrasing, or sensitive intents (billing disputes, legal). Pass a generated summary to the agent so the customer doesn't repeat themselves.

Cost and latency. Small model for routing and simple FAQs, larger model for complex questions. Prompt caching for the system prompt. Cache answers to the top FAQ queries (scoped by locale and product version). Conversation history summarized after N turns.

Safety. Injection classifier on input; retrieved KB text treated as data; no tools with side effects at launch; output moderation; PII redaction in logs.

25–32 min Trade-offs

Decision	Option chosen	Alternative
Model size	Routing small/large	One large model: simpler, 3–5× costlier
Knowledge	RAG	Fine-tune: stale, no citations
Actions	Read-only first	Tool use with confirmation in v2
Answer caching	Exact-match top FAQs only	Semantic cache: higher hit rate, risk of wrong matches

Risks: hallucinated policies (mitigate with grounding checks), stale articles, over-escalation or under-escalation, adversarial users.

32–37 min Evaluation and rollout

Golden set of 300 real past tickets with correct answers and expected handoff decisions.
Offline gates in CI for any prompt/model/retrieval change.
Launch to 5% of traffic, compare resolution and CSAT against human-only control; expand gradually.
Weekly review of escalations and thumbs-down; add failures to the golden set.

37–40 min Likely follow-ups

How do you add actions like refunds? Tool calls scoped to the authenticated user, policy checks in code, explicit user confirmation, limits, audit logs.
How do you handle multiple languages? Multilingual embeddings, translate-on-ingest or per-locale indexes, answer in user's language, eval per locale.
What if KB coverage is poor? Mine unanswered questions to drive new article creation.

What interviewers listen for

Clarifying scope, risk and handoff.
Metrics first, then architecture.
Retrieval, guardrails, routing, evaluation, rollout.

Red flags

A diagram with "LLM" in the middle and no evaluation, handoff or safety.