Home / AI Architecture
AI Architecture & Design
18 system design questions, each with an architecture diagram you can sketch on the whiteboard, linked to its full 40-minute answer.
How to read the diagrams
Applications
Support chatbot over a knowledge base
Design a customer-support chatbot that answers from a company knowledge base.
RAG over internal documents
Walk me through how you'd build a RAG system over a company's internal docs.
Real-time moderation of streaming LLM output
Design a content moderation service that inspects LLM outputs in real time.
Retrieval
Hybrid retrieval with rank fusion
Why use hybrid search instead of only vector search?
Permission-aware retrieval
How do you enforce document permissions in RAG?
Zero-downtime embedding model migration
Your embedding provider releases a better model. How do you migrate?
Semantic cache with safeguards
Explain semantic caching and its risks.
Serving & reliability
Latency breakdown before and after optimization
How would you reduce the latency of an LLM feature that currently takes 8 seconds?
Model router with a verification cascade
How would you design a model router?
Central, token-aware rate limiting
How do you handle rate limits from an LLM provider at scale?
Validate-and-repair loop for structured output
How do you get reliable JSON out of an LLM?
Streaming LLM responses to the browser
How would you stream an LLM response to a web client?
LLM observability pipeline with privacy tiers
What would you log for an LLM request, and what would you avoid logging?
Model rollout pipeline with gates
How would you evaluate a new model before switching production traffic to it?