Home / Question bank / AI System Design
Explain semantic caching and its risks.
30-second answer
Store past query embeddings with their answers; for a new query, return a cached answer if a previous query is similar enough. It saves cost and latency for repetitive traffic. Risks: near-identical questions can need different answers ("cancel my order" vs. "don't cancel my order"), personalization and permission differences, and stale answers. Use conservative thresholds, scope caches by user/tenant, add TTLs, and evaluate false-hit rates.
Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.
Architecture diagram
Sketch this on the whiteboard as you talk through the answer. See all AI architecture diagrams →
Scroll sideways to see the whole diagram.
0–5 min Clarify and frame
Ask about the traffic: how repetitive are queries? Are answers personalized? How often does the underlying content change? Are any responses action-taking?
Framing: "Semantic caching returns stored answers for similar questions. It can save significant cost and latency, but similarity isn't equivalence, so the design is mostly about preventing wrong hits."
5–15 min Core explanation
How it works:
- Normalize and embed the incoming query (after rewriting follow-ups into standalone questions).
- Search a vector index of past queries, scoped by cache key (tenant, locale, product version).
- If the best match exceeds a similarity threshold (and passes optional checks), return its stored answer.
- Otherwise call the LLM, return the answer, and store
(embedding, query, answer, metadata, timestamp).
Three different caches people confuse:
| Cache | What's reused | Correctness risk |
|---|---|---|
| Exact-match response cache | Answer for an identical normalized prompt | None (if inputs truly identical) |
| Semantic response cache | Answer for a similar query | Wrong answer for a subtly different question |
| Provider prompt caching | Computation for a repeated prompt prefix | None; still generates fresh output |
15–25 min Deep dive: risks and safeguards
| Risk | Example | Safeguard |
|---|---|---|
| Opposite intent | "Cancel my order" vs. "don't cancel my order" | High threshold; equivalence check; never cache action intents |
| Entity differences | "Price of Pro plan" vs. "price of Team plan" | Extract key entities and include them in the cache key |
| Personalization | "What's my balance?" | Exclude personal queries or scope cache per user |
| Permissions | Answer built from restricted documents | Scope by tenant and permission set |
| Staleness | Policy changed yesterday | TTLs; invalidate entries linked to updated source documents |
| Conversation context | "What about the second one?" | Cache only standalone rewritten queries |
Equivalence check: before serving a hit, a small model or cross-encoder judges whether the two questions ask the same thing. This adds a few milliseconds and catches many false hits.
Safer variant: cache retrieval results (chunk IDs) rather than final answers, so the response is still generated fresh for the exact question.
25–32 min Measuring and tuning
- Hit rate and estimated cost/latency savings.
- False-hit rate: sample hits and have a judge or human check whether the cached answer correctly answers the new question. This is the metric that matters most.
- Threshold tuning: build a labelled set of query pairs (equivalent vs. not), plot precision/recall of "equivalent" across thresholds, choose a high-precision point.
- Monitor by segment; some topics are safe to cache, others not.
32–37 min Production example
A public FAQ bot with very repetitive questions added a semantic cache keyed by locale and product version, with a high threshold and 24-hour TTL. Hit rate reached roughly a third of traffic. A weekly sample review found false hits between plan names ("Pro" vs. "Team"), so they added extracted plan names to the cache key. Account-specific intents were excluded entirely by the intent router.
37–40 min Likely follow-ups
- Where do you store the cache? A vector index with TTL support (in-memory for speed, or the existing vector store).
- How do you invalidate on content changes? Store the source document IDs used for each answer; on update, delete dependent entries.
- Is it worth it for low-repetition traffic? Often not; measure the potential hit rate from logs first.
What interviewers listen for
- Distinguishing semantic caching from exact and prompt caching.
- Concrete false-hit risks and safeguards.
- Measuring false hits, not just hit rate.
Red flags
- Caching personalized or action-taking responses globally.