Walk me through how you'd build a RAG system over a company's internal docs.

30-second answer

Offline: load and parse docs keeping structure, chunk by section with overlap, prepend title/section headers, embed, store with metadata (source, date, ACL). Online: rewrite the query with conversation context, run hybrid retrieval (BM25 + vectors) with permission filters, rerank top ~50 to ~5, build a prompt with delimited, ID-tagged chunks and instructions to cite and to say when the answer isn't present, then generate. Wrap it with an eval set measuring recall@k and faithfulness, plus logging and feedback.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

Architecture diagram

Sketch this on the whiteboard as you talk through the answer. See all AI architecture diagrams →

Scroll sideways to see the whole diagram.

RAG over internal documents. Connectors feed an incremental ingestion pipeline into a hybrid index with ACL metadata. At query time the question is rewritten, retrieved with permission filters, reranked, answered with citations and checked.

0–5 min Clarify and frame

This is a mini system design, so scope it first:

Users and questions: employees asking policy/how-to questions? Engineers searching runbooks?
Corpus: size, formats (PDF, Confluence, Google Docs, tickets), update frequency.
Permissions: does everyone see everything, or per-team/document ACLs?
Quality bar: must answers cite sources? What's the cost of a wrong answer?
Scale: queries per day, latency target (interactive, so streaming with TTFT ~1s).

Then state success metrics: answer correctness and groundedness on a golden set, retrieval recall@k, citation accuracy, user feedback rate, p95 latency, cost per query.

5–15 min Core architecture

INGESTION (offline / incremental)
  Connectors → Parse & clean → Chunk (structure-aware) → Add context headers
            → Embed → Upsert to index (vectors + BM25 + metadata: source, date, ACL)

QUERY (online)
  User + chat history → Query rewrite (standalone question)
    → Hybrid retrieve (BM25 + vector, filtered by user ACL) → top 50
    → Cross-encoder rerank → top 5–8
    → Prompt: instructions + ID-tagged chunks + question
    → LLM (streaming) → Citation check → Answer with source links
  Logging/tracing → Eval pipeline + feedback store

15–25 min Deep dives

Ingestion.

Use connectors with change feeds or webhooks for incremental updates; full reindex nightly as a safety net.
Parse with structure: headings, tables (convert to Markdown), code blocks. Strip navigation and boilerplate.
Chunk by section, ~300–800 tokens with overlap; prefix each chunk with Document title > Section path.
Store metadata: doc_id, url, updated_at, acl_groups, source_system.
Handle deletes: removed documents must disappear from the index quickly.

Retrieval.

Rewrite follow-up questions into standalone queries using chat history.
Hybrid search merges BM25 (exact terms, IDs, acronyms) with dense vectors (paraphrases) via Reciprocal Rank Fusion.
Filter by the user's ACL groups inside the search, not after.
Rerank with a cross-encoder; keep the best few chunks.

Generation.

System prompt: answer only from provided sources, cite source IDs, say when the answer isn't present, keep it concise.
Put the strongest chunks first and last to counter lost-in-the-middle.
Post-check citations; render them as links.

25–32 min Trade-offs and decisions

Decision	Options	Choice and reason
Vector store	pgvector vs. dedicated DB	pgvector if <~5M chunks and Postgres exists; dedicated if heavy filtering/scale
Chunk size	Small vs. large	Small for precision + parent expansion for context
Model	Large vs. small	Mid-size model for most queries, route hard ones up
Freshness	Real-time vs. batch	Incremental via webhooks, nightly reconciliation
Long context instead?	Stuff whole docs	Too costly at this corpus size; no per-user filtering

Risks: stale or duplicated docs, permission leaks, prompt injection hidden in documents (treat document text as data), and users asking questions the corpus can't answer.

32–37 min Evaluation and operations

Golden set of ~200 real questions with labelled source docs and reference answers, including unanswerable and multi-hop cases.
Retrieval metrics: recall@5, MRR. Generation: faithfulness, correctness, citation precision.
Run in CI whenever chunking, embeddings, prompts or models change.
Online: thumbs up/down, "was this helpful", follow-up rephrasing as a failure signal, click-through on citations.
Dashboards: latency breakdown (retrieval, rerank, TTFT, total), cost per query, empty-retrieval rate.

37–40 min Likely follow-ups

How do you handle tables and PDFs? Layout-aware parsing; convert tables to Markdown; keep table header rows with each chunk.
What if two documents conflict? Prefer newer or authoritative sources via metadata; surface the conflict with both citations.
How do you scale to 100M chunks? Sharded ANN index, compression (PQ), tiered storage, caching for frequent queries.

What interviewers listen for

Clarifying permissions, freshness and quality requirements early.
Hybrid retrieval, reranking and ACL filtering at retrieval time.
An evaluation plan separating retrieval from generation.

Red flags

Jumping straight to "embed everything into a vector DB" with no chunking, filtering or eval.
Relying on the prompt to enforce access control.