Home / Question bank / RAG & Vector Search

RAG & Vector SearchSeniorQuestion 10 of 12

Your embedding provider releases a better model. How do you migrate?

30-second answer

Embeddings from different models aren't comparable, so you must re-embed the corpus. Build a new versioned index in parallel, backfill in batches, evaluate it against the old one on the golden set, dual-write new documents to both during transition, then shift traffic gradually and keep the old index for rollback.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

Architecture diagram

Sketch this on the whiteboard as you talk through the answer. See all AI architecture diagrams →

Scroll sideways to see the whole diagram.

Zero-downtime embedding model migrationComponents: Index v1 (old embedding model); Queries; Doc updates; Dual-write (during migration); Query router (flag: % to v2); Shadow compare (eval set + feedback); Backfill job (batched, checkpointed); Index v2 (new embedding model). Flow: Doc updates to Dual-write; Dual-write to Index v1; Dual-write to Index v2; Backfill job to Index v2 (re-embed corpus); Queries to Query router; Query router to Index v1; Query router to Index v2; Query router to Shadow compare (mirror).Index v1old embedding modelQueriesDoc updatesDual-writeduring migrationQuery routerflag: % to v2Shadow compareeval set + feedbackBackfill jobbatched, checkpointedIndex v2new embedding modelre-embed corpusmirror
Zero-downtime embedding model migration. A new index is backfilled with the new model while updates are dual-written to both. A flag-controlled router shifts query traffic gradually, with shadow comparison against the eval set.

0–5 min Clarify and frame

Ask about corpus size, embedding cost, uptime requirements and whether the new model has a different dimension. Framing: "Embeddings from different models live in different spaces, so this is a data migration: build a new index alongside the old one, validate, and cut over safely."

5–15 min Core explanation

You can't mix vectors from two models: a query embedded with the new model compared to documents embedded with the old one gives meaningless similarities. So the entire corpus must be re-embedded.

Plan:

  1. Evaluate first: compare old vs. new model on your retrieval eval set (recall@k, MRR), latency and cost. Public leaderboard gains don't always transfer.
  2. Version the index: docs_v1 (old model), docs_v2 (new model). Store embedding_model and version in metadata.
  3. Backfill v2 in batches with rate limiting, checkpointing and retries.
  4. Dual-write: new and updated documents go to both indexes during migration.
  5. Validate v2: counts match, spot-check results, run the eval suite.
  6. Shadow traffic: run queries against both, compare overlap and quality.
  7. Gradual cutover: route a percentage of traffic to v2, monitor feedback and metrics.
  8. Decommission v1 after a rollback window.

15–25 min Deep dive: practical details

  • Cost estimate: total tokens in corpus × embedding price; for self-hosted, GPU-hours at measured throughput.
  • Throughput: batch requests, run in parallel within rate limits, prioritize frequently retrieved documents first.
  • Dimension change: the index schema changes; plan storage (higher dimension = more memory). Some models support truncatable (Matryoshka) embeddings to trade dimension for quality.
  • Query side: switch query embeddings at the same time as the index you query.
  • Re-tune: chunk size, hybrid weights and reranker thresholds may need adjusting.
  • Deletes during migration: apply to both indexes.

25–32 min Alternatives

  • Embedding adapters: learn a linear mapping from new to old space (or vice versa) to avoid full re-embedding. Workable as a stopgap but usually lossy.
  • Stay on the old model: if gains are small relative to migration cost.
  • Pin model versions with hosted providers to avoid surprise changes.

32–37 min Production example

A team with 40M chunks evaluated a new embedding model that improved recall@10 on their eval set. Backfill ran over three days at controlled throughput, prioritising the 20% of documents that served 80% of retrievals. Shadow comparisons showed better results on long questions. They cut over 10% → 50% → 100% over a week and kept v1 for two weeks before deleting it.

37–40 min Likely follow-ups

  • How do you avoid downtime? Parallel index plus alias switching.
  • What if the new model is worse for some queries? Segment eval results by category before deciding; consider routing or keeping the old index for those.

What interviewers listen for

  • Recognizing vectors aren't comparable across models.
  • A safe, staged migration with evaluation and rollback.

Red flags

  • "Just re-embed new documents with the new model."