Home / Question bank / AI System Design

AI System DesignSeniorQuestion 10 of 12

How would you design a model router?

30-second answer

Classify each request by difficulty or type with a cheap model or heuristics, then send easy ones to a small model and hard ones to a large model. Alternatively, cascade: try the small model and escalate if a verifier or confidence signal is low. Measure the quality/cost frontier on an eval set, include fallbacks for provider outages, and monitor routing distribution for drift.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

Architecture diagram

Sketch this on the whiteboard as you talk through the answer. See all AI architecture diagrams →

Scroll sideways to see the whole diagram.

Model router with a verification cascadeComponents: Fallback provider (used during outages); Monitor (quality · cost · drift); Request; Router (classifier on embedding); Small model (~70% of traffic); Verifier / tests (cascade check); Response; Large model (hard requests). Flow: Request to Router; Router to Small model (easy); Router to Large model (hard); Small model to Verifier / tests; Verifier / tests to Response (pass); Verifier / tests to Large model (fail → escalate); Router to Fallback provider (provider down); Response to Monitor (log outcome).Fallback providerused during outagesMonitorquality · cost · driftRequestRouterclassifier on embeddingSmall model~70% of trafficVerifier / testscascade checkResponseLarge modelhard requestseasyhardpassfail → escalateanswerprovider downlog outcome
Model router with a verification cascade. A cheap classifier sends easy requests to a small model and hard ones to a large model. Small-model answers that fail verification escalate. A fallback provider covers outages, and outcomes feed monitoring.

0–5 min Clarify and frame

Ask about the traffic mix (simple FAQs vs. complex reasoning), quality bar per request type, latency targets, and which models are available. Framing: "A router sends each request to the cheapest model that handles it well enough. I'll cover strategies, how to build and evaluate a learned router, cascades, and operational concerns like drift and fallbacks."

5–15 min Core explanation: strategies

  1. Static rules: route by feature, endpoint, user tier, input length or detected language. Transparent and cheap but coarse.
  2. Classifier router: a lightweight model predicts whether a small model will be good enough, or which category a request belongs to.
  3. Cascade: send to the small model first; escalate to a larger model when a check fails (schema invalid, verifier score low, tests fail, low retrieval confidence).
  4. LLM-as-router: a small LLM reads the request and chooses. Flexible, but adds latency and cost.

Separately, availability routing shifts traffic across providers or regions when one is degraded.

15–25 min Deep dive: building a classifier router

  1. Collect data: run a representative sample of requests through both the small and large models.
  2. Label: for each request, was the small model's answer acceptable? Use an LLM judge with a rubric plus human spot checks.
  3. Features: request embedding, length, presence of code or math, conversation depth, feature/endpoint.
  4. Train a simple model (logistic regression or gradient boosting on embeddings often suffices).
  5. Pick a threshold by plotting the quality vs. cost frontier on held-out data: at each threshold, the share routed to the small model and the overall quality.
threshold   % to small   quality   cost/request
0.9         35%          99.5%     0.72×
0.7         62%          98.9%     0.47×   ← chosen
0.5         80%          96.8%     0.31×

25–32 min Trade-offs

Approach Pros Cons
Rules Simple, explainable Misses difficulty within a category
Classifier Fast, cheap, data-driven Needs labels; drifts as traffic changes
Cascade Quality safety net Escalated requests pay both latencies
LLM router Handles nuance Added cost and latency on every request

Operational concerns:

  • Drift: user behaviour and models change; retrain and re-evaluate periodically.
  • Monitoring: routing distribution, per-route quality metrics, escalation rate.
  • User escape hatch: "regenerate with a stronger model" option, which also produces training signal.
  • Consistency: switching models mid-conversation can change tone; prefer routing per conversation for chat.

32–37 min Production example

A coding assistant routed autocomplete and short explanations to a small model and multi-file refactors to a large model via a classifier. About 70% of traffic went to the small model. Generated code ran through tests; failures escalated to the large model (a cascade). On their eval set, quality differences were within noise and cost per request dropped by more than half.

37–40 min Likely follow-ups

  • How do you route during an outage? Fallback list per route; degrade gracefully with clear messaging.
  • Can the router consider load or price changes? Yes, by adjusting thresholds dynamically.
  • What signals indicate a bad route? Thumbs-down, regenerate clicks, follow-up rephrasing, escalation rate.

What interviewers listen for

  • Several strategies with trade-offs.
  • A data-driven router evaluated on a quality/cost frontier.
  • Drift and monitoring.

Red flags

  • Routing by intuition with no evaluation.