Home / Question bank / Coding with AI APIs

Coding with AI APIsSeniorQuestion 9 of 11

How do you handle rate limits from an LLM provider at scale?

30-second answer

Client-side: exponential backoff with jitter honoring Retry-After. System-side: a token-aware rate limiter or queue in front of the provider, prioritizing interactive over batch traffic, per-tenant quotas, request batching where supported, prompt caching to reduce token load, and fallbacks to other models or regions. Monitor utilization against provisioned limits and request increases ahead of launches.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

Architecture diagram

Sketch this on the whiteboard as you talk through the answer. See all AI architecture diagrams →

Scroll sideways to see the whole diagram.

Central, token-aware rate limitingComponents: Tenant quotas (daily token budgets); Batch API (offline, discounted); Services (chat · agents · jobs); Token limiter (Redis token bucket); Priority queue (interactive first); LLM client (backoff + jitter); LLM provider (RPM · TPM limits); Fallback model (other region / provider). Flow: Services to Token limiter; Tenant quotas to Token limiter; Token limiter to Priority queue; Priority queue to LLM client; Priority queue to Batch API (batch work); LLM client to LLM provider; LLM provider to LLM client (429 + Retry-After); LLM client to Fallback model (overloaded).Tenant quotasdaily token budgetsBatch APIoffline, discountedServiceschat · agents · jobsToken limiterRedis token bucketPriority queueinteractive firstLLM clientbackoff + jitterLLM providerRPM · TPM limitsFallback modelother region / providerbatch work429 + Retry-Afteroverloaded
Central, token-aware rate limiting. All services reserve tokens from a shared bucket with per-tenant quotas. Interactive work is prioritized, batch work goes to the batch API, clients back off on 429s, and overloaded requests can fall back.

0–5 min Clarify and frame

Ask about traffic shape (interactive vs. batch, peaks), the provider's limits (requests per minute, input/output tokens per minute), number of services sharing a key, and fairness needs across tenants. Framing: "Handle 429s gracefully in clients, but mostly prevent them by managing capacity centrally: token-aware limiting, prioritization, quotas and fallbacks."

5–15 min Core explanation

Client-level handling:

  • Retry on 429 and transient 5xx/overloaded responses with exponential backoff plus jitter, honouring Retry-After.
  • Don't retry 400-class errors other than 429; they're bugs.
  • Cap attempts and total wait; surface a graceful message.
def call_with_retry(fn, max_attempts=5, base=0.5, cap=20):
    for attempt in range(max_attempts):
        try:
            return fn()
        except RateLimited as e:
            delay = e.retry_after or min(cap, base * 2 ** attempt)
            time.sleep(delay * random.uniform(0.5, 1.5))   # jitter avoids synchronized retries
        except Overloaded:
            time.sleep(min(cap, base * 2 ** attempt) * random.uniform(0.5, 1.5))
    raise ServiceBusy()

System-level prevention:

  • Central token-aware rate limiter (token bucket in Redis) shared by all instances.
  • Priority queues: interactive traffic ahead of batch.
  • Per-tenant quotas so one customer can't starve others.
  • Batch APIs for offline work.
  • Prompt caching and shorter prompts to reduce token consumption.
  • Fallbacks to other models, regions or providers.

15–25 min Deep dive: token-aware limiting

Provider limits are often on tokens per minute, so counting requests isn't enough.

  1. Estimate tokens before sending: counted input tokens + max_tokens for output.
  2. Reserve that amount from the bucket; wait or queue if insufficient.
  3. After the response, reconcile with actual usage and return unused capacity.
  4. Keep separate buckets per model/limit type.

Setting realistic max_tokens matters: over-reserving output tokens wastes capacity.

25–32 min Trade-offs

  • Queuing protects the provider limit but adds latency; set queue timeouts and shed load with clear messages.
  • Aggressive client retries across many instances can create retry storms; jitter and central limiting prevent this.
  • Fallback models may differ in quality; route only suitable traffic to them.
  • Capacity is a business decision: request limit increases or provisioned throughput ahead of launches.

32–37 min Production example

A nightly batch summarization job consumed most of the team's token quota, causing 429s for interactive users in another region. Fixes: moved batch work to the provider's batch API, added a Redis token bucket with separate priority for interactive traffic, and set per-tenant daily quotas. Interactive 429s disappeared and batch cost fell because of the batch discount.

37–40 min Likely follow-ups

  • How do you plan capacity? Forecast peak tokens per minute by feature, add headroom, monitor utilization against limits, alert at 70–80%.
  • What metrics do you track? 429 rate, queue depth and wait time, tokens per minute vs. limit, fallback usage.

What interviewers listen for

  • Backoff with jitter and honouring Retry-After.
  • Token-aware central limiting, prioritization and quotas.

Red flags

  • Tight-loop retries, or only per-instance limits.