Home / Question bank / AI System Design
Your LLM bill doubled last month. How do you investigate and reduce it?
30-second answer
Break cost down by feature, model, tenant and input vs. output tokens using request logs. Look for prompt bloat (too many retrieved chunks, long histories), runaway agent loops, retries, or traffic growth. Reduce with routing to cheaper models, prompt caching, shorter outputs, conversation summarization, response caching, batch APIs for offline work, and budgets/limits per user or tenant.
Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.
0–5 min Clarify and frame
Ask: did traffic double, or cost per request? Which features and models? Any recent changes (prompt, model version, agent rollout)? Framing: "Investigate with data, then reduce with the highest-impact levers without hurting quality."
5–15 min Core explanation: investigation
- Pull usage logs: tokens in/out by feature, model, tenant, day.
- Decompose:
cost = requests × (input_tokens × in_price + output_tokens × out_price). - Compare month over month for each factor.
Common culprits:
- Traffic growth (good problem) or a single heavy tenant.
- Prompt bloat: more retrieved chunks, longer histories, new tool definitions.
- An agent loop running many iterations or failing and retrying.
- A model upgrade to a pricier tier.
- Missing caching after a refactor.
- Abuse: scripted usage of a free tier.
15–25 min Deep dive: reduction levers
| Lever | Typical impact | Watch out for |
|---|---|---|
| Route easy requests to a smaller model | Large | Quality on routed slice |
| Prompt caching for static prefixes | Large for long system prompts/tools | Prefix must be stable and first |
| Trim retrieved context with reranking | Medium | Recall |
| Summarize long conversation history | Medium | Lost context |
| Cap output length | Medium | Truncated answers |
| Response caching (exact/semantic) | Varies with repetition | Staleness, wrong matches |
| Batch API for offline jobs | Large for batch workloads | Latency (hours) |
| Agent step limits and loop detection | Large if loops were the cause | Task completion |
| Per-user/tenant quotas | Prevents spikes | User experience |
| Fine-tune/distill small model for a high-volume task | Large | Upfront effort, maintenance |
25–32 min Governance
- Cost dashboards per feature and tenant; alerts on anomalies.
- Budgets in code: max tokens per request, max steps per agent run.
- Cost included in eval reports so changes are judged on quality and cost.
- Pricing/plan design: heavy usage tiers pay accordingly.
32–37 min Production example
Investigation showed the new research-agent feature caused 70% of the increase: average 14 steps per task, each resending a growing history with large tool outputs. Fixes: truncate tool outputs to relevant fields, summarize history every 5 steps, cap at 10 steps, cache the long tool definitions, use a smaller model for the planning sub-steps. Cost per task fell by about two thirds with similar success rate.
37–40 min Likely follow-ups
- How do you attribute cost? Tag every request with feature, tenant, user; log usage from API responses.
- Self-hosting to save money? Compare total cost (GPUs, ops, utilization) vs. API; usually only wins at high, steady volume.
What interviewers listen for
- Data-driven root cause, not guesses.
- Multiple levers ranked by impact with quality safeguards.
Red flags
- Cutting quality-critical context blindly.