Home / AI System Design

Latency, cost and reliability

Concrete levers for making LLM features fast, affordable and resilient in production.

9 min read

Latency

Total latency ≈ network + queueing + prefill (prompt length) + decode (output length × time per token) + tool calls.

Levers, roughly in order of impact:

  1. Stream tokens so users see progress immediately. Optimize time-to-first-token.
  2. Shorten outputs. Output tokens are the slowest part. Ask for concise formats.
  3. Use smaller models for simpler steps (routing, classification, extraction).
  4. Parallelize independent retrieval and tool calls.
  5. Cache at several layers (below).
  6. Trim prompts: fewer, better retrieved chunks; remove redundant instructions.
  7. Self-hosted serving optimizations: continuous batching, paged KV cache, quantization, speculative decoding.

Cost

Estimate early: cost/day ≈ requests/day × (input_tokens × input_price + output_tokens × output_price). Output tokens usually cost several times more than input tokens.

  • Model routing / cascades: try a cheap model first, escalate when confidence is low.
  • Prompt caching: providers can cache a long, stable prompt prefix (system prompt, tools, documents) for a large discount and faster prefill. Put static content first and variable content last.
  • Response caching: exact-match cache for repeated queries; semantic cache (embedding similarity) for near-duplicates, with care, because a near-duplicate question can need a different answer.
  • Batch APIs for non-interactive work, often at a significant discount.
  • Distillation or fine-tuning a small model on outputs of a large one for a narrow, high-volume task.

Reliability

LLM APIs fail in ways regular APIs don't: rate limits, overloaded errors, timeouts on long generations, and successful responses that are wrong or malformed.

  • Retries with exponential backoff and jitter for 429/5xx; respect Retry-After.
  • Timeouts sized for long outputs, plus streaming so partial progress is visible.
  • Fallbacks: secondary model or provider, or a degraded non-LLM path.
  • Structured outputs: use JSON schema or tool-calling modes, validate with a schema library, and retry with the validation error on failure.
  • Idempotency for tool calls that have side effects.
  • Circuit breakers so a struggling provider doesn't take your service down.

Observability

Log per request: model and version, prompt template version, token counts, latency breakdown (retrieval, TTFT, total), tool calls, guardrail decisions, cost, and user feedback. Use tracing (spans per step) so you can see where a slow or bad answer came from. Be deliberate about PII in logs: redact or restrict access.