Home / Question bank / AI System Design
When should you NOT use an LLM?
30-second answer
When a deterministic rule, regex, lookup or small classifier solves the problem reliably; when latency or cost budgets are tiny and volume huge; when outputs must be exactly reproducible or provably correct; or when you can't tolerate any unverified errors and can't add verification. LLMs are best where inputs are messy and some fuzziness is acceptable or checkable.
Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.
0–5 min Clarify and frame
Framing: "LLMs are excellent for messy, open-ended inputs, but they're slower, costlier, non-deterministic and occasionally wrong. I'll give criteria for when another tool is better, list concrete alternatives, and show hybrid designs that use LLMs where they add the most value."
It's worth stating the properties you're trading off: accuracy, latency, unit cost, determinism, explainability, and development time.
5–15 min Core explanation: criteria
Prefer a non-LLM approach when:
- Deterministic rules suffice. Validating an email format, routing on known keywords, date arithmetic, unit conversion.
- The answer lives in structured data. "What's my order status?" is a database query. An LLM can phrase the result, but must not compute or guess it.
- Latency or volume is extreme. Millions of decisions per minute at single-digit milliseconds.
- Exactness is mandatory. Financial calculations, tax rules, cryptographic operations.
- Decisions must be reproducible and explainable. Credit decisions or compliance checks where regulators expect consistent, auditable logic.
- A narrow model already performs well. A classifier trained on labelled data can be more accurate and far cheaper for a fixed task.
- Errors can't be checked. If no one or nothing can verify outputs and errors are costly, an LLM is risky.
15–25 min Deep dive: alternatives and hybrids
| Task | Usually better | Where an LLM still helps |
|---|---|---|
| Spam/toxicity at huge scale | Small fine-tuned classifier | Labelling training data; handling ambiguous cases |
| Extracting fields from fixed templates | Parsers, regex | Messy or varied documents |
| Product search | Search engine + learned ranking | Query understanding, conversational refinement |
| Recommendations | Ranking / collaborative filtering | Explanations, cold-start descriptions |
| Arithmetic, statistics | Code execution | Writing the code |
| Routing requests | Rules or small classifier | Long-tail ambiguous requests |
Hybrid patterns:
- Distillation into a classifier: an LLM labels tens of thousands of examples (spot-checked by humans), a small model trains on them and runs in production.
- Head and tail: rules or a classifier handle common cases; low-confidence or unusual cases go to the LLM.
- LLM as translator to formal systems: the model writes SQL or code; deterministic systems execute and validate it.
- LLM for language, systems for truth: retrieval and databases provide facts; the LLM only explains them.
25–32 min Decision process
- Define the task's success metric and constraints (latency, cost per unit, error tolerance).
- Build a quick baseline with the simplest approach (rules, existing classifier).
- Try an LLM prompt on the same eval set.
- Compare quality, unit economics at expected volume, and operational complexity.
- Choose, or combine, based on data rather than novelty.
32–37 min Production example
A team used an LLM to categorize 20M support emails a month. It worked, but cost and latency were high. They used LLM labels on 50k emails, had humans review a sample, and trained a small classifier that ran in milliseconds on CPU. Only the ~8% of emails with low classifier confidence went to the LLM. Accuracy stayed about the same and cost dropped by over an order of magnitude.
37–40 min Likely follow-ups
- How do you justify an LLM to stakeholders? Show eval results against a baseline and unit economics at projected volume.
- When would you replace an LLM feature later? When volume grows, the task stabilizes, and enough labelled data exists to train something cheaper.
What interviewers listen for
- Concrete criteria and named alternatives.
- Hybrid designs and an evidence-based decision process.
Red flags
- Using an LLM for arithmetic, lookups or tasks a regex handles.