How do you test code that calls an LLM?

30-second answer

Unit-test the surrounding code with a mocked client (prompt rendering, parsing, retries, tools). Validate output against schemas. Maintain an eval set graded by exact match, programmatic checks or a calibrated LLM judge, run it in CI on prompt/model changes, and look at pass rates over repeated runs. Add online monitoring and user feedback.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "LLM features combine ordinary deterministic code with non-deterministic model behaviour. I'll test them differently: unit tests for code, contract tests for integration, eval suites for model behaviour, and monitoring in production."

5–15 min Core explanation: the layers

Layer	What it covers	Runs
Unit tests	Prompt rendering, parsing, retries, tool implementations, error handling (LLM client mocked)	Every commit, fast
Contract tests	Real calls on a few inputs: schema validity, tool-call shape, no API errors	Every merge / nightly
Eval suites	Model behaviour on a dataset with graders	On prompt/model/retrieval changes, in CI
Online monitoring	Feedback, guardrail triggers, error rates, A/B tests	Continuously

Mocking example:

def test_parses_tool_call(fake_llm):
    fake_llm.queue(tool_use("get_order", {"order_id": "A1"}))
    fake_llm.queue(text("Your order shipped."))
    result = assistant.handle("where is A1?", llm=fake_llm)
    assert result.text == "Your order shipped."
    assert fake_llm.calls[1].tool_results[0]["tool_use_id"] == fake_llm.calls[0].tool_ids[0]

15–25 min Deep dive: eval suites

Dataset: inputs with expected outputs or grading criteria, tagged by category, sourced from real usage and known failures.

Graders, cheapest first:

Exact/set match: labels, extracted fields.
Programmatic: schema validation, length limits, required/forbidden content, code compiles and tests pass, SQL returns expected rows.
Similarity to reference: for short factual answers.
LLM-as-judge with rubric: for helpfulness, tone, faithfulness. Ask for a short justification, calibrate against human labels, randomize order in pairwise comparisons.
Human review: for subjective or high-stakes outputs and to calibrate judges.

Handling non-determinism: run key cases multiple times and assert a pass rate (e.g. ≥ 9/10); compare suite-level scores against a baseline with a tolerance rather than requiring perfection.

@pytest.mark.parametrize("case", load_cases("triage.jsonl"))
def test_triage_category(case):
    passes = sum(triage(case["input"]).category == case["expected"] for _ in range(3))
    assert passes >= 2

25–32 min Process

Eval sets in version control; track scores per category over time.
CI job compares against the main branch baseline; block merges on regressions beyond tolerance.
Record cost and latency of each eval run.
Every production bug becomes a new eval case.
Keep a red-team set (jailbreaks, injections) and track attack success rate.

32–37 min Production example

A summarization feature regressed when a teammate "improved" the prompt; nobody noticed for a week. The team added a 60-case eval with faithfulness and coverage judged by a rubric, calibrated on 30 human-labelled cases. The next prompt change showed a drop in the "long technical document" category in CI, and was fixed before merge.

37–40 min Likely follow-ups

How do you test streaming? Record provider event streams and replay them through client code.
Snapshot tests? Good for rendered prompts; poor for free-form model outputs.
How big should eval sets be? Start with 20–50 sharp cases; grow from production.

What interviewers listen for

Separate strategies for deterministic code vs. model behaviour.
Practical graders and handling of non-determinism.
Evals in CI fed by production failures.

Red flags

Exact-string assertions on free-form outputs, or no tests at all.