Your prompt change improved one test case but you're unsure it's better overall. What do you do?

30-second answer

Run the full eval set, ideally multiple times to account for non-determinism, and compare per-category scores against the baseline. Check for regressions in safety and format adherence. If the offline result is positive, roll out behind a flag or A/B test and watch online metrics before making it the default.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "One improved example is an anecdote. I'd measure the change across a representative eval set, account for variance, look per category and at side effects, then roll it out behind a flag."

If there's no eval set yet, say so explicitly: the first step is building one from logs and known failures.

5–15 min Core explanation: the process

Add the motivating case to the eval set (it's a known failure).
Run the full suite on the old and new prompt versions with the same inputs, model and settings.
Repeat runs (e.g. 3–5) to estimate variance from sampling.
Compare per category and overall.
Check side effects: output length, latency, cost, refusal rate, format/schema errors, tool-call behaviour.
Read the diffs: examine cases where outputs changed, especially regressions.
Decide, then roll out behind a feature flag with online monitoring.

15–25 min Deep dive: is the difference real?

With small eval sets, a few points of difference can be noise.

Use paired comparisons: each input is scored under both prompts, so per-input differences cancel out input difficulty.
For pass/fail metrics, count cases that flipped: fixed (fail → pass) vs. broken (pass → fail). A sign test or McNemar's test tells you whether the net change is likely real.
For scores, bootstrap confidence intervals of the mean difference.
Watch for overfitting to the eval set: keep a held-out portion you don't tune on.

Example report:

Category	n	Old	New	Fixed	Broken
Refund questions	40	82%	90%	4	1
Shipping	35	94%	86%	0	3
Account	30	90%	90%	1	1

Here the change helps refunds but breaks shipping: not shippable as is.

25–32 min Rollout

Prompts versioned in source control with IDs; every request logs the prompt version.
Feature flag or percentage rollout; A/B test with online metrics (task success, thumbs-down rate, escalation rate, cost).
Pre-agreed rollback triggers.
Communicate changes so teammates know which version is live.

32–37 min Production example

A support prompt tweak fixed a reported bug about refund timelines. The eval suite showed shipping answers regressed: the new instruction made the model over-apply refund language. A revised prompt fixed both categories; a 20% A/B test confirmed lower thumbs-down rates before full rollout.

37–40 min Likely follow-ups

How big should the eval set be? Big enough per category to see the differences you care about; grow it from production failures.
Can you automate this in CI? Yes: run on prompt-file changes, post a comparison table on the pull request, block on regressions beyond tolerance.

What interviewers listen for

Systematic, paired evaluation with variance awareness.
Per-category analysis and side effects.
Staged rollout with versioning and rollback.

Red flags

Shipping because one example looked better.