Make Failure Boring with Mocks
Part 3 of 9 in the Getting Started with Mocks series. Previously: Your First Useful Mock.
Every codebase has a failure path nobody has run. Not through laziness, but because reproducing it requires a backend dependency to misbehave on cue. In the package notifier, the carrier must refuse, stall, or return nonsense at the exact moment the test runs. So the retry logic ships unverified and everyone hopes.
The seam from post 2 already gives the test control. A seam is a place where you can change what code does without editing that code. The notifier stopped reaching for the carrier itself and started taking the shipment-status lookup as a parameter. The replacement that made a package delayed on purpose can also make the carrier fail on purpose. In this post, the stand-in starts returning bad news.
This is part 3 of a nine-part series. The code is in Java, Node.js, Go and Python.
Here is the app as it stands, and where this post changes it. The carrier seam is the one from post 2. The clock is new, and it exists so retry tests do not wait in real time.
flowchart LR
N[Package notifier] --> S{{ShipmentStatus seam}}
N --> W{{Sleep seam}}
S -->|in production| H[HTTP carrier client]
H --> C[(Carrier API)]
S -->|in a test| B["Stub: refuses or stalls"]
W -->|in production| R[Real sleep]
W -->|in a test| X["No-op: returns at once"]
- The seam that gives you a happy path also opens the failure paths to testing.
- Retry logic needs a second seam, the clock, or the suite waits in real time.
- An unrecognized answer is not automatically an error. Decide what it means before production decides for you.
The failure path nobody runs
The notifier asks a carrier for a shipment status. Three things can go wrong, and all three happen in production.
The carrier can refuse with a 500, usually when a truck depot’s systems are having a bad morning. It can go quiet, leaving the request hanging until something times out. Or it can return a status nobody wrote code for because a new one was added and the release notes went to a mailing list you are not on.
Against a real carrier you cannot ask for any of those on demand. Against a stand-in you can ask for all of them in a single test file. Doing this against real infrastructure rather than a stand-in is its own discipline, covered in application level dependency chaos testing; this post stays inside the unit suite.
Making the carrier refuse
The stand-in for a refusal is one line: a function that fails instead of answering. Each language gets a named error so the test asserts on the kind of failure rather than a message string.
Java
@Test
void reportsCarrierUnavailable() {
ShipmentStatus unavailable = trackingNumber -> {
throw new CarrierUnavailableException("carrier returned 500");
};
PackageNotifier notifier = new PackageNotifier(unavailable, NO_SLEEP);
assertThrows(CarrierUnavailableException.class, () -> notifier.notify("TRACK-123"));
}Node.js
test('reports that the carrier is unavailable', async () => {
const unavailable = async () => {
throw new CarrierUnavailableError('carrier returned 500')
}
const notifier = new Notifier(unavailable, noSleep)
await assert.rejects(() => notifier.notify('TRACK-123'), CarrierUnavailableError)
})Go
func TestNotifyReportsCarrierUnavailable(t *testing.T) {
unavailable := func(string) (string, error) { return "", ErrCarrierUnavailable }
n := New(unavailable, noSleep)
message, err := n.Notify("TRACK-123")
if !errors.Is(err, ErrCarrierUnavailable) {
t.Fatalf("expected a carrier-unavailable error, got %v", err)
}
if message != "" {
t.Fatalf("expected no message, got %q", message)
}
}Python
def test_reports_carrier_unavailable(self) -> None:
def unavailable(_: str) -> str:
raise CarrierUnavailableError("carrier returned 500")
notifier = Notifier(unavailable, no_sleep)
with self.assertRaises(CarrierUnavailableError):
notifier.notify("TRACK-123")A timeout is the same test with a different failure. Each language has a
natural one already: a deadline error in Go, a timeout exception in Java, a
built-in timeout error in Python. Node’s fetch reports timeouts as a
DOMException, which is awkward to throw from a test, so the carrier client
translates it into a named error first.
The third case has no exception at all. The carrier answers cheerfully with a
status the code has never seen. The companion tests run an empty string, a
plausible in_transit, and a held_at_customs that nobody planned for. All
three mean the same thing here: there is nothing to tell the customer. That is
a product decision, and writing the test is what forces you to make it rather
than discovering the answer in a stack trace.
Retrying needs a second seam
A single failure should not become a customer’s problem. The notifier retries.
Retrying introduces a new dependency that is easy to miss: time. If the code sleeps between attempts, every retry test pays that cost, and a suite that takes a minute is a suite people stop running.
So the wait becomes a seam too, exactly like the carrier did.
Java
private String lookup(String trackingNumber) throws IOException, InterruptedException {
IOException lastFailure = null;
for (int attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
try {
return status.lookup(trackingNumber);
} catch (IOException failure) {
lastFailure = failure;
if (attempt < MAX_ATTEMPTS) {
sleep.pause(BACKOFF);
}
}
}
throw lastFailure;
}Node.js
async #lookup(trackingNumber) {
let lastFailure
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
try {
return await this.#getShipmentStatus(trackingNumber)
} catch (failure) {
lastFailure = failure
if (attempt < maxAttempts) {
await this.#sleep(backoffMs)
}
}
}
throw lastFailure
}Go
func (n *Notifier) lookup(trackingNumber string) (string, error) {
var lastErr error
for attempt := 1; attempt <= maxAttempts; attempt++ {
status, err := n.status(trackingNumber)
if err == nil {
return status, nil
}
lastErr = err
if attempt < maxAttempts {
n.sleep(backoff)
}
}
return "", lastErr
}Python
def _lookup(self, tracking_number: str) -> str:
last_failure: Exception | None = None
for attempt in range(1, MAX_ATTEMPTS + 1):
try:
return self._get_shipment_status(tracking_number)
except (CarrierUnavailableError, TimeoutError, OSError) as failure:
last_failure = failure
if attempt < MAX_ATTEMPTS:
self._sleep(BACKOFF_SECONDS)
raise last_failureProduction passes the real sleep. Every test passes one that returns immediately. The retry tests below run in microseconds and still prove the backoff happened, because the test’s version counts the calls.
That is the same trade as the carrier, applied to a different dependency. Time is a collaborator like any other, and a test that cannot control the clock is at the mercy of it.
A carrier that recovers
Here is the case that justifies the whole retry loop: the carrier fails once, then answers. The stand-in counts its own calls and changes its answer on the second one.
Java
AtomicInteger calls = new AtomicInteger();
ShipmentStatus flaky = trackingNumber -> {
if (calls.incrementAndGet() == 1) {
throw new CarrierUnavailableException("carrier returned 500");
}
return "delayed";
};
AtomicInteger sleeps = new AtomicInteger();
PackageNotifier notifier = new PackageNotifier(flaky, duration -> sleeps.incrementAndGet());
assertEquals("Package TRACK-123 is delayed", notifier.notify("TRACK-123").orElseThrow());
assertEquals(2, calls.get());
assertEquals(1, sleeps.get());Node.js
let calls = 0
const flaky = async () => {
calls++
if (calls === 1) {
throw new CarrierUnavailableError('carrier returned 500')
}
return 'delayed'
}
let sleeps = 0
const notifier = new Notifier(flaky, async () => {
sleeps++
})
assert.equal(await notifier.notify('TRACK-123'), 'Package TRACK-123 is delayed')
assert.equal(calls, 2)
assert.equal(sleeps, 1)Go
calls := 0
flaky := func(string) (string, error) {
calls++
if calls == 1 {
return "", ErrCarrierUnavailable
}
return "delayed", nil
}
sleeps := 0
n := New(flaky, func(time.Duration) { sleeps++ })
message, err := n.Notify("TRACK-123")
// message is "Package TRACK-123 is delayed", calls is 2, sleeps is 1Python
calls = 0
def flaky(_: str) -> str:
nonlocal calls
calls += 1
if calls == 1:
raise CarrierUnavailableError("carrier returned 500")
return "delayed"
notifier = Notifier(flaky, count_sleep)
self.assertEqual("Package TRACK-123 is delayed", notifier.notify("TRACK-123"))
self.assertEqual(2, calls)
self.assertEqual(1, sleeps)Notice what the stand-in just gained. It is no longer only answering a question, it is recording how many times it was asked. That is a different job, and post 4 gives it a name.
A carrier that never recovers
The other half of a retry budget is giving up. A stand-in that always fails proves the loop stops, and that the failure reaches the caller rather than being swallowed into a silent success.
Java
@Test
void givesUpAfterTheBudget() {
AtomicInteger calls = new AtomicInteger();
ShipmentStatus broken = trackingNumber -> {
calls.incrementAndGet();
throw new CarrierUnavailableException("carrier returned 500");
};
PackageNotifier notifier = new PackageNotifier(broken, NO_SLEEP);
assertThrows(CarrierUnavailableException.class, () -> notifier.notify("TRACK-123"));
assertEquals(PackageNotifier.MAX_ATTEMPTS, calls.get());
}Node.js
test('gives up after the budget', async () => {
let calls = 0
const broken = async () => {
calls++
throw new CarrierUnavailableError('carrier returned 500')
}
const notifier = new Notifier(broken, noSleep)
await assert.rejects(() => notifier.notify('TRACK-123'), CarrierUnavailableError)
assert.equal(calls, maxAttempts)
})Go
func TestNotifyGivesUpAfterTheBudget(t *testing.T) {
calls := 0
broken := func(string) (string, error) {
calls++
return "", ErrCarrierUnavailable
}
n := New(broken, noSleep)
if _, err := n.Notify("TRACK-123"); !errors.Is(err, ErrCarrierUnavailable) {
t.Fatalf("expected a carrier-unavailable error, got %v", err)
}
if calls != maxAttempts {
t.Fatalf("expected %d attempts, got %d", maxAttempts, calls)
}
}Python
def test_gives_up_after_the_budget(self) -> None:
calls = 0
def broken(_: str) -> str:
nonlocal calls
calls += 1
raise CarrierUnavailableError("carrier returned 500")
notifier = Notifier(broken, no_sleep)
with self.assertRaises(CarrierUnavailableError):
notifier.notify("TRACK-123")
self.assertEqual(MAX_ATTEMPTS, calls)Seven tests per language now, all of them running with no network, no carrier account and no waiting. Five of the seven cover paths that were previously untestable.
What these tests prove
How the notifier behaves in each class of failure. A refusal and a timeout both surface to the caller rather than being mistaken for a delivered package. An unrecognized status produces silence rather than a crash. A carrier that stumbles once still sends the customer’s message. A carrier that never answers stops being asked, and the failure is reported.
Those are the decisions the retry loop exists to make, and they now hold on every run.
What these tests do not prove
That the real carrier fails in any of these shapes. Every failure here was invented. The real carrier might return a 503 rather than a 500, or a 200 with an error body, or hold the connection open for ninety seconds before a socket error your code never catches.
There is a longer catalog of situations worth simulating in Ways to Use Mock Services. The tests also say nothing about whether the retry budget is the right one. Three attempts with a fixed wait is a guess, and a stand-in will happily confirm whatever guess you encode.
Post 7 puts the real HTTP client in front of a fake server, which is where error status codes and malformed bodies get tested for real. Post 8 is about what happens when the invented failures drift away from the real ones.
Try it
Make the retry budget configurable instead of a constant, then test the boundary: a carrier that fails exactly as many times as the budget allows, and one that fails once more.
Next, the notifier stops returning a message and starts sending one, which leaves the tests with nothing to assert on. Did It Actually Send?
Series navigation: Previous: Your First Useful Mock · All nine posts · Next: Did It Actually Send?