Test Behavior, Not Choreography

The spy from post 4 is a sharp tool. Once a test can record every interaction, it is tempting to assert on all of them. The result looks thorough, but it is usually a transcript rather than a useful specification.

This post takes a test written that way, makes a change that no customer could possibly notice, and watches the test fail anyway.

This is part 5 of a nine-part series. The code is in Java, Node.js, Go and Python.

A test that checks everything

Here is the give-up path from post 3, tested exhaustively. The stand-ins write every interaction into a shared list, and the test asserts on the entire sequence: the carrier was asked, then the code waited, then it asked again, and so on until the budget ran out.

List<String> events = new ArrayList<>();
ShipmentStatus broken = trackingNumber -> {
    events.add("carrier " + trackingNumber);
    throw new CarrierUnavailableException("carrier returned 500");
};
SpySender spy = new SpySender();
PackageNotifier notifier = new PackageNotifier(
        broken, spy, duration -> events.add("sleep " + duration.toMillis() + "ms"));

// ... notify, catching the expected failure ...

assertEquals(
        List.of(
                "carrier TRACK-123",
                "sleep 100ms",
                "carrier TRACK-123",
                "sleep 100ms",
                "carrier TRACK-123"),
        events);

const events = []
const broken = async (trackingNumber) => {
  events.push(`carrier ${trackingNumber}`)
  throw new CarrierUnavailableError('carrier returned 500')
}
const spy = spySender()
const notifier = new Notifier(broken, spy.send, async (ms) => {
  events.push(`sleep ${ms}ms`)
})

await assert.rejects(() => notifier.notify('TRACK-123'))

assert.deepEqual(events, [
  'carrier TRACK-123',
  'sleep 100ms',
  'carrier TRACK-123',
  'sleep 100ms',
  'carrier TRACK-123',
])

var events []string
broken := func(trackingNumber string) (string, error) {
	events = append(events, "carrier "+trackingNumber)
	return "", ErrCarrierUnavailable
}
spy := &spySender{}
n := New(broken, spy.send, func(d time.Duration) {
	events = append(events, "sleep "+d.String())
})

_ = n.Notify("TRACK-123")

want := []string{
	"carrier TRACK-123",
	"sleep 100ms",
	"carrier TRACK-123",
	"sleep 100ms",
	"carrier TRACK-123",
}

events: list[str] = []

def broken(tracking_number: str) -> str:
    events.append(f"carrier {tracking_number}")
    raise CarrierUnavailableError("carrier returned 500")

spy = SpySender()
notifier = Notifier(broken, spy, lambda seconds: events.append(f"sleep {seconds}s"))

with self.assertRaises(CarrierUnavailableError):
    notifier.notify("TRACK-123")

self.assertEqual(
    [
        "carrier TRACK-123",
        "sleep 0.1s",
        "carrier TRACK-123",
        "sleep 0.1s",
        "carrier TRACK-123",
    ],
    events,
)

Drawn out, the sequence that assertion pins looks like this. Every arrow is something the test insisted on, including the two waits in the middle.

sequenceDiagram
    participant T as Test
    participant N as Notifier
    participant C as Carrier stub
    participant S as Sender spy
    T->>N: Notify TRACK-123
    N->>C: shipment status
    C-->>N: carrier unavailable
    N->>N: wait 100ms
    N->>C: shipment status
    C-->>N: carrier unavailable
    N->>N: wait 100ms
    N->>C: shipment status
    C-->>N: carrier unavailable
    N-->>T: failure surfaces
    Note over S: never called

Read that assertion and ask what it is protecting. Three carrier calls. Two waits. A specific order. A specific duration.

Only some of that is behavior. The rest is a description of how the retry loop happens to be written today.

A change nobody can notice

Now make an ordinary improvement. Hammering a struggling carrier every hundred milliseconds is impolite, so the backoff doubles after each failed attempt.

static Duration backoffFor(int attempt) {
    return BASE_BACKOFF.multipliedBy(1L << (attempt - 1));
}

function backoffFor(attempt) {
  return baseBackoffMs * 2 ** (attempt - 1)
}

func backoffFor(attempt int) time.Duration {
	return baseBackoff * time.Duration(1<<(attempt-1))
}

def backoff_for(attempt: int) -> float:
    return BASE_BACKOFF_SECONDS * 2 ** (attempt - 1)

Consider what a customer experiences after this change. The carrier is still asked three times. Nothing is still sent. The same failure still reaches the caller. The whole thing still finishes in well under a second.

The externally visible result did not change. The test fails anyway:

--- FAIL: TestGiveUpChoreography (0.00s)
    notifier_test.go:166: event 3: expected "sleep 100ms", got "sleep 200ms"

That failure is not telling you about a bug. It is telling you that you edited the code, which you already knew.

The smell

The test knew the choreography, but knowing the steps is not the same as understanding the outcome.

Choreography is the sequence of steps the code performs internally. Outcome is what the world looks like afterwards. Users, API clients, and downstream services experience outcomes. Only the test sees the choreography. A transcript can help with debugging, but it is a poor specification because it changes when the implementation changes.

Tests that pin choreography have a predictable cost. They fail on refactors, which trains people to update the expected sequence without reading it. After enough of that, the assertion is a snapshot of whatever the code did last Tuesday, and it protects nothing.

Worse, the cost lands on exactly the work you want to encourage. Cleaning up a retry loop, extracting a helper, or reordering two independent calls does not change what the software does, but each can turn the suite red. A team that pays that tax often enough stops refactoring, which is a strange thing for a test suite to accomplish.

Hand-written spies make this mistake visible, because somebody has to type the expected sequence out. Mocking frameworks make the mistake easier to hide. A strict mock that fails on any unexpected interaction is an over-specified test by default, and you opt into that behavior before you have written a single assertion.

The rewrite

The same scenario, asserted on what a customer could actually notice: nothing was sent, and the failure surfaced instead of being swallowed.

@Test
void givingUpTellsNobodyAndSurfacesTheFailure() {
    ShipmentStatus broken = trackingNumber -> {
        throw new CarrierUnavailableException("carrier returned 500");
    };
    SpySender spy = new SpySender();
    PackageNotifier notifier = new PackageNotifier(broken, spy, NO_SLEEP);

    assertThrows(CarrierUnavailableException.class, () -> notifier.notify("TRACK-123"));
    assertTrue(spy.sent.isEmpty());
}

test('giving up tells nobody and surfaces the failure', async () => {
  const broken = async () => {
    throw new CarrierUnavailableError('carrier returned 500')
  }
  const spy = spySender()
  const notifier = new Notifier(broken, spy.send, noSleep)

  await assert.rejects(() => notifier.notify('TRACK-123'), CarrierUnavailableError)
  assert.deepEqual(spy.sent, [])
})

func TestGivingUpTellsNobodyAndSurfacesTheFailure(t *testing.T) {
	broken := func(string) (string, error) { return "", ErrCarrierUnavailable }
	spy := &spySender{}
	n := New(broken, spy.send, noSleep)

	err := n.Notify("TRACK-123")

	if !errors.Is(err, ErrCarrierUnavailable) {
		t.Fatalf("expected the carrier failure to surface, got %v", err)
	}
	if len(spy.sent) != 0 {
		t.Fatalf("expected nothing to be sent, got %q", spy.sent)
	}
}

def test_giving_up_tells_nobody_and_surfaces_the_failure(self) -> None:
    def broken(_: str) -> str:
        raise CarrierUnavailableError("carrier returned 500")

    spy = SpySender()
    notifier = Notifier(broken, spy, no_sleep)

    with self.assertRaises(CarrierUnavailableError):
        notifier.notify("TRACK-123")
    self.assertEqual([], spy.sent)

This version passes before the backoff change and after it. It would also survive switching to jitter, raising the budget to five attempts, or replacing the loop with a library. It fails for exactly one reason: the notifier started telling customers about a package it knows nothing about, or stopped reporting the failure. Both are bugs worth a red build.

The companion keeps the over-specified version, skipped rather than deleted, so you can remove the skip and watch it break for yourself.

Which interaction assertions survive

The rewrite still asserts on an interaction. An empty spy.sent list is a claim about a call that did not happen.

That one stays because sending is behavior. A customer receives the message or does not. There is no meaningful difference between the outcome and the interaction, because the interaction is the whole point of the notifier.

The waits are different. Nobody outside the process can tell a hundred milliseconds from two hundred. The retry count is borderline: post 3 asserted it, correctly, because the budget was the feature being introduced. Once the budget is settled, repeating that assertion in every test pins a number that should be free to move.

A rule of thumb that holds up: assert on what a caller, a user or another service could observe. Treat everything else as private. When you are unsure, imagine changing the thing and ask who files the bug. If the only complainant is the test, the assertion is not earning its place.

What these tests prove

That a customer whose carrier never answers is told nothing, and that the failure reaches the caller rather than disappearing into a silent success.

What these tests do not prove

Anything about the retry schedule, and that is now deliberate. If the backoff matters to you, because a partner rate-limits you at a known threshold, that is a real requirement and it deserves its own named test that says so. What it does not deserve is to be smuggled into every test that happens to have a spy attached.

These tests also still prove nothing about the network. Five posts in, the real HTTP client has never once run under test. That changes in post 7.

Try it

Open your own suite and find one interaction assertion. Decide which side of the line it is on: could a user, a caller or another service tell the difference if it changed?

Next, the series stops saying “mock” for everything. You have now used three different kinds of stand-in without needing their names, which is the right moment to learn them. They Aren’t All Mocks.

Series navigation: Previous: Did It Actually Send? · All nine posts · Next: They Aren’t All Mocks