Did It Actually Send?

The notifier has returned a message throughout this series, which made testing almost suspiciously easy. Assert on the return value and you are done.

Real notifiers do more than build strings: they send them. Once a message goes to an email provider or SMS gateway, the function may return nothing useful.

When that change lands, every existing test loses the value it asserted on.

This is part 4 of a nine-part series. The code is in Java, Node.js, Go and Python.

The return value disappears

Start with the change. The notifier gains a sender and stops handing the message back to its caller. The sender is a second seam beside the carrier lookup: a place where you can change what code does without editing that code, which is what lets a test slide its own sender in where the real one goes.

public void notify(String trackingNumber) throws IOException, InterruptedException {
    String shipmentStatus = lookup(trackingNumber);
    if ("delayed".equals(shipmentStatus)) {
        sender.send("Package " + trackingNumber + " is delayed");
    }
}

async notify(trackingNumber) {
  const status = await this.#lookup(trackingNumber)
  if (status === 'delayed') {
    await this.#send(`Package ${trackingNumber} is delayed`)
  }
}

func (n *Notifier) Notify(trackingNumber string) error {
	status, err := n.lookup(trackingNumber)
	if err != nil {
		return err
	}
	if status == "delayed" {
		return n.send(fmt.Sprintf("Package %s is delayed", trackingNumber))
	}
	return nil
}

def notify(self, tracking_number: str) -> None:
    status = self._lookup(tracking_number)
    if status == "delayed":
        self._send(f"Package {tracking_number} is delayed")

This is a better design. The notifier’s job was always to notify. Until now, it built a message and left a caller this series never wrote to send it.

It is also a testing problem. A delayed package and a delivered one now look identical from the outside. Both return nothing.

The app now has three seams, and they do not all do the same kind of work. Two supply something the notifier needs. The third carries something out on its behalf, and that is the one with no return value to inspect.

flowchart LR
    N[Package notifier] --> S{{ShipmentStatus seam}}
    N --> W{{Sleep seam}}
    N --> D{{Sender seam}}
    S -->|in production| H[HTTP carrier client]
    S -->|in a test| B["Stub: answers"]
    W -->|in a test| X["No-op sleep"]
    D -->|in production| E[Email or SMS provider]
    D -->|in a test| P["Spy: records messages"]

Two fixes that look reasonable and are not

The first instinct is usually to assert on a log line. Have the notifier log what it sent, capture the log in the test, search it for the message. This works, and it quietly makes your logging format part of your public contract. The first person to reword a log message breaks a test that has nothing to do with logging.

The second instinct is to point the sender at the provider’s sandbox account. That does test the real integration, and it drags the network, credentials and someone else’s uptime back into a unit test. It is a fine thing to run occasionally on a schedule. It is a bad thing to run on every commit.

Both instincts are trying to observe the send from the outside. The cheaper move is to be the thing that receives it.

The spy

A spy is a stand-in that remembers. It satisfies the sender seam, and it keeps a list of every message it was handed so the test can look afterwards.

private static final class SpySender implements Sender {
    private final List<String> sent = new ArrayList<>();

    @Override
    public void send(String message) {
        sent.add(message);
    }
}

function spySender() {
  const sent = []
  const send = async (message) => {
    sent.push(message)
  }
  return { sent, send }
}

type spySender struct {
	sent []string
}

func (s *spySender) send(message string) error {
	s.sent = append(s.sent, message)
	return nil
}

class SpySender:
    def __init__(self) -> None:
        self.sent: list[str] = []

    def __call__(self, message: str) -> None:
        self.sent.append(message)

That is the entire pattern. A list, and something that appends to it. No library, no framework, no code generation. If you have written a mocking framework’s verify(sender).send(anyString()) before, this is what it was doing on your behalf.

One send, and none

The two tests from post 2 come back, rewritten around the recorded list instead of a return value.

@Test
void sendsOneMessageForADelayedPackage() throws Exception {
    ShipmentStatus delayed = trackingNumber -> "delayed";
    SpySender spy = new SpySender();
    PackageNotifier notifier = new PackageNotifier(delayed, spy, NO_SLEEP);

    notifier.notify("TRACK-123");

    assertEquals(List.of("Package TRACK-123 is delayed"), spy.sent);
}

@Test
void sendsNothingForADeliveredPackage() throws Exception {
    ShipmentStatus delivered = trackingNumber -> "delivered";
    SpySender spy = new SpySender();
    PackageNotifier notifier = new PackageNotifier(delivered, spy, NO_SLEEP);

    notifier.notify("TRACK-123");

    assertTrue(spy.sent.isEmpty());
}

test('sends one message for a delayed package', async () => {
  const delayed = async () => 'delayed'
  const spy = spySender()
  const notifier = new Notifier(delayed, spy.send, noSleep)

  await notifier.notify('TRACK-123')

  assert.deepEqual(spy.sent, ['Package TRACK-123 is delayed'])
})

test('sends nothing for a delivered package', async () => {
  const delivered = async () => 'delivered'
  const spy = spySender()
  const notifier = new Notifier(delivered, spy.send, noSleep)

  await notifier.notify('TRACK-123')

  assert.deepEqual(spy.sent, [])
})

func TestNotifySendsOneMessageForADelayedPackage(t *testing.T) {
	delayed := func(string) (string, error) { return "delayed", nil }
	spy := &spySender{}
	n := New(delayed, spy.send, noSleep)

	if err := n.Notify("TRACK-123"); err != nil {
		t.Fatal(err)
	}
	if len(spy.sent) != 1 {
		t.Fatalf("expected exactly 1 message, got %d: %q", len(spy.sent), spy.sent)
	}
	if spy.sent[0] != "Package TRACK-123 is delayed" {
		t.Fatalf("unexpected message: %q", spy.sent[0])
	}
}

def test_sends_one_message_for_a_delayed_package(self) -> None:
    delayed = lambda _: "delayed"
    spy = SpySender()
    notifier = Notifier(delayed, spy, no_sleep)

    notifier.notify("TRACK-123")

    self.assertEqual(["Package TRACK-123 is delayed"], spy.sent)

Asserting on the whole list rather than just its first entry matters more than it looks. A test that only checks sent[0] passes happily when a retry loop sends the same warning three times. Nobody notices in review, because the assertion that would have caught it was never written. Comparing the entire list makes duplicate sends a failing test instead of a support ticket.

It is also worth resisting the urge to assert that the spy was merely “called”. Most mocking frameworks offer that, and it is the weakest useful claim you can make. A notifier that sends an empty string, or sends a warning about the wrong tracking number, satisfies it. Content and count are what a customer experiences.

Two stand-ins, two jobs

Look at what a single test now contains. On one side of the notifier, a function that answers delayed because the test said so. On the other, an object that writes down what it was told to send.

Both are stand-ins. They are not doing the same work. One supplies an input the code needs in order to proceed. The other captures an output the code produced. Confusing the two is how test suites end up asserting on things nobody cares about, which is the subject of the next post.

Every failure test from post 3 now gets a second assertion for free: nothing was sent. A carrier that refuses, times out, or answers with something unrecognized must not produce a message. Before this post there was no way to check that at all.

What these tests prove

That the notifier asked for exactly one message, with the expected text, when the carrier reports a delay. That it asked for none when the package was delivered, when the status was unrecognized, and in every failure case.

What these tests do not prove

That anything was sent. The spy is not an email provider. It cannot tell you whether the provider accepted the message, rendered it, rate-limited it, put it in a spam folder, or whether the address was valid.

It also proves nothing about the message a human eventually reads. The spy compares a string to a string. If the provider truncates at 140 characters or mangles the tracking number, this test stays green.

Those are real risks, and they belong to a different layer of testing. What this layer settles is whether the notifier makes the right decision about sending, which is the part with the branching logic and therefore the part most likely to be wrong.

Try it

Make the spy’s send fail, and decide what the notifier should do: retry it, surface it, or swallow it. Write the test first, then make the code match.

Next, the same spy makes it far too easy to assert on things that are none of the test’s business. Test Behavior, Not Choreography.

Series navigation: Previous: Make Failure Boring with Mocks · All nine posts · Next: Test Behavior, Not Choreography