Your Mock Is Lying

The carrier ships a new version of its API. In the shipment payload, status becomes shipmentStatus. Nothing else changes: same endpoint, same auth, same 200 OK, same valid JSON, every other field in place.

In production, the notifier stops warning anyone. It reads a field that is no longer there, gets nothing back, decides no package is delayed, and goes quiet. No alarm fires, because nothing failed. The carrier is up. Latency is fine. The error rate is zero.

Run the test suite and it is green. All of it. The stubs from post 2 never parsed JSON, and the fake server from post 7 returns whatever body we typed into it.

The problem has a name

Six posts of test doubles, and every one of them was written by us. The stub answers delayed because we said so. The spy records what we send it. The fake recorder implements the promise we decided the database makes. The fake HTTP server returns bytes we typed.

That is not a flaw in any of those doubles. It is the whole point of them, and it is what let us make a delayed package happen on demand. But it has a consequence worth stating plainly: a test suite made entirely of doubles is a test of our beliefs about the world, run against a copy of those same beliefs. It agrees with itself, forever, for free.

Dependencies do not send release notes to your test suite. Something has to go and look.

flowchart LR
    C[(Real carrier API)] -->|changes shape| DRIFT{Drift}
    B["Our doubles: stub,<br/>fake server, fixture"] -->|frozen at write time| DRIFT
    DRIFT -->|invisible to| G["Every mocked test, still green"]
    DRIFT -->|caught by| L["Something that touches the real API"]

The whole of this post is the bottom arrow: what that something is, how often it runs, and what it compares.

Fix 1: fixtures that were true at least once

The first change is the cheapest. Stop typing response bodies and start capturing them.

Post 7’s fake server answered with {"status":"delayed"}, twenty characters that never came from a carrier. Replace that with a file captured from a real response:

{
  "trackingNumber": "TRACK-123",
  "status": "delayed",
  "carrier": "example-carrier",
  "lastScan": {
    "location": "Memphis, TN",
    "timestamp": "2026-08-28T14:22:11Z"
  },
  "estimatedDelivery": "2026-09-02",
  "delayReason": "weather"
}

Most of those fields the notifier never reads. Keep them. They cost nothing, they are what a real response looks like, and they are what makes the file worth comparing against a live response later, which is fix 4.

A captured fixture does not stop drift. It does mean the fixture was true once, and that is a real improvement over a body that was never true at all. Record three things alongside it: when it was captured, what it was captured against (production and a sandbox account are different contracts), and how, so a colleague can recapture it. Generating fixtures from recorded real traffic, which is what tools like proxymock do, is the general form of this fix.

Fix 2: make missing fields loud

Capturing a fixture exposes a second bug, one that was there all along. Ask the old client for a field that is missing and it hands back an empty string. The notifier reads “not delayed” and stays silent.

So the client stops guessing:

if (!shipment.has("status")) {
    throw new CarrierContractException("no status field");
}
return shipment.get("status").asText();

if (!Object.hasOwn(shipment, 'status')) {
  throw new CarrierContractError('no status field')
}
return shipment.status

	if shipment.Status == nil {
		return "", fmt.Errorf("%w: no status field", ErrCarrierContract)
	}
	return *shipment.Status, nil

if "status" not in shipment:
    raise CarrierContractError("no status field")
return shipment["status"]

Go needs a pointer and Java needs has() rather than path() for the same reason: both languages will otherwise hand you a zero value that looks exactly like a real answer. Telling “missing” apart from “empty” is the entire fix.

The new error is deliberately not the carrier-unavailable error from post 3, and the retry budget skips it. An outage is worth asking again about. A carrier that answers correctly in a shape we do not understand will answer identically three times in a row, and retrying only adds load to a service that is working.

Now the renamed fixture fails the suite loudly:

@Test
void rejectsTheRenamedField() throws Exception {
    carrier.enqueue(new MockResponse().setBody(fixture("carrier-shipment-delayed-v2.json")));

    IOException error = assertThrows(
            CarrierContractException.class,
            () -> new CarrierClient(carrierUrl(), "secret-key").lookup("TRACK-123"));

    assertFalse(error instanceof CarrierUnavailableException);
}

test('rejects the renamed field', async () => {
  serveFixture('carrier-shipment-delayed-v2.json')

  await assert.rejects(
    () => new CarrierClient(CARRIER, 'secret-key').lookup('TRACK-123'),
    (error) => {
      assert.ok(error instanceof CarrierContractError)
      assert.ok(!(error instanceof CarrierUnavailableError))
      return true
    },
  )
})

func TestRenamedFieldIsRejected(t *testing.T) {
	server := serveFixture(t, fixture(t, "carrier-shipment-delayed-v2.json"))

	_, err := NewCarrierClient(server.URL, "secret-key").Lookup("TRACK-123")
	if !errors.Is(err, ErrCarrierContract) {
		t.Fatalf("expected ErrCarrierContract, got %v", err)
	}
	if errors.Is(err, ErrCarrierUnavailable) {
		t.Fatal("a contract change is not an outage")
	}
}

def test_rejects_the_renamed_field(self) -> None:
    with FakeCarrier(body=fixture("carrier-shipment-delayed-v2.json")) as carrier:
        with self.assertRaises(CarrierContractError) as caught:
            CarrierClient(carrier.url, "secret-key").lookup("TRACK-123")

    self.assertNotIsInstance(caught.exception, CarrierUnavailableError)

This is progress with a hard limit. The suite now catches the rename once someone hands it the renamed fixture. Nobody has yet gone and asked the carrier.

Fix 3: go and ask

A contract test calls the real thing and asserts that its answers still have the shape the code depends on. It needs credentials and the network, so it is opt-in and stays out of the commit loop:

CARRIER_CONTRACT_TEST=1 \
CARRIER_URL=https://api.your-carrier.example \
CARRIER_API_KEY=... \
CARRIER_TRACKING_NUMBER=... \
  go test ./08-your-mock-is-lying/... -run Contract

Every language in the companion repository skips these unless that variable is set: t.Skip in Go, @EnabledIfEnvironmentVariable in JUnit, the skip option in node:test, and @unittest.skipUnless in Python. Put them in a nightly job. When they fail, the carrier changed, and you find out on a Tuesday morning rather than from a customer.

Fix 4: compare the fixture to reality

The live contract test checks only the fields your code reads today. The next step is stricter: fetch a live response, load the captured fixture, and compare the shape.

Compare field names, not values. Tracking numbers and timestamps are supposed to differ; the set of keys is not. A field in the fixture that is missing from the live response fails the test. A field in the live response that is not in the fixture is reported rather than failed: it means the carrier added something and your fixture is due for a recapture.

This catches renames the notifier would not otherwise notice, including in the fields it never reads, which is precisely where next quarter’s bug is hiding.

Fidelity and speed both belong

The obvious reaction is to distrust the fast tests. That is the wrong lesson.

The mocked suite runs in milliseconds and can create any situation on demand, which is why it carries the coverage. The contract tests are slow, need credentials, and cannot make a package be late, but they are the only ones connected to reality. Each is bad at what the other is good at.

A practical arrangement is mocked tests on every commit, contract tests on a schedule, and a comparison between the fixtures and a live response so the two cannot silently disagree.

What this test proves

The client rejects a response that does not carry the field it needs, instead of treating a missing field as a negative answer. With the contract tests enabled, it also proves the carrier’s current response still has the shape the fixture captured.

What this test does not prove

Offline, it proves nothing about today’s carrier, only about the fixture. The contract tests close that gap while they are running, which is once a night, so between runs you are trusting a snapshot.

The comparison also only checks top-level field names. A field that keeps its name and changes its meaning, such as status going from delayed to DELAYED or switching to a numeric code, passes every check here.

Try it

Rename a second field in the drifted fixture, estimatedDelivery to eta, and run the suite. The client never reads that field, so the contract error does not fire and every offline test stays green. Only the fixture-versus-live comparison notices. That gap is the difference between testing what you parse and testing what you were sent. The full suite is in the companion repository.

Next: the carrier grows to forty endpoints and hand-written doubles stop paying for themselves. When Handwritten Mocks Stop Scaling.

Series navigation: Previous: Mock the Wire · All nine posts · Next: When Handwritten Mocks Stop Scaling