Chaos Monkey Won't Find Your Bug

We shipped a chaos feature that never caused any chaos.

Our mock server has had a fault-injection effect for years with a straightforward job: withhold the response entirely and see whether the caller copes. Last week I audited it against the actual code path. It had never withheld anything.

The handler returned early without writing a response. Go’s net/http then did what it is designed to do, which is synthesize a 200 OK and flush the recorded body. Every experiment anybody ever ran with that effect handed the application a perfectly good answer, and every one of them passed.

sequenceDiagram
    participant Caller
    participant Handler as Mock handler
    participant HTTP as net/http
    Caller->>Handler: Request
    Handler-->>Handler: Withhold effect fires
    Handler--xHTTP: Returns without writing a response
    HTTP->>Caller: Synthesizes 200 OK + recorded body
    Note over Caller: Experiment "passes"

The bug survived because we validated it the way you validate infrastructure chaos. Did the experiment fire? It fired. The counter went up. Nobody asked what the application on the other end actually received.

That is this entire post in miniature. Infrastructure chaos and application chaos are not two intensities of the same practice. They break different layers, they surface different bug classes, and buying one does not cover you for the other.

The layer you break decides the bug you find

When you kill a pod, you are testing Kubernetes. You are asking whether the scheduler reschedules, whether the readiness gate holds traffic, whether the load balancer drains connections before the container dies. Those are good questions. They are questions about your platform.

When you make a dependency return 503, you are testing your code. You are asking whether the retry budget is bounded, whether the circuit breaker opens, whether the fallback returns a sane default instead of a stack trace.

flowchart LR
    subgraph Infra[Infrastructure chaos]
        A1[Kill a pod] --> A2[Scheduler reschedules?]
        A3[Drop a node] --> A4[Load balancer drains?]
    end
    subgraph App[Application chaos]
        B1[Return 503] --> B2[Retry budget bounded?]
        B3[Null field] --> B4[Fallback returns a default?]
    end

The second category is where the incidents come from. The canonical study here is still Yuan et al.’s analysis of 198 production failures across Cassandra, HBase, HDFS, MapReduce and Redis, which found that 92% of catastrophic failures resulted from incorrect handling of non-fatal errors that the software had already explicitly signaled. The error arrived. Something noticed it. The handler was wrong.

The same study found 35% of those catastrophic failures came from error handlers so trivially broken they violated basic practice: a handler that swallows the error, a handler that over-catches and aborts the process, a handler whose body is a TODO comment. And 77% of the failures were reproducible in a unit test.

None of that is reachable by draining a node. The fault has to arrive where the code expects data.

Netflix built the second system for a reason

This is not a new observation, and the people who invented infrastructure chaos are the ones who made it.

Chaos Monkey came first and killed instances. Chaos Kong scaled that up to regions. Then Netflix built Failure Injection Testing, which injects request-level faults into individual microservices in production by hooking the shared Java client libraries every service already used. Then they built the Chaos Automation Platform on top of FIT to run those experiments continuously against canary and control clusters through Spinnaker.

Read that sequence as an architecture decision, not a product history. Netflix had the best instance-killing tooling on earth and still needed a separate mechanism, at a separate layer, with a separate integration point, to reach the failures that live inside application call paths.

Tool categories overlap. Chaos Mesh supports infrastructure experiments and HTTPChaos response delays, replacements, and patches. Istio fault injection, Toxiproxy, Netflix’s FIT, and API-level mocking tools also act on request paths, with different scopes and failure modes. Compare the experiment support before deciding that buying a second tool is necessary.

Most organizations buy from the first list, run a quarterly game day, and record chaos engineering as covered.

The worst outage of last year was not an infrastructure failure

On November 18, 2025, Cloudflare went down for five hours and thirty-eight minutes. The cause was a permissions change on a ClickHouse cluster that made a query return duplicate column metadata. That query generated a Bot Management feature file. The file doubled in size, crossed a hard limit of 200 features that the consuming code assumed it would never approach, and the proxy panicked.

Every machine was healthy. Nothing was partitioned. No region was lost. A dependency returned a payload with a different shape than the consumer expected, and the consumer’s handling of that case was to die.

Infrastructure chaos cannot find that failure, because there is no infrastructure fault to inject. The fault arrives as data, through a code path that thinks it is on the happy road.

The failures that live in the gap

There is a third category, and it is the one that makes running both layers worth the budget line.

Bronson et al. described metastable failures as systems that degrade under a transient stressor and then fail to recover once the stressor is gone. Retry storms, congestive collapse, death spirals. The paper’s key point is that the root cause is not a hardware fault and not a software bug. It is emergent behavior from work amplification.

That failure needs both halves to reproduce. The trigger is usually infrastructure shaped, a node lost or a brief latency spike. The amplifier is always application shaped: an unbounded retry, a client timeout longer than the server’s, a cache stampede. Test either layer alone and the system looks fine.

Running them in tandem without doubling the budget

The two practices have different economics, and the mistake is running them on the same cadence under the same owner.

Infrastructure chaos has a blast radius that crosses team boundaries. It needs a human watching, a rollback plan, and a scheduled window. That is a platform team responsibility and a game-day rhythm, measured in weeks.

Application chaos is scoped to a single service and its dependencies. It is deterministic enough to assert on, cheap enough to run per commit, and it belongs in the service team’s pipeline next to the integration tests. Measured in minutes.

Three practical rules that have held up for me:

• Own them separately. If the platform team owns both, application faults get scheduled behind infrastructure faults forever and never make it into CI.
• Assert at the caller, not at the injector. The question is never “did the experiment fire.” It is “what did the application receive, and what did it do next.” That is the exact mistake our own no-op fault effect survived on.
• Derive faults from real traffic. A 503 you invented tests the branch you imagined. A malformed payload your actual dependency sent last Tuesday tests the branch you did not.

That last point is where I have a horse in the race, so take it accordingly. proxymock’s chaos support works by manipulating individual request response times, status codes and data patterns against recorded dependency traffic, and our own documentation has said for years that the approach is complementary to Gremlin and Chaos Monkey rather than a replacement. That framing was correct before it was convenient.

The counter went up

If you take one thing from this: the reason our fault effect stayed broken for years was not that the test was missing. The test existed and it was green. It asserted that the injector ran.

Chaos engineering has the same failure mode at the organizational level. The game day happened. The report says the pods came back. Nobody asked what the application received, because the tool that broke things was never sitting close enough to the application to know.

Kill the pod. Then lie to the service. They are different questions, and your customers will find whichever one you skipped.

Frequently asked questions

What is the difference between infrastructure chaos and application chaos?

Infrastructure chaos breaks the platform underneath a service: it kills pods, drops nodes, saturates CPU, partitions networks. Application chaos breaks what a service receives from its dependencies: a 503 instead of a 200, a field that arrives null, a response that takes nine seconds. The first tests whether your orchestrator and your topology recover. The second tests whether your own error handling code is correct.

If I already run LitmusChaos or Gremlin, do I need application-level fault injection too?

You need coverage for application-level failures, but that does not necessarily require another product. Review which experiments your current tooling supports. Killing a pod does not test a malformed success response; add response-level scenarios for retry limits, fallback values, and unexpected data when those are part of the contract.

Why did Netflix build FIT and ChAP if they already had Chaos Monkey?

Chaos Monkey and Chaos Kong test resilience to instance and regional failure. Netflix built Failure Injection Testing to inject request-level faults into individual microservices in production, then built the Chaos Automation Platform on top of it to make those experiments safe and continuous. The infrastructure tools could not reach the failure modes that live inside application call paths.

Which chaos experiments belong in CI and which belong in a game day?

Deterministic, isolated experiments with explicit assertions fit well in CI. Experiments that affect shared infrastructure or have a wider blast radius need coordination and recovery criteria. The right location depends on scope and controls rather than a rule that all infrastructure faults must be manual.

What kind of outage does infrastructure chaos testing miss entirely?

A pod-kill or network-partition experiment does not exercise every malformed-payload failure. Cloudflare's November 18, 2025 outage involved a generated Bot Management feature file exceeding a consumer limit. Reproducing that condition requires a data or application-level scenario rather than only removing infrastructure.