Break One Dependency, Not The Whole Cluster

Scoped chaos rules are now in proxymock. A filter query picks the traffic, an effect perturbs it, and every response that gets touched is labelled so you can tell an injected failure from a real one for the rest of the run. Available in v2.5.892 and newer.

The short version of why: kill a pod and you learn something real, but you do not learn what your service does when a dependency stays up and starts lying to it.

The branch that has never run

Somewhere in your service is code that handles a dependency being unavailable. It reads from a cache, returns a stale value, marks the response degraded so callers know not to trust it too far. It was written carefully and reviewed properly.

Whether it works is an open question, because nothing in the test suite makes that dependency fail and the dependency does not fail on request. The first execution of that code will be during an outage.

That is the case for chaos engineering in one paragraph. The Principles of Chaos Engineering put it more formally: establish a steady state, hypothesize it holds, introduce a variable, then look for the difference. The value is not the breakage. It is converting an open question about resilience into a closed one, on a Tuesday afternoon, on purpose, when you are watching.

The pressure behind this only grows. A service with four dependencies has a handful of degraded states; one with twenty has more than anyone can enumerate, let alone write fixtures for. Test suites cover the states someone thought to write down, and the interesting failures are the ones nobody did.

What infrastructure chaos does, and where it stops

The tooling built over the last decade is very good, and it is almost entirely aimed at the platform. Netflix’s Chaos Monkey started by terminating instances. LitmusChaos and Chaos Mesh, both CNCF incubating projects, kill pods, throttle bandwidth, exhaust CPU, and corrupt disk IO. Gremlin and AWS Fault Injection Service do the same commercially.

Read the canonical list of disruptions to introduce and the shape is obvious: “servers that crash, hard drives that malfunction, network connections that are severed.” All infrastructure.

That work is genuinely valuable and none of this replaces it. But notice what the whole category is good at: breaking the platform your service runs on. The failures it produces reach your code as timeouts, connection errors, and missing endpoints, and defensive code handles those reasonably well, because they are the failures everyone thinks of first.

	Infrastructure chaos	Application chaos in proxymock
Blast radius	a pod, node, or network link	a filter query: one endpoint, host, or header value
Fault source	described in a spec	a recorded response, perturbed
Failure your code sees	timeout, connection error, missing endpoint	a 503 with a body, a null where a number belongs
Runs where	a cluster	your laptop, CI, or a cluster
Attribution	correlate by timestamp	a marker on the response itself

To be fair to the strongest counter-example: Chaos Mesh’s HTTPChaos does reach this layer, and can abort or delay HTTP calls. Even then you describe the fault abstractly. What none of these tools do is take the response your dependency actually sent, on a real day, and perturb that.

A 503 you invented tests the branch you imagined. A malformed payload your dependency really sent last Tuesday tests the branch you did not.

Getting started

Everything below runs locally. No cluster, no account.

1. Record ordinary traffic. Chaos needs something to perturb, and the point is that nothing rare has to be in the recording:

proxymock record -- ./my-app

2. Break one dependency. Serve the recording back, with a rule:

proxymock mock --in ./proxymock/recording \
  --chaos '(url CONTAINS "/v1/inventory"): status=503,percent=100'

Everything before the colon selects traffic; everything after decides what happens to it. The scope is the same filter syntax the Requests grid and --query-string use, and every group must be parenthesized, which is the most common thing to get wrong.

3. Confirm it on the wire. Injected failures announce themselves:

curl --proxy http://127.0.0.1:4140 --dump-header - \
  http://localhost:8090/v1/inventory/SSC-4110

HTTP/1.1 503 Service Unavailable
X-Speedscale-Chaos: effect=status code;status=503;rule=chaos-1

4. Read the run. proxymock web gives you a Chaos column, a “chaos applied” filter, and a per-rule summary in the Report view:

proxymock web --port 7788

5. Keep the rule. Author rules in Chaos Rules, under Overrides in proxymock-web, and they save to proxymock/chaos.json in the workspace. The next proxymock mock picks them up with no flag at all.

6. Run it in a cluster. Put the rules on the test config your TrafficReplay names, and every replay using that config runs the same experiment:

apiVersion: speedscale.com/v1
kind: TrafficReplay
metadata:
  name: test-1
spec:
  snapshotID: abf5c088-48f2-43a6-bf59-8b12f04144b4
  testConfigID: chaos-inventory-down   # a test config carrying chaos.rules
  workloadRef:
    kind: Deployment
    name: my-app

Full syntax, every effect, and the runtime semantics are in the chaos documentation.

Rule anatomy

Rules are evaluated in order and the first match wins, so one rule applies to any given response. The effects inside that rule all apply, and they compose: latency=2s,status=503 does both.

Effect	Values
latency	2s, 3x (a multiple of the recorded latency), 100ms-2s
status	500, 503, any code
connection	refuse, reset, stall, drop
body	corrupt, truncate, truncate:<bytes>
header	Name:Value
no-response	ends the exchange with no reply at all
payload	perturbs one field inside the body

That last one is the sharpest tool here, and it is the one infrastructure chaos cannot reach at all: a null where the client assumes a number, a type flipped, a key deleted, while the status stays 200 and the contract still looks satisfied.

Then the knobs: percent= for how often the rule fires, seed= for reproducibility, sticky to give every occurrence of a signature one shared verdict, start-after= and duration= to bound the rule to a window, and a @<percent> suffix on an individual effect. That suffix is Toxiproxy’s toxicity rather than a selection weight, so in latency=2s@50,status=503 the status fires every time the rule matches and the latency half of those times.

--chaos '(url CONTAINS "/v1/inventory"): status=503,start-after=10s,duration=30s'

That is the difference between this endpoint is flaky and this dependency went down for thirty seconds and came back, which is the shape a chaos hypothesis usually takes.

Every injected failure carries its name

X-Speedscale-Chaos: effect=status code;status=503;rule=inventory-down

The marker rides on the wire as a response header and is persisted as a tag on the recorded pair, so it survives into the grid, the drawer, the report, and the dashboard.

There is no none value. An untouched response carries no marker at all, which means the marker never has to be interpreted: if it is there, something was injected, and it says which rule did it. Three hours later, looking at a 503 in a report, “was that ours?” has an answer attached to the response itself.

The two decisions that make injected failure usable

Injecting a failure is easy. Keeping the rest of your analysis honest afterwards took two decisions.

Chaos-affected pairs are excluded from drift and match-rate analysis. An injected 503 is not mock drift. If it counted as drift, turning on chaos would tank your match rate and the number would stop meaning anything. Those pairs come out of the denominator, and the report says how many were suppressed rather than quietly shrinking.

They are never excluded from pass and fail. Whether the application absorbed the injected failure is the entire question you came to ask.

There is a third that surprises people: the recorded pair keeps its pre-chaos status. If the mock recorded a 200 and chaos sent a 503, the file on disk still says 200, because that pair is mock input for a later run and rewriting it would change what a re-replay does. The grid shows what the client actually received, with the recorded value alongside.

Reproducible, with the limit stated

seed= makes a run repeatable. The roll is a pure function of the rule, the request signature, and the occurrence count, with no clock in it: the Nth lookup of a given SKU always gets the same verdict.

It is not a promise that two runs are bit-identical. A run that issues a different number of requests for a signature diverges after that point. That is stronger than ordering-based reproducibility, which is worthless the moment the responder serves requests concurrently, and weaker than full determinism. In practice it means a failure you find this way is one you can hand to a teammate with the command that produced it.

What it does not do

It injects into the responses the responder serves for your application’s outbound dependencies. It does not touch inbound traffic, and it is HTTP only today. A rule whose scope selects gRPC or SQL reports that it matched nothing rather than silently doing nothing, and a rule that matched no traffic at all is called out at the end of the run.

It also does not replace killing pods. Infrastructure chaos and application chaos find different bug classes, and the case for running both has not changed.

Try it on a bug that is already there

The chaos lab is a storefront with a fallback cache that has never run, because nothing in its test suite makes its inventory dependency fail. One scoped rule makes inventory and only inventory fail, and the fallback path runs on demand.

The lab ships with a real bug. With its only dependency returning 503 on every call, the storefront reports degraded:false and source:"inventory" for every SKU. Nothing in the response contract changed, which is exactly why no status assertion and no response diff would have caught it.

The step-by-step walkthrough is in Application Level Dependency Chaos Testing, and the full reference is in the chaos documentation.