On October 20, a power failure in one of Google Cloud’s three London zones disrupted Revolut, Lloyds, HSBC and others for hours. One zone failed, errors spread region-wide, and Google warned customers they might not be able to create new workloads to fail over to.
Most teams find out how resilient they are during an outage, not before one. These five questions let you find out first.
1. Know your blast radius
You can’t protect what you can’t see. Before you test anything, you need an honest picture of what your system depends on.
Question 1: What do our services really depend on?
Your architecture diagram shows the dependencies you chose. It rarely shows the ones you inherited: the auth provider your API gateway calls, the managed queue a library writes to, the payment processor’s own cloud region. Outages hide in those transitive dependencies.
Question 2: Which of those dependencies are single points of failure?
Once you have the list, ask where each one runs. A vendor hosted in one zone, a database without a replica in another, a config service every pod calls at startup: any of these can take you down even when your own services are spread across zones.
In London, one zone failed and the whole region saw errors.
How to answer it
Diagrams go stale. Network traffic doesn’t. Capturing real API traffic between your services shows every upstream and downstream call your application makes, including third-party APIs, managed databases and identity providers that never made it onto a whiteboard.
From that traffic you can build a dependency map based on observed behavior instead of memory. Then flag every call that leads to a single zone, region or vendor. That list is your blast radius.
2. Prove your failover works
Most teams have designed for resilience. Far fewer have tested it. A failover that has never run is a hypothesis.
Question 3: When did we last simulate losing a zone, and with what traffic?
If the answer is “never” or “at last year’s game day,” you’re not alone. And if the test used a handful of synthetic requests, it proved less than you think. Hand-written scripts cover the paths someone thought to write. Real outages break the paths nobody thought of.
In London, Google’s first update took two and a half hours and offered no workaround.
Question 4: What happens when a dependency is slow, not down?
A dependency that fails fast is easy to handle. One that hangs for 30 seconds is not. Slow responses exhaust thread pools, pile up connections and trigger retries that make everything worse. Many “outages” are really cascading slowdowns.
How to answer it
Replay recorded production traffic into a pre-production environment. You exercise the real mix of requests, edge cases included, without touching production.
Pair that with service mocks built from the same recorded traffic. Now any dependency can return errors, time out or respond slowly on demand. Remove a zone, degrade a database, slow down the auth provider, and watch what your system does.
Because it’s all simulated, you can rehearse “this zone just disappeared” as often as you like, without coordinating with other teams or waiting for the next real incident.
3. Plan for the comeback
The outage isn’t over when the status page turns green. Recovery is its own stress test, and teams almost never rehearse it.
Most teams plan for the outage itself. The surge after it is where a second failure often starts.
Question 5: Can our systems handle the surge when everything recovers at once?
When a zone comes back, it doesn’t return to normal traffic. Clients retry everything they queued. Message backlogs flush downstream all at once. Autoscalers that scaled down during the outage race to catch up. The result can be a second outage, caused by the recovery from the first.
Meanwhile your surviving zones may have been carrying far more load than they were sized for. If new capacity can’t be created during the incident, whatever you pre-provisioned is all you have.
In London, Revolut was still degraded after Google closed the incident.
How to answer it
Recovery traffic doesn’t look like normal traffic, so don’t test with normal traffic. Replay captured production traffic at two, five or ten times its usual volume to reproduce the spike. Watch whether autoscaling keeps up, connection pools hold and rate limits protect downstream services.
Then make it routine. Build these replay tests into your CI/CD pipeline so every release is checked against a recovery surge before it ships, not after the next outage shows it can’t handle one.
Don’t wait for the next outage to test you
The tools to answer all five questions exist: capture real traffic, replay it, mock what you can’t control, and scale it up. Here’s the checklist to bring to your next team meeting:
- ☐ What do our services really depend on, including transitive dependencies?
- ☐ Which of those dependencies are single points of failure?
- ☐ When did we last simulate losing a zone, and with realistic traffic?
- ☐ What happens when a dependency is slow, not down?
- ☐ Can we handle the surge when everything recovers at once?
You don’t need to answer all five this week. Pick one and test it this quarter, before the next outage tests it for you.