---
title: "Inside Kraken: Regression Testing With Traffic Replay | Speedscale"
description: "See how Kraken combines traffic replay, stable dependency mocks, and evolving regression coverage to catch Speedscale release failures before production."
canonical: "https://speedscale.com/blog/kraken-traffic-replay-regression-testing/"
---

![Kraken replays recorded traffic against a candidate build with stable dependency mocks and checks the resulting behavior.](https://speedscale.com/_astro/featured-image.BINryh2t_Z2ucvWS.webp)

# Inside Kraken: Regression Testing With Traffic Replay

Oct 26, 2026

[By Matt LeRay](https://speedscale.com/blog/author/matt-leray/)

[API testing](https://speedscale.com/blog/category/api-testing/) [DevOps](https://speedscale.com/blog/category/devops/)

* * *

A readiness check should make a release safer. During one release validation, ours kept Kraken’s traffic replays stuck in `Initializing` until they timed out. The operator was asking Kubernetes for EndpointSlices. Its installed permissions did not allow that request.

The code needed a real cluster, the generated permissions, and a replay trying to start before that mistake became obvious. Kraken put those pieces together during release validation. We removed the unnecessary lookup and kept the readiness checks on the responder deployments and target workloads.

I would love to say our tests make that kind of mistake impossible. Kraken exists because we keep finding ways to make them.

🎯 Key Takeaways

*   Kraken is Speedscale’s internal release-validation system. It runs traffic replay and other integration checks against candidate components in Kubernetes.
*   Recorded dependency responses give regression tests repeatable inputs while the real application and replay infrastructure execute.
*   Coverage stays current through versioned fixtures, fresh capture validations, and new scenarios for changed behavior. Updating a baseline still requires judgment.

## Why we built Kraken

Speedscale records application traffic, turns dependency interactions into service mocks, and replays requests against another version of an application. We need to test that entire path before asking customers to trust it.

Unit tests cover individual pieces. They cannot, by themselves, establish that the installed operator has the right permissions, a captured database conversation survives analysis, or a responder serves the response a real client expects.

Those failures often live between components. Each component can look reasonable in isolation while the assembled system fails.

Kraken adds a regression layer around that assembled system. Its repository contains release manifests, Kubernetes workloads, recorded traffic, test configurations, and executable validation scripts. GitLab CI brings those pieces together against our staging backend and Kraken cluster.

The question is practical: can this set of candidate binaries and containers still capture, analyze, mock, and replay the scenarios we rely on?

That extends the [traffic replay workflow](https://speedscale.com/blog/definitive-guide-to-traffic-replay/) into our own release process. We use the product to exercise the product, alongside targeted checks for installation, configuration, and other behavior a replay alone would miss.

## How traffic replay runs inside Kraken

The release manifest identifies the component versions under validation. Setup installs the operator and prepares the cluster. Individual validators deploy their workloads, create or load snapshots, run their checks, and inspect the results.

The core replay path looks like this:

```
flowchart TD
    V[Candidate component versions] --> K[Kraken CI and cluster setup]
    F[Recorded traffic and test configuration] --> S[Snapshot analysis]
    K --> O[Operator provisions a replay]
    S --> G[Generator sends recorded inbound requests]
    S --> R[Responder loads recorded dependency responses]
    O --> G
    O --> R
    G --> A[Real application under test]
    A <-->|Dependency exchange| R
    A --> C[Response comparison and validation report]
    C --> J[Kraken assertions determine the job result]
```

The generator supplies inbound requests. The application executes its real code. Its outbound calls go to the responder for the dependencies selected for mocking. The responder matches those calls to recorded interactions and returns the corresponding responses.

The operator, generator, responder, capture path, and analysis services remain real components under test. Mocking an application’s dependencies gives us control over part of the environment without replacing the machinery we need to validate.

Consider Kraken’s Postgres capture validation. It starts a Postgres server and a Java service, sends writes and reads through that service, and captures the database conversation. The validator checks that the snapshot contains authentication, bound statement values, and returned rows. It then replays the application requests with outbound dependencies mocked.

That replay requires a passing report and a `matchPct` of 100. The read endpoint returns the rows supplied by the database mock, so the response comparison checks more than whether an HTTP server answered. It exercises capture, storage, analysis, mock serving, and application behavior together.

## Stable mocks make failures explainable

A regression comparison becomes harder to interpret when the inputs change between runs. A live dependency might return a different record, rotate a token, or fail temporarily. The application changed too. Now several explanations compete for the same failure.

Recorded dependency responses let us hold that part of the experiment steady. Given the same scenario and matching configuration, the application receives the responses represented in the recording.

Speedscale’s [service mocking](https://docs.speedscale.com/concepts/service_mocking/) uses request signatures to select recorded responses. Those signatures distinguish interactions using protocol and request information. Matching rules matter: an overly broad signature can give different requests the same answer, hiding a regression.

Stability also requires handling values that should change. An expired JWT should not turn every replay into an authentication failure. Our fixtures include transforms for cases such as [re-signing JWTs](https://docs.speedscale.com/guides/replay/resign-jwt/). Those transforms belong to the test configuration and need review alongside the recording.

The same applies to response assertions. Ignoring a volatile timestamp can make a comparison useful. Ignoring a business field because it changed can remove the very check that would have caught the bug.

This is the practical value of [controlling a dependency with a mock](https://speedscale.com/blog/what-mocks-are-actually-for/): a failure has fewer moving parts to investigate. It does not make the entire Kubernetes environment deterministic, and it does not establish that a real external API still accepts the request.

## Keeping regression coverage current

Stable recordings and current coverage solve different problems. We want a scenario to remain repeatable while the suite grows with the product.

Kraken maintains both through two complementary paths.

**Committed fixtures preserve known scenarios.** Setup uploads test configurations, user data, and transforms from the repository. It creates replay snapshots from committed raw traffic and passes their generated IDs to downstream jobs. A new snapshot ID means the current analysis path processed the fixture; it does not mean the underlying recording came from today’s traffic.

**Live capture validations exercise the current pipeline.** Tests such as the Postgres validation deploy a workload, generate interactions, capture them, inspect the snapshot, and replay it. These checks test whether today’s candidate components can still turn a real protocol conversation into working mocks.

```
flowchart TD
    A[Known regression scenarios in Git] --> B[Build snapshots with candidate components]
    C[Deploy workload and generate fresh traffic] --> D[Capture and inspect a new snapshot]
    B --> E[Replay and check expected behavior]
    D --> E
    E --> F[Preserve evidence and investigate failures]
    F --> G[Review fixes and add regression coverage]
    G --> A
```

New validation directories are discovered by the pipeline generator when they contain an executable `validate` script, unless explicitly excluded. That makes adding durable coverage straightforward: put the scenario and its assertions in the repository, and CI can run the same script engineers use locally.

This is what I mean by keeping Kraken up to date. We exercise current builds, regenerate snapshot artifacts, and add coverage as capabilities evolve. We still choose which interactions matter. A refresh cannot invent an unrecorded edge case.

When an API intentionally changes, its expected behavior needs review. Re-recording immediately after a failure can bless the regression. Keep the failing evidence, establish whether the change is intended, then update the fixture or assertion with that decision visible in review.

## From replay failure to release decision

The EndpointSlice failure at the start of this post surfaced a mismatch between code and the permissions delivered with it. A separate Kraken validation exposed another deployment assumption: the operator tried to synchronize a cache for Argo Rollouts in a cluster without that optional resource definition.

The operator exited during startup. During its restart window, replay creation failed because the operator webhook service had no available endpoints.

The recorded errors included:

```
failed to start operator, exiting
no endpoints available for service "speedscale-operator"
```

Those are excerpts from the failure, not a sample of a healthy run. The fix made operator startup tolerate the missing optional Argo Rollouts definition. A real cluster without that definition was the condition that exposed the assumption.

Both examples explain why we keep this layer close to releases. Customers install containers, permissions, and configuration together. Kraken can expose failures in that combination while a candidate is under validation, before we promote it further.

A failing job still needs diagnosis. We start with three questions:

1.  Did setup and provisioning complete, or did the replay fail before requests could run?
2.  Did the application produce different responses, or did the mocks fail to match its dependency calls?
3.  Did the configured goals and the validator’s own checks actually evaluate the behavior we intended?

Kraken’s report assertions check completion, required metadata, configured goals, and other scenario-specific evidence. The gRPC replay validator, for example, checks the report status and assertion results rather than treating a completed command as success.

We also keep an expected-failure replay in the matrix. Its configuration demands a `passAssertPct` above 100, and the validator inverts the expected result. That deliberately impossible goal checks that the validation path recognizes failure. A gate needs evidence that it can turn red.

Release decisions depend on which jobs are required. Kraken includes checks with explicit exclusions or allowed failures, so a green pipeline needs that context. A disabled check provides no protection, however valuable its script might be when run manually.

The [CI/CD integration guide](https://docs.speedscale.com/guides/integrations/cicd/) covers the general pattern for adding replay to your own pipeline. Kraken applies it to a product whose capture and replay components are themselves changing.

## What we learned running Kraken

The most uncomfortable lesson was a chaos replay that passed without applying chaos. Its recording contained inbound requests but no outbound dependency interactions. The responder had nothing to perturb. Ordinary goals and report checks could still pass.

We added an assertion that configured chaos must leave evidence in the responder’s recorded responses. We also retained the original report as a regression fixture. That checks the behavior the test was supposed to exercise, rather than assuming a configuration setting proves execution.

This was a coverage failure we discovered and corrected. It is a reminder to inspect what a green result establishes, especially when adding a feature to an existing scenario.

Infrastructure noise needs similar care. A transport timeout and an assertion mismatch require different responses. Kraken’s shared helpers retry selected transient API failures; they do not retry a returned validation failure until it happens to pass. Performance measurements also need suitable isolation. A benchmark competing with other validators on shared nodes can measure its neighbors as much as the candidate.

If you build a similar system, start with one important workflow. Record its inbound requests and dependency responses, establish a passing baseline, then introduce a deliberate behavioral change and confirm the test catches it. Add that replay to CI before expanding the suite.

Kraken has given us a place for integration bugs to fail while we can still act on them during release validation. Stable mocks make those failures easier to investigate. Maintained scenarios make the protection useful as the product changes.

The release that sits in `Initializing` is frustrating. Discovering why there, before promoting it to production, is the point.

## Table of Contents

## Capture and replay your production traffic

Speedscale records live traffic from your Kubernetes services and turns it into tests, mocks, and load tests automatically. 30-day free trial, full feature set.

[Start free trial](https://app.speedscale.com/signup) [Book a demo](https://speedscale.com/company/demo/)