Part 3: Designing a Reliable API Traffic Replay System
An API can return HTTP 200 with the wrong answer. In the existing proxymock CI tutorial, changing a project total from 24 to 25 leaves the status unchanged but fails the replay gate. That is the kind of regression a traffic replay system needs to expose.
This is Part 3 of the series on capturing traffic and transforming traffic. It explains how the replay components fit together. The linked tutorial supplies the runnable Go example and CI script.
Traffic replay Q&A
What is traffic replay?
Traffic replay sends saved requests to a running application and compares its responses with expected behavior. It can catch regressions on the recorded paths, including a wrong response body behind HTTP 200. Its coverage depends on the requests and assertions in the test.
What makes traffic replay hard?
Requests may depend on missing database rows, expired credentials, or IDs generated by earlier requests. Prepare the test data and carry fresh values through the workflow. Compare stable business values while handling volatile fields such as timestamps explicitly.
Should I replay against real dependencies or mocks?
Use recorded mocks to isolate application behavior. Use real test dependencies when database semantics or their current contracts matter. Check the application’s routes in either case: starting a mock server does not prevent connections that bypass it.
How do you know a replay passed?
Confirm that every intended request ran and that its required response checks passed. Inspect dependency calls and the process exit status. Prove the gate rejects a deliberate wrong result before relying on it in CI. A successful HTTP status or a high aggregate match rate is insufficient.
Start with a failure the system must catch
The CI tutorial uses eight recorded requests from the mock-lab Go application, including GET /api/stats. Its controlled verification used proxymock v2.5.1250 and synthetic traffic. The unchanged candidate passed; the wrong total produced one stable-body mismatch and exit 3; restoring the candidate passed again.

For that request, each part of the system has a concrete job:
| Component | Job in the wrong-total example |
|---|---|
| Traffic store | Supply the same recorded GET /api/stats request for both versions. Retain the recording’s request identity. |
| Replay orchestrator | Start the candidate behind its dependency mocks, wait for readiness, and send all eight requests. |
| Response validator | Compare the observed status and stable body values. Detect total: 25 where the accepted result is total: 24. |
| Results reporter | Save the per-request mismatch and return a failing exit status to CI. |
The store also holds the captured outbound traffic used by the mocks. These responsibilities can share a process, but the expected result must remain independent of the candidate. Regenerating expectations from the broken version would accept 25 as correct.
With proxymock, --baseline points to an earlier replay output directory, including observed responses and replay-verdict.json. It does not point to the source recording. The verdict reference describes the comparison rules and gate options.
Choose what the replay should exercise
Direct replay against test dependencies
Send captured requests to a test instance of the application, with the real test services and database needed for that workflow. This is useful when you need to exercise an integration’s current behavior, such as whether the application still works with an upgraded database.

Both reads and writes can work if the environment has suitable data and cleanup. Set the destination explicitly so recorded production addresses cannot become accidental test targets. Replaying at HTTP level also requires the method, path, query, body, and relevant headers; a k6 script cannot consume an arbitrary PCAP as a ready-made application test.
Shadow live traffic
Shadowing copies live requests to a candidate while the primary service handles the originals. It exposes the candidate to the current request mix, without waiting for a saved recording to be replayed.

Copies can still trigger writes, emails, or external calls. Give the candidate separate state and controlled dependency routes. To compare responses, the system needs to collect both results and correlate them to the same original request. Copying traffic alone does not provide that comparison.
Replay with recorded dependency responses
Run the application while mocks supply responses for the dependencies you want to hold constant. In the wrong-total example, this helps test the application’s response logic without depending on a live downstream API.

A disposable database can remain real when SQL behavior matters. A database mock can instead exercise application behavior under captured database responses. Choose that boundary according to the change: a recorded database response cannot establish that a new query works against the real database.
With proxymock, --no-passthrough rejects unmatched requests that reach its proxy. A connection that bypasses the proxy still needs routing or network controls. Inspect the mock results as well as the inbound replay verdict.
Prepare state before sending requests
A request to update order_123 needs that order to exist. Seed committed fixtures, create the order through earlier requests, or mock the database response, depending on the test boundary. Check that the application can read the initial state before starting the workload.
A load generator’s database transaction cannot roll back writes made through the application’s separate connections. Uncommitted fixture inserts may also be invisible to those connections; see PostgreSQL’s transaction visibility documentation. Reset owned test data or dispose of the database after the run.
Authentication and generated IDs need a different treatment. The local Go walkthrough initially matches six of eight requests. Its token/order blueprint carries a fresh access_token and order_id into later requests so the protected workflow can complete. Replacing every ID with an arbitrary fixed value would lose that relationship.
When requests run concurrently, partition mutable state by session or actor. Two virtual users updating the same order can interfere even when their client-side variables are separate. k6 virtual users and scenarios control execution; they do not create isolated application accounts or database rows.
Schedule requests without breaking the workflow
A login must finish before a later request can use its token. Preserve that causal order within a session, then parallelize independent sessions.
Timing needs its own decision. If a runner waits for each response and then sleeps for the recorded gap, response time gets added to the schedule. A slow candidate will receive requests more slowly. Scheduling requests against offsets from the start of the recording avoids that particular drift, but the runner still has to handle dependencies between requests and limits on concurrency.
Record whether the run preserves original timing, caps long gaps, or increases playback speed. Those choices change the load presented to the application. A functional replay can prioritize reproducing a workflow; a capacity test needs a load model that matches the performance question.
Make response checks fail the job
For GET /api/stats, checking status 200 is only half the example. The validator must also check total. Error cases need their own expected statuses: a recorded negative test may correctly return 400 or 404.
In k6, check() records outcomes without making failed checks fail the process. Add a threshold for checks that must gate CI. These two excerpts belong in a runner that supplies response and expected; they illustrate the validation policy rather than a complete replay script.
export const options = {
thresholds: {
'checks{scope:replay}': ['rate==1'],
},
};
Import check from k6 and apply the same tag when validating each response:
check(response, {
'status matches expected': (r) => r.status === expected.status,
'total matches expected': (r) => r.json('total') === expected.total,
}, { scope: 'replay' });
The tagged threshold requires all emitted replay checks to pass. The runner must separately detect missing requests and setup or execution errors. A response check cannot tell you that another request never ran.
Mock configuration needs comparable care. Matching only method and path can return the same fixture for requests with different bodies or query parameters. If you use WireMock, its request-matching reference describes the available matchers. Confirm the mappings were accepted before replay starts.
In k6, local fixture loading with open() belongs in initialization; HTTP calls to configure mocks belong in setup(). setup() and teardown() run once per test. Handle cleanup on setup failures, because teardown will not run if setup throws. The lifecycle documentation covers these boundaries.
LLM prompt for this step
Add tagged k6 replay checks for expected status and the meaningful total response field. Configure a threshold requiring every scope=replay check to pass, then run the known-good and intentionally broken cases and verify their process exit codes differ.
Run the system in an order you can diagnose
Define the acceptance policy before execution, then bring up the dependencies and candidate. Wait for readiness and confirm the application’s database and outbound routes. Only then send the workload. Evaluate the response and performance results after it finishes, and save the evidence before cleanup.
| Failure | What the result should tell you |
|---|---|
| Candidate never becomes ready | Setup failed; the application behavior was not tested. |
| Only seven of eight requests complete | The workload is incomplete, even if every received response matches. |
All eight run, but the project total is 25 | The response regressed; identify the request and changed field. |
| Responses match, but a required dependency call bypasses the mock | The isolation requirement failed; the response verdict alone is insufficient. |
For proxymock, enable the gate you intend to enforce and retain replay-verdict.json. An ungated replay can report differences while exiting successfully. CI must preserve nonzero exit statuses, including operational failures; --exit-zero defeats that requirement.
Associate each run identifier with its recording, transform settings, candidate revision, and results. Save failed-run artifacts too, so someone investigating a red job can see the request and observed response. Review changes to the baseline and assertions separately from the application change they are judging.
Treat performance as a separate test requirement
A high match rate can hide the one checkout regression that matters. A single latency percentage can likewise hide different endpoint requirements. Use response assertions for correctness and the service’s latency, throughput, and resource budgets for performance.
State the concurrency, duration, resource allocation, and timing model alongside a performance result. More load can saturate the generator or dependency mock before it saturates the application. A replay with recorded dependencies also measures a different boundary from a run against live test services.
Production recordings will not contain every new requirement. Add cases for new endpoints and error paths, but check their expected results against the requirement before accepting them. The candidate’s output should not automatically become its own expected result. Remove sensitive data before sharing fixtures or reports.
Try the existing eight-request gate
Use the mock-lab CI tutorial to save a known-good baseline, introduce the wrong total, and inspect the failure. Its script checks that all eight requests ran, requires the token/order blueprint, and propagates the replay gate’s exit status. The local Go walkthrough covers recording and correlation first if you need that setup.
These are the existing tutorials’ results; the architecture described here has not been implemented as a separate system or benchmarked. The tutorials document runtime prerequisites and non-interactive CI activation requirements.
Once the deliberate failure is caught, apply the same workflow to one change in your own service. Choose a request whose business result matters, confirm the expected response, and make CI reject the wrong answer.
Run this workflow with an LLM
Give this prompt to a coding assistant with access to your project and this article. Supply environment-specific values it cannot discover.
Whole-workflow LLM prompt
Read https://speedscale.com/blog/how-to-build-a-traffic-replay-system/ and inspect my project instructions first. Implement the existing eight-request replay gate against my local test target. Define the failure it must catch, preserve a reviewed baseline, prepare request state, isolate dependencies as intended, and make both transport errors and semantic response mismatches fail CI. Prove the unchanged baseline passes and a known broken response fails. Check installed tool help before adapting commands. Report the checks actually run, their results, and anything you could not verify.