AWS re:Invent happy hour: Dec 1, 6 PM PST at MGM Grand RSVP

Eight demo requests pass on the baseline; changing the project total from 24 to 25 fails the replay gate.

Testing AI Code in CI/CD with a Verified Replay Gate

(Updated: )

An API can return HTTP 200 with the wrong answer. A CI check that only counts successful requests will miss that regression, whether a developer or an AI agent wrote the change.

For testing AI code in CI/CD, run the candidate against a reviewed recording and fail the job when its responses introduce a regression. Speedscale’s proxymock CLI supplies the dependency mocks, replays the recorded requests, and writes a verdict you can inspect. Keep your build, unit tests, and security checks alongside that gate.

This tutorial reuses the mock-lab Go example. Its committed traffic covers eight requests, including project queries and an authenticated order flow. You’ll save a baseline, introduce a wrong project count, and confirm the unchanged recording rejects it. No new demo application or live capture is needed.

What the replay gate checks

The candidate receives the same recorded inbound requests as the known-good version. Its outbound HTTP calls go through proxymock’s mock server, which serves the recorded dependencies. --no-passthrough returns an error for an unmatched call instead of forwarding it to the real service.

The gate compares response statuses and stable body values against the saved baseline. The example’s existing blueprint carries fresh tokens and order IDs between requests. Preserve that blueprint with the recordings. Variable fields need deliberate treatment: a normalized comparison does not mean every response byte is identical. See the replay verdict reference for the comparison rules and exit codes.

The baseline is part of the test specification. Review changes to it, the recording, the blueprint, and the gate script as carefully as changes to assertions in a unit test. Recreating the baseline from a failing candidate can approve the bug you’re trying to catch.

1. Start with the existing Go example

You’ll need Git, Bash, curl, Python 3, Go, and an activated proxymock CLI. Keep local ports 8080 and 4140 free. The commands below were checked with Go 1.27.3 and proxymock v2.5.1250 on macOS. The GitHub workflow later in this article targets Ubuntu on x86-64; it is a configuration example, not a report of a hosted Actions run.

Follow the installation guide and run proxymock init for a new workstation. Reuse an existing valid configuration if you have one. Non-interactive CI requires proxymock Pro or Speedscale Enterprise; free local activation does not grant CI access.

Clone a fresh copy so the deliberate failure won’t affect your own work:

git clone https://github.com/speedscale/mock-lab.git
cd mock-lab
git switch -c ai-code-replay-gate 382e4a79e1eb1ae65960f6bec0be8b4b9a546f65
cd languages/go
mkdir -p ci
printf 'proxymock/results/\n' >> .gitignore

Stay in languages/go for the local commands. Use the committed proxymock/recording and proxymock/blueprints directories. The fixture is synthetic demo traffic. For your own service, capture approved known-good behavior and remove sensitive data before sharing either recordings or replay artifacts.

LLM prompt for this step
Prepare the isolated mock-lab checkout at commit 382e4a79e1eb1ae65960f6bec0be8b4b9a546f65, enter languages/go, and create the CI script directory. Verify prerequisites and preserve any existing baseline before recording.

2. Save a reviewed baseline

Create ci/replay-gate.sh with this script. It builds the app, starts it behind fail-closed mocks, waits for readiness, and runs all eight recorded requests. Its exit trap stops the process it started. A missing baseline, missing requests, inactive required blueprint, or failed verdict stops the run.

#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."

mode=${1:-candidate}
gate=(--fail-if 'requests.failed != 0')
case "$mode" in
  baseline)
    test ! -e ci/baseline
    out=ci/baseline
    ;;
  candidate)
    test -s ci/baseline/replay-verdict.json
    gate+=(--baseline ci/baseline --fail-on-new-mismatch)
    ;;
  *) echo 'Usage: replay-gate.sh baseline|candidate' >&2; exit 2 ;;
esac

mkdir -p proxymock/results
run_dir=$(mktemp -d proxymock/results/ci.XXXXXX)
out=${out:-$run_dir/replayed}
go build -o "$run_dir/demo" .
proxymock mock --in proxymock/recording --no-passthrough \
  --out "$run_dir/mocked" --timeout 2m -- "./$run_dir/demo" \
  >"$run_dir/mock.log" 2>&1 &
mock_pid=$!
cleanup() {
  kill "$mock_pid" 2>/dev/null || true
  wait "$mock_pid" 2>/dev/null || true
}
trap cleanup EXIT

ready=false
for _ in {1..30}; do
  kill -0 "$mock_pid" 2>/dev/null || break
  if curl -sS -o /dev/null http://localhost:8080/ 2>/dev/null; then
    ready=true
    break
  fi
  sleep 1
done
if [[ "$ready" != true ]]; then
  cat "$run_dir/mock.log" >&2
  exit 1
fi

proxymock replay --in proxymock/recording \
  --test-against http://localhost:8080 --test-config regression \
  --require-blueprint 'mock-lab smart replace (token + order_id)' \
  --out "$out" --timeout 1m \
  --fail-if 'requests.total != 8' \
  "${gate[@]}"

python3 - "$out/replay-verdict.json" <<'PY'
import json, sys
with open(sys.argv[1]) as f:
    verdict = json.load(f)
if verdict['verdict'] != 'pass' or verdict['summary']['pairs'] != 8:
    sys.exit('Expected eight passing replay verdicts; inspect ' + sys.argv[1])
print('PASS: eight recorded requests, no response regressions')
PY

Create the baseline once from the unchanged app, then check the same app against it:

bash ci/replay-gate.sh baseline
bash ci/replay-gate.sh candidate

Both should exit 0 and print PASS: eight recorded requests, no response regressions. Inspect ci/baseline/replay-verdict.json and its observed responses before committing them. Baseline mode refuses to overwrite that directory. If the initial run fails, diagnose it before accepting a baseline.

The candidate command uses the explicit gate:

--baseline ci/baseline --fail-on-new-mismatch

A new per-request mismatch exits 3. Other nonzero codes can indicate a missed goal or an operational error; CI should fail on those too. The script also checks the final verdict and request count. requests.failed alone cannot detect a wrong response body, and an ungated replay can report mismatches while exiting successfully.

LLM prompt for this step
Review and implement ci/replay-gate.sh with separate baseline and candidate modes, finite startup waits, process cleanup, and explicit failure gates. Save the baseline once, run the unchanged candidate, and verify the candidate command cannot overwrite the reviewed baseline.

3. Prove a broken change fails

Change the aggregation in this fresh example so /api/stats reports one extra project:

python3 - <<'PY'
from pathlib import Path
p = Path('main.go')
source = p.read_text()
old = '"total":       len(projects),'
new = '"total":       len(projects) + 1,'
assert source.count(old) == 1, 'main.go differs from the pinned commit'
p.write_text(source.replace(old, new))
PY

bash ci/replay-gate.sh candidate

Expected result: exit 3, with a new mismatch on GET /api/stats. The response still has status 200; its total changes from 24 to 25. Inspect the newest proxymock/results/ci.*/replayed/replay-verdict.json, including bodyChanges and gate.exitCode.

These are the results from the local verification of this example:

App versionRequests replayedNew response mismatchesExit code
Unchanged candidate800
Candidate with len(projects) + 181, wrong /api/stats total3
Restored candidate800

Restore only that deliberate edit and rerun:

python3 - <<'PY'
from pathlib import Path
p = Path('main.go')
source = p.read_text()
old = '"total":       len(projects) + 1,'
new = '"total":       len(projects),'
assert source.count(old) == 1, 'main.go differs from the pinned commit'
p.write_text(source.replace(old, new))
PY

bash ci/replay-gate.sh candidate

Commit the reviewed ci/replay-gate.sh and ci/baseline directory to your example repository. Keep the recording and blueprint unchanged during this test. CI will run candidate mode only; it must not generate its own passing baseline.

LLM prompt for this step
In the disposable Go example, change the /api/stats total calculation by one exactly as shown, run the existing candidate gate, and verify a nonzero result caused by the count mismatch. Restore the original source and prove the gate passes again without regenerating its baseline.

4. Run the candidate gate in GitHub Actions

Fork mock-lab, push the ai-code-replay-gate branch with ci/replay-gate.sh and ci/baseline committed, and put this workflow at .github/workflows/replay-gate.yml in that fork. For your own repository, change working-directory to your service directory and commit its baseline first. It runs after a pull request is opened or updated. In your application pipeline, place replay after the build and unit tests and before deployment.

First create a GitHub environment named replay-ci, configure required reviewers, and add SPEEDSCALE_API_KEY as an environment secret. Review the exact code, scripts, baseline changes, and workflow before approving a run that receives that key. An environment name in YAML alone does not configure those protections; see GitHub’s environment setup instructions.

name: Replay regression gate
on:
  pull_request:
permissions:
  contents: read
jobs:
  replay:
    if: >-
      github.event.pull_request.head.repo.full_name == github.repository &&
      github.actor != 'dependabot[bot]'
    environment: replay-ci
    runs-on: ubuntu-latest
    timeout-minutes: 10
    defaults:
      run:
        shell: bash
        working-directory: languages/go
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
        with:
          go-version: '1.27.3'
          cache: false
      - name: Reject changes to the gate inputs
        env:
          BASE_SHA: ${{ github.event.pull_request.base.sha }}
        run: |
          git fetch --depth=1 origin "$BASE_SHA"
          git diff --exit-code "$BASE_SHA" -- ci proxymock/recording proxymock/blueprints
      - name: Install the verified CLI version
        run: |
          curl -fsSL \
            https://downloads.speedscale.com/proxymock/v2.5.1250/proxymock-linux-amd64 \
            -o "$RUNNER_TEMP/proxymock"
          printf '%s  %s\n' \
            269064db8c9867c3aedd6f96cae403095bd868fa0279b3d6987d18c34d0dd1d0 \
            "$RUNNER_TEMP/proxymock" | sha256sum -c -
          chmod +x "$RUNNER_TEMP/proxymock"
          echo "$RUNNER_TEMP" >> "$GITHUB_PATH"
      - name: Initialize the fresh runner
        env:
          SPEEDSCALE_API_KEY: ${{ secrets.SPEEDSCALE_API_KEY }}
        run: |
          test -n "$SPEEDSCALE_API_KEY"
          proxymock init --api-key "$SPEEDSCALE_API_KEY" --yes --no-rcfile-update
      - name: Verify candidate responses
        run: bash ci/replay-gate.sh candidate
      - name: Remove stored credentials
        if: always()
        run: rm -rf ~/.speedscale
      - name: Preserve replay evidence
        if: always()
        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
        with:
          name: replay-results
          path: |
            languages/go/proxymock/results/ci.*/replayed/replay-verdict.json
            languages/go/proxymock/results/ci.*/mock.log
          retention-days: 7
          if-no-files-found: warn

The action pins were checked against the official checkout, setup-go, and upload-artifact releases. The CLI download is versioned and checksum-checked. Keep those pins under review as supported versions change.

The gate-inputs step fails any pull request that edits the script, baseline, recording, or blueprint, so a change cannot approve its own regression. Refresh those files in a separate, reviewed change that runs outside this job. proxymock init stores the API key on the runner, and the next steps build and run pull-request code that could read it. The cleanup step removes the stored key, but it does not stop code that already ran, so keep required reviewers and an ephemeral runner. The artifact holds only the verdict and mock log; the verdict can include response values, so keep real-traffic runs restricted.

Fork and Dependabot PRs are excluded from this secret-bearing job. Run unprivileged checks for them, then use a maintainer-reviewed replay run when appropriate. A skipped replay provides no passing replay evidence. A same-repository branch still needs review before secret access. Follow GitHub’s secure-use guidance; do not switch to a privileged pull_request_target workflow that executes untrusted PR code.

LLM prompt for this step
Adapt the GitHub Actions candidate replay gate to my repository, keeping the reviewed recording and baseline fixed. Use the required secret only in the intended workflow context, bound execution, retain artifacts on failure, and verify the broken-count case blocks CI.

What this proves before deployment

A passing replay means this candidate preserved the checked behavior for these eight recorded requests and mocked dependencies. That is a useful check for code produced in Cursor, Copilot, or another coding agent. It does not establish complete correctness or coverage of paths the recording never exercised.

Keep unit tests for calculations and edge cases, security checks for vulnerable code and dependencies, and live integration tests for current third-party contracts. This mock-lab example has no Go unit-test suite; the replay is additional executable coverage, not evidence of unit-test coverage. Performance needs a separate workload and explicit thresholds. This eight-request functional run is not a load test.

When behavior changes intentionally, review the expected responses and recording diff before refreshing the baseline. Keep failed reports, and restrict artifact access and retention when using real traffic. Never use --exit-zero, continue-on-error, or a swallowed shell exit status on a required gate.

For the full product workflow, see AI code verification, the proxymock task selector, and the CI/CD setup reference.

Run this workflow with an LLM

Give this prompt to a coding assistant with access to your project and this article. Supply environment-specific values it cannot discover.

Whole-workflow LLM prompt
Read https://speedscale.com/blog/testing-ai-code-in-cicd-made-simple-for-developers/ and inspect my project instructions first. Run the pinned mock-lab Go CI replay-gate tutorial. Preserve a reviewed baseline, verify the unchanged candidate passes, introduce the intentional GET /api/stats count bug, and prove the same gate fails before reverting it. Add the verified candidate gate to GitHub Actions with secrets and artifacts handled explicitly. Check installed tool help before adapting commands. Report the checks actually run, their results, and anything you could not verify.

Stop writing API mocks by hand

proxymock records real traffic from your running app and replays it as mocks for HTTP, gRPC, Postgres, Kafka, and more. Install locally, activate once, and start recording.