Your OpenAPI Spec Is Not a Test
Someone asked me how to add tests to an endpoint that had never seen traffic. I answered in about four seconds. “Point proxymock generate at the OpenAPI spec. Done.”
Then I ran that advice against our own tutorial service.
Four of the twelve generated tests sent placeholder values the service rejects. The tests for the 400 and 422 error paths came back 201 Created. The spec described error responses. It did not describe requests that cause them.
My four-second answer produced a test suite that would have blessed the wrong behavior.
The spec is a driver
Here is the rule I should have started with. Every starting point you have is a way to send requests. None of them is the test.
The test is what the service actually does when those requests arrive. So send them while proxymock records the service, and keep the recording. It holds the real responses, and every call the service made to its database and its APIs. That one directory becomes both the regression test and the mocks.
Once you see it that way, the question changes. It stops being “do I have a spec?” and becomes “what is the cheapest thing in this repo that sends realistic requests?”
Ask the repo first
So we built the question into proxymock. proxymock inventory scans a service directory and ranks what it finds. The ranking puts realistic values ahead of convenient ones:
- An existing recording
- A Speedscale cloud capture of the same service
- Traffic already in the repo: Postman collections, HAR files, k6 scripts, e2e suites, traffic drivers
- An OpenAPI spec, then protobuf, then a GraphQL schema
- The code itself
The code is always on the list. Whatever the base, the new endpoint is never in it, so your coding agent writes those requests from the handler.
On the mock-lab tutorial service, the inventory found the OpenAPI spec and ranked it below the traffic drivers, one per language port. Trimmed:
BASE 2 repo-traffic available found 4 traffic drivers
go/cmd/traffic
java/src/main/java/com/speedscale/tutorial/traffic
node/traffic.mjs
python/traffic.py
3a openapi available 1 openapi file
contract/openapi.yaml
FILL 4 code available the routes and handlers in the code; always the source for the new behavior
recommended: base 2, filler 4The authors had already written a traffic driver, error cases included. Their requests beat anything a generator would guess.
Generated tests lie politely
When the spec is the best you have, generate still earns its place. Just stop treating the output as finished. proxymock now lists the tests whose requests still carry placeholders. The file list is trimmed:
PLACEHOLDERS: 4 of 12 inbound test(s) still send generated placeholders ("example_value" or ${{param:...}}).
Generated inbound tests are drivers, not a baseline: replace the placeholders with real values
before replaying them at a real implementation or recording them through proxymock record.Fix those, delete the ones you cannot fix, and write the error cases by hand. The 201s taught me that one.
Let the agent do the boring part
I gave Claude Code one prompt: bootstrap a regression test for the tutorial service with the quality-loop skill. It ran the inventory and recorded the service with the driver. Then it applied four mock match-rate fixes and a strict test config until the gate passed on unchanged code.
The recording had three hosts in one directory: the service’s inbound requests, the CNCF projects API it calls, and Postgres. Nobody hand-wrote a mock.
Then the part that matters. It planted a change, watched the gate fail with exit code 3, and put the code back. A gate you have never seen fail is a rumor.
Both prompts in the guide ran headless in a fresh session for about $13 of tokens together. That is less than the coffee I drank arguing about whether generated tests were good enough.
The endpoint nobody recorded
Next I added GET /orders/{id}/receipt. The old gate stayed green, and it would have stayed green if the receipt were wrong. No recorded request touched it.
This time the inventory picked the existing recording as the base. The agent re-ran the original driver and added requests for the receipt: one order created and its receipt read, an unknown order, and a malformed ID. It did not replay the old recording, because that would have minted new order IDs the later requests never knew about.
Then I broke the receipt on purpose. It counted lines instead of items:
NEW MISMATCH: GET /orders/6008e18a-2c7e-4aab-a086-a93f896800be/receipt status 200, body items 3 -> 2
Goal verdict: FAILOne wrong number, named by field.
Commit the baseline too
Teams get the recording into Git and forget the baseline replay. Then CI rebuilds the baseline from the same commit it is testing, compares the code to itself, and passes forever.
Commit the recording, the blueprints, the test config, the baseline and the gate script. Then CI needs no agent at all.
If your base is a production capture, review it before a model or Git sees it. That is the first question every enterprise security team I work with asks, and they are right to.
What I tell people now
I still use proxymock generate. I just stopped calling its output tests.
The full walkthrough has both prompts, every command and the output from a real run on the mock-lab tutorial service.
Frequently asked questions
How do I write a regression test for an endpoint that has never been recorded?
Send it requests while proxymock records the service. The requests can come from a traffic driver, a Postman collection, an OpenAPI spec, a production capture, or requests your coding agent writes from the handler. The recording holds the service's observed responses and captured dependency exchanges, and can be used as tests and mocks after review and tuning.
Can I use tests generated from an OpenAPI spec as a baseline?
Not directly. Generated requests carry placeholder values like example_value and templated path parameters, and requests generated for error statuses are often valid requests. Use them as drivers: fix the placeholders, send them through the recorder, and let the real responses become the baseline.
What should I commit for a recorded regression test?
Commit the recording, the blueprints and test config that make it replay cleanly, the reviewed baseline results, and the gate script. A baseline regenerated from the candidate cannot detect behavior changes relative to the previous build, so keep an independently reviewed reference for regression comparisons.