The query count in our snapshot drawer went blank, and the unit test for that request kept passing. When we went looking in recorded traffic, the API gateway had rejected that request 210 times in a single recording. Every one of those rejections came back as 200 OK.
So we searched the rest. We record our own API gateway with Speedscale, and two recording windows from November 3 to 5 (about 38 hours between them) held 1,238 responses with gRPC status INVALID_ARGUMENT. All 1,238 returned HTTP 200. About two-thirds came from our dev environment, much of it QA testing filters on purpose. Another 382 came from the production dashboard and our playground. Engineers’ local builds sent 43, and 28 had no Origin header.
Why a rejection looks like a success
Our dashboard talks to the gateway over gRPC-web. Here’s one of the captured exchanges, trimmed to the lines that matter:
POST /rrpair.Speedscale/RetrieveTrace HTTP/2.0
Content-Type: application/grpc-web-text
HTTP/2.0 200 OK
Grpc-Status: 3
Grpc-Message: could not retrieve trace rrpairs: failed to validate filters: service filters must not be provided when getting snapshot trafficgRPC-web carries the real result in a header, and status 3 is INVALID_ARGUMENT. Search the recording for 4xx and 5xx responses and you’ll walk right past it. proxymock writes HTTP and gRPC exchanges as Markdown, so the search that finds all 1,238 is a grep:
grep -rliaE '^grpc-status: [1-9]' proxymock/What two days of rejections traced back to
We grouped the rejections by error message and checked each group against current code. A recorded error isn’t automatically a bug, so a group only counted once we could point at the code that sent the bad request.
| What someone did in the dashboard | Rejections | What they saw |
|---|---|---|
| Opened a request in a snapshot | 222 | A blank query count |
| Typed a regex filter that didn’t parse | 150, all on dev | “No traffic” |
| Watched the live mocked-response rate | 68 | No rate at all, including the count that had loaded |
Opened an old link with ?service= in the URL | 14 | “No traffic” |
| Picked a time range longer than 30 days | 38 | “No traffic” |
The largest group isn’t in the table: 649 rejections of overlong session IDs, all from QA on dev. QA had already filed that bug before we searched the traffic, so we didn’t count it as a find.
All five in the table showed the user less data instead of an error. The gateway explained what was wrong every time, in plain English, and the dashboard threw the message away. Fixes for all five have since merged.
Three bugs, one empty grid
The Traffic page’s list hook caught a failed query and returned an empty array. The grid couldn’t tell an empty array from a time window with no traffic in it, so it showed its normal “No traffic” overlay. The 30-day range, the legacy ?service= URL and the bad regex all ended up there. The live-rate panel failed a different way: it fetched its counts together, and when one was rejected it dropped all of them.
Both paths are fixed now. A failed query puts “Traffic could not be loaded” in the grid, and the rate panel keeps the count it loaded when the second, scoped count is rejected.
One of them up close: the mock that agreed with the bug
The snapshot query count is the case where the tests should have caught it, and didn’t. The unit test asserted the exact filter list the gateway was rejecting:
expect(got).toEqual(['service', 'snapshotID', 'timeRange', 'database']);Its mock answered that request with spans. The code and the test were written from the same assumption, so the mock agreed with both.
We gave a coding agent the recording through proxymock’s MCP server and the traffic-to-code skill, running on a small, inexpensive model. It searched the recording and read both captured exchanges in full before touching any code, then proposed dropping the service filter whenever the query is scoped to a snapshot:
- const filters = [createPredicate('service', 'is', rrpair.service)];
+ const filters = snapshotId ? [] : [createPredicate('service', 'is', rrpair.service)];That run cost $0.014 across 18 model calls. The one-line fix was right and incomplete. Automated review on the merge request found that a snapshot can hold several services, so dropping the filter could attach another service’s queries to the wrong request, and that the 500-row cap made some counts look exact when they weren’t. Both follow-ups went in before merge. The recording never pointed at either one.
The test mock changed too. It used to return spans for any filter list. Now it rejects the same combination the gateway rejected, with the gateway’s own message:
export function snapshotScopeError(expression?: FilterExpression): string | undefined {
const kinds = filterCriteria(expression).map((item) => item.oneofKind);
return kinds.includes('snapshotID') && kinds.includes('service')
? 'service filters must not be provided when getting snapshot traffic'
: undefined;
}It’s still a rule we wrote by hand, but a second test decodes the two recorded request frames and checks that the recording contains Grpc-Status: 3 and that same message, so the rule stays tied to what the gateway actually returned.
What we didn’t count
Plenty of the errors in those recordings aren’t bugs. A separate inventory from the same window has 4,571 NotFound responses. Deleted reports and stale IDs return NotFound legitimately, and we haven’t gone through them one by one, so none of them count here. We also set aside 35 rejections for analyzing an empty snapshot, which is a guard working as designed.
Some groups are still open. Thirty-seven count requests arrived with no scope, and we haven’t traced them to a caller yet. Another 39 rejections from the playground use the same unsupported filter operator as the ?service= links but came through ordinary filter URLs, so they may be a second path to that bug. And local replay makes 234 lookups for a report literally named proxymock, then ignores the answer. We’ve reproduced that one but haven’t filed it.
We also haven’t run the agent on these bugs without the recording, so we can’t tell you how much the traffic helped it compared with reading the code alone.
Try it on your own service
If your frontend talks gRPC-web, or any API that reports errors in the body or a header instead of the status code, you can run the same search this afternoon. Open the repo for that service and install proxymock:
brew install speedscale/tap/proxymockStart the service under proxymock and use the app for a while:
proxymock record -- <your-app-command>Then run the grep above against the new recording under proxymock/. If your API uses a different error convention, proxymock search --in proxymock --query <error text> finds a string anywhere in the recorded headers or bodies.
When you find something worth fixing, hand your coding agent the Speedscale install instructions and ask it to connect proxymock’s MCP server and skills to the client you use. Restart the client afterward. The traffic-to-code skill drives the investigation, and the record-traffic skill helps if the failing call isn’t in a recording yet. Run your team’s sanitization before any headers or payloads go to a model.
Then try this prompt:
Use traffic-to-code to investigate [symptom] in this repository. Search [recording directory] and read the relevant requests and responses before editing. Show which exchange supports your diagnosis. Add regression coverage in the existing test harness, make a scoped fix, and run the check on both the original and changed code. Preserve the captured traffic.
When it comes back, read the exchange it cites yourself. Ask for the test result on the original code too. If the new test passes there, it isn’t testing your bug.
Frequently asked questions
How do I find gRPC-web errors that return HTTP 200?
gRPC-web puts the real result in a Grpc-Status header, so a rejected call still returns HTTP 200 OK. Record the traffic between your frontend and API, then search the recorded responses for a Grpc-Status other than 0. proxymock writes HTTP and gRPC exchanges as Markdown files, so grep works on them.
How can I use recorded traffic to help my coding agent fix a bug?
Connect proxymock to your coding agent and use the traffic-to-code skill with a recording of the failure. The agent searches the captured requests and responses and follows the relevant fields into your code. Read the exchange it cites yourself, then check that the new regression test fails on the original code and passes with the fix.