--- title: "How to Measure TTFT and Inter-Token Latency With Recorded Traffic | Speedscale" description: "Run the mock-lab LLM streaming lab: record streamed chat traffic with proxymock, replay it, and fail the run when TTFT or inter-token latency regresses." canonical: "https://speedscale.com/blog/how-to-measure-ttft-with-recorded-traffic/" --- [Site index](https://speedscale.com/blog/llms.txt) | [Markdown](https://speedscale.com/blog/how-to-measure-ttft-with-recorded-traffic/index.md) | [Canonical page](https://speedscale.com/blog/how-to-measure-ttft-with-recorded-traffic/) ![The Stream tab of a replayed chat completion in Speedscale, with each frame on a time axis and time to first token and the last output marked](https://speedscale.com/_astro/stream-timeline.Q6jn0uLu_Z2dNiUW.webp) # How to Measure TTFT and Inter-Token Latency With Recorded Traffic Nov 12, 2026 [By Matt LeRay](https://speedscale.com/blog/author/matt-leray/) [AI/ML](https://speedscale.com/blog/category/ai/ml/) [Load Testing](https://speedscale.com/blog/category/load-testing/) [API Testing](https://speedscale.com/blog/category/api-testing/) * * * This guide builds a check that fails when time to first token (TTFT) or inter-token latency (ITL) regresses on recorded LLM traffic. It uses the [LLM streaming lab in mock-lab](https://github.com/speedscale/mock-lab/tree/main/labs/llm-streaming): a small model server whose timing you control, a client that sends it streamed chat requests, and `make` targets for every step. You record traffic, replay it against the fast server, slow the server down and watch the same replay fail. Then you apply the same commands to your own model server. The metric definitions follow [NVIDIA AIPerf](https://github.com/ai-dynamo/aiperf/blob/main/docs/metrics-reference.md). For why measuring them on recorded traffic matters, see [Measure LLM Latency on the Prompts Your Users Actually Send](https://speedscale.com/blog/measure-llm-latency-on-real-user-prompts/). ### Follow the guide with a coding agent The expandable **Agent prompt** blocks are an alternative to typing the commands. Copy each one into the same Codex, Claude or other coding-agent conversation, in order. The agent needs a terminal with Go and `proxymock` installed. Run either the commands or the prompt for a step, not both. ## Prerequisites Install Go 1.23 or newer, `curl`, `nc`, `jq` and [`proxymock`](https://docs.speedscale.com/proxymock/getting-started/quickstart/) 2.5.1370 or newer. Clone the lab: ``` git clone https://github.com/speedscale/mock-lab.git cd mock-lab/labs/llm-streaming ``` The model server stands in for an inference server such as vLLM or NVIDIA NIM. It streams a canned answer one token at a time, with the timing set by `FIRST_TOKEN_MS` (default 300), `TOKEN_GAP_MS` (default 20), `TOKENS` (default 20) and `TOKENS_PER_FRAME` (default 1). It speaks all four formats Speedscale measures: OpenAI-compatible chat completions, Anthropic Messages, Ollama’s native chat and Amazon Bedrock `ConverseStream`. The numbers below come from running the lab on a MacBook with proxymock 2.5.1376. Your values will differ by a millisecond or two. ## 1\. Start the model server ``` make server ``` Leave it running on port 8000 and open a second terminal in the same directory. ## 2\. Record streamed traffic ``` make record ``` The target starts `proxymock record --app-port 8000`, sends eight streamed chat completions through proxymock’s inbound port 4143 and stops the recorder. The prompts mix one-line questions with pasted tickets and error messages, behind the same system prompt, and each request sets `stream_options.include_usage`. Each line the client prints is one request: ``` openai 200 first byte 2.0 ms done 703.5 ms 4559 bytes "Where is order 48213?" ``` The first byte arrives at once because the server’s first frame only names the role. The first token arrives about 300 ms later, and that is the moment TTFT measures. A load tool that times the first byte would report this server as instant. `proxymock record` stores the arrival time of every frame of an HTTP/1.1 or HTTP/2 streamed response. Each recorded request in `proxymock/recording/localhost/` has an `httpStream` block listing them. In this recording the first frame with output arrived at 301.1 ms, and each one after it about 21 ms later. That recorded timing is what a replay is compared against. In Kubernetes, the Speedscale sidecar records the same frame timing. Traffic captured by the eBPF-based nettap capture, imported from a HAR or other file, or generated from a spec has frames but not their timing, so its replays show replayed values only. Agent prompt: run the lab and record ``` Clone https://github.com/speedscale/mock-lab and work in labs/llm-streaming. Read its README. Confirm Go, jq and proxymock 2.5.1370 or newer are installed. Start `make server` in a background terminal and check http://localhost:8000/healthz. Run `make record` and report the number of recorded requests and, from one recorded file's httpStream block, the offset of the first frame with output and the gap between the frames after it. ``` ## 3\. Replay and gate on the LLM metrics ``` make replay ``` The target empties `proxymock/results/replay`, so each replay is scored on its own pairs, and runs this command: ``` proxymock replay --in proxymock/recording --test-against http://localhost:8000 \ --out proxymock/results/replay \ --fail-if 'llm.ttft-p95 > 800' \ --fail-if 'llm.itl-p95 > 40' ``` Against the fast server both gates pass and `proxymock` exits 0: ``` 2 EVALS PASSED ✔ passed eval "llm.ttft-p95 > 800.00" - observed llm.ttft-p95 was 301.67 ✔ passed eval "llm.itl-p95 > 40.00" - observed llm.itl-p95 was 21.06 ``` Every duration is in milliseconds from the moment the request was sent: * **TTFT** runs until the first frame that carries output: text, reasoning or tool-call arguments. A role-only frame, a keepalive or a usage report is not output. * **Request latency** runs until the last frame that carries output. Usage reports and end markers that follow it do not count. * **ITL** is (request latency - TTFT) / (output tokens - 1). A stream with one token has none. * **Tokens per second per user** is 1 / ITL. **Output tokens per second** divides all output tokens by the run’s LLM window. These are the names `--fail-if` accepts, with the goal name for a test config: | `--fail-if` name | Goal metric | Unit | | --- | --- | --- | | `llm.ttft-p50`, `llm.ttft-p95`, `llm.ttft-max` | `llmTtftP50`, `llmTtftP95`, `llmTtftMax` | ms | | `llm.e2e-p50`, `llm.e2e-p95`, `llm.e2e-max` | `llmE2eP50`, `llmE2eP95`, `llmE2eMax` | ms, request latency | | `llm.itl-p50`, `llm.itl-p95`, `llm.itl-max` | `llmItlP50`, `llmItlP95`, `llmItlMax` | ms | | `llm.user-tokens-per-sec-p50`, `llm.user-tokens-per-sec-p95` | `llmUserTokensPerSecP50`, `llmUserTokensPerSecP95` | tokens per second | | `llm.output-tokens-per-sec` | `llmOutputTokensPerSec` | tokens per second | | `llm.output-tokens` | `llmOutputTokenCount` | tokens | | `llm.requests` | `llmRequestCount` | streams | | `llm.incomplete` | `llmIncompleteCount` | streams | | `llm.token-count-approx-pct` | `llmTokenCountApproxPct` | percent | A replay that sent no LLM stream reports none of these, and a percentile with no samples is left out rather than reported as zero. ## 4\. Catch a regression Stop the server with Ctrl-C and start a slower one, as a heavier model, a smaller GPU or a gateway change might produce: ``` make slow-server ``` It waits 900 ms for the first token and 50 ms between tokens. Replay the same recording from the second terminal: ``` make replay ``` Both gates fail. `proxymock` exits 1, so `make` stops with an error: ``` 2 EVALS FAILED ✘ failed eval "llm.ttft-p95 > 800.00" - observed llm.ttft-p95 was 901.59 ✘ failed eval "llm.itl-p95 > 40.00" - observed llm.itl-p95 was 51.08 ``` When a gate fails, the output lists every LLM metric with the recorded value beside the replayed set: ``` llm.ttft-p95: 901.59 llm.itl-p95: 51.08 llmTtftP95Recorded: 301.29 llmItlP95Recorded: 21.07 ``` The same run prints `Replay verdict: ... (pass)`. The slow server returned the same status, headers and body shape, so a response comparison finds nothing wrong. The latency table shows each request took 1.87 s instead of 0.7 s, but only TTFT shows that users waited three times as long for the first word. Agent prompt: catch the regression ``` In mock-lab/labs/llm-streaming, run `make replay` against the running `make server` and report both eval lines and the exit code. Then stop the server, start `make slow-server`, run `make replay` again and report the eval lines, the exit code, the replay verdict line and the recorded values (llmTtftP95Recorded, llmItlP95Recorded) printed beside the replayed ones. Explain why the verdict passed while the gates failed. ``` ## 5\. Put the gates in a test config Thresholds that live in a test config apply to every replay that uses it, and cloud test configs accept the same goal names. The lab includes `proxymock/testconfigs/llm-gates.json`, made with `proxymock test-config new llm-gates --from regression` and these rules added to its `rules` array: ``` { "metricName": "llmTtftP95", "type": "TOO_HIGH", "value": 800, "action": "ALERT" }, { "metricName": "llmItlP95", "type": "TOO_HIGH", "value": 40, "action": "ALERT" }, { "metricName": "llmIncompleteCount", "type": "TOO_HIGH", "value": 0, "action": "ALERT" } ``` Use `TOO_HIGH` for durations and counts and `TOO_LOW` for rates such as `llmUserTokensPerSecP50`. LLM goals apply to the whole run, so they take no endpoint location. The `regression` config asserts the status code, the `Content-Type` header and the response schema, not the generated text, so it works for nondeterministic model output. With the slow server still running: ``` make replay-goals ``` ``` TEST CONFIG GOALS (llm-gates) FAIL llmTtftP95 <= 800 - observed 901.66 (same measure as --fail-if llm.ttft-p95) FAIL llmItlP95 <= 40 - observed 51.08 (same measure as --fail-if llm.itl-p95) PASS llmIncompleteCount <= 0 - observed 0 (same measure as --fail-if llm.incomplete) ``` The response assertions passed for all eight streamed requests. Against `make server` every goal passes and `proxymock` exits 0. In the dashboard’s replay launcher the same goals appear with labels such as **LLM TTFT p95**. ## 6\. Get exact token counts ITL and the token rates divide by the output token count. Speedscale takes it from the stream’s usage report: `usage.completion_tokens` for OpenAI-compatible servers, `usage.output_tokens` for Anthropic, `eval_count` for Ollama and `outputTokens` or `outputTokenCount` for Bedrock. [OpenAI’s API specification](https://github.com/openai/openai-openapi) sends usage in a stream only when the request sets `stream_options.include_usage`, as an extra chunk before `data: [DONE]`. Without a usage report, each output frame counts as one token. The lab shows what that does to a server that batches tokens. Restart the server with four tokens per frame and record once without usage and once with it: ``` make server TOKENS_PER_FRAME=4 make record RECORDING_DIR=proxymock/recorded-no-usage USAGE=false make record RECORDING_DIR=proxymock/recorded-usage ``` A passing replay prints only the metrics it gates on. `make llm-metrics` replays a recording and prints every LLM metric of the run: ``` make llm-metrics RECORDING_DIR=proxymock/recorded-no-usage make llm-metrics RECORDING_DIR=proxymock/recorded-usage ``` The two runs gave these results: | Recording | Output tokens | ITL p50 | Approximate share | | --- | --- | --- | --- | | Without usage | 40 | 21.06 ms | 100% | | With usage | 160 | 4.43 ms | 0% | The server is the same in both runs. Without usage it looks almost five times slower per token, because 20 tokens arrive in five frames. A replay sends the recorded request unchanged, so the usage setting has to be in your clients’ requests when you record. Gate on `--fail-if 'llm.token-count-approx-pct > 0'` when token rates matter. ## 7\. Read the LLM tab and the Stream tab A replay run in Speedscale cloud produces a report with an **LLM** tab. It shows TTFT, request latency, ITL and tokens per second per user at p50, p95 and max, with the number of streams, output tokens and the share of approximate token counts. When the recording had frame timing, each value has the recorded value under it, and a change of 10% or more in the wrong direction is highlighted. ![The LLM tab in demo mode with fixture data: TTFT about 20% slower than the recording, and one stream cut off before its end marker](https://speedscale.com/_astro/llm-report-tab.ik-VHFqQ_2tMumF.webp) Open any streamed response in the traffic viewer or a report and select the **Stream** tab to see its frames on a time axis from the moment the request was sent. For a replayed LLM stream it also marks TTFT and the last output, and shows the format, the output token count and the request’s ITL. Those values are stored as tags on the replayed request, such as `llmTtftMs` and `llmItlMs`. ![The Stream tab of a replayed chat completion: 23 frames, TTFT at 301.5 ms, the last output at 700.5 ms and an ITL of 21 ms](https://speedscale.com/_astro/stream-timeline.Q6jn0uLu_Z2dNiUW.webp) Both screenshots use dashboard demo data shaped like the lab’s server: a 300 ms first token and 20 tokens about 21 ms apart. ## 8\. Apply it to your own model server Check that your server’s stream is measurable before you record it: ``` curl -sN -D /dev/stderr http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"my-model","stream":true, "stream_options":{"include_usage":true}, "messages":[{"role":"user","content":"Say hello"}]}' | tail -n 6 ``` * **Content type** must be `text/event-stream` (OpenAI-compatible and Anthropic), `application/x-ndjson` (Ollama) or `application/vnd.amazon.eventstream` (Bedrock). Speedscale recognizes the format from the stream itself, whatever the path. * **Content-Encoding** must be absent or `identity`. A compressed stream is not split into frames. * **The end marker** must arrive: `[DONE]`, `message_stop`, `done: true`, or `messageStop` and the final chunk for Bedrock. Streams that end without it count toward TTFT and the incomplete count, and are left out of latency and token rates. Then run the lab’s steps against your server: `proxymock record --app-port ` with your clients pointed at port 4143, and `proxymock replay --test-against ` with your thresholds. If the service you test is a gateway in front of the model, record and replay the gateway. For load, `make load` shows the form: `--load-test --vus 4 --for 20s` reports the same LLM metrics. Agent prompt: apply it to my model server ``` Ask me for the URL of my model server or LLM gateway and how its clients reach it. Send one streamed request with curl and report the Content-Type, any Content-Encoding, the end marker and whether a usage report arrives. Then propose the proxymock record and replay commands, following mock-lab/labs/llm-streaming, with --fail-if thresholds based on my recorded values. Do not change client configuration or start a recording until I approve the plan. Report the commands actually run and their results. ``` ## Limits * **Mocked LLMs answer at once.** When the service under test calls an LLM and the replay mocks that call, the mock sends the recorded stream in one write. Measure a model server by replaying traffic against it. To stop paying for model calls in other tests, see [LLM simulation](https://docs.speedscale.com/guides/mocking/llm-simulation/). * **Run-wide only.** Metrics cover the whole replay, with no per-stage values and no concurrency sweep table like AIPerf’s. To compare concurrency levels, run one replay per `--vus` value. * **Calibrated definitions, not calibrated hardware.** The formulas match AIPerf, but results depend on where the replay runs and how far it is from the model server. Compare runs made from the same place. To get AIPerf’s own numbers on the same prompts, `proxymock export aiperf --in proxymock/recording --out ./aiperf-bundle` writes a bundle whose `trace.jsonl` AIPerf loads with `--custom-dataset-type mooncake_trace`. It removes credential headers; mask sensitive request bodies with your DLP rules first. The [LLM streaming metrics guide](https://docs.speedscale.com/guides/replay/llm-streaming-metrics/) is the reference for every name above. ## Frequently asked questions ### Why does my report say token counts are approximate? The stream had no usage report, so each output frame was counted as one token. OpenAI-compatible servers send usage in a stream only when the request sets stream\_options.include\_usage. In the lab, a server sending four tokens per frame measured 40 tokens instead of 160 and an inter-token latency of about 21 ms instead of 4.4 ms without it. ### Why did the replay verdict pass when the LLM gates failed? The verdict compares responses: status, headers and body shape. A slower model server returns the same responses, so the verdict passes. The --fail-if conditions and test-config goals check the streaming metrics, and either one failing makes proxymock exit 1. ### How do I compare concurrency levels? Run one replay per level with a different --vus value and compare the results. LLM metrics cover the whole run, with no per-stage values or sweep table. ## Table of Contents ## Stop writing API mocks by hand proxymock records real traffic from your running app and replays it as mocks for HTTP, gRPC, Postgres, Kafka, and more. Install locally, activate once, and start recording. [Install proxymock](https://docs.speedscale.com/proxymock/getting-started/installation/) [View the docs](https://docs.speedscale.com/proxymock/)