Reliability Engineering in the AI Era
See why reliability engineering must span code, testing, telemetry, and incidents as AI agents erase the boundary between pre-production and production.
Posts about observability
Browse 30 posts in this category
Synthetic Monitoring Is Broken. Your Production Traffic Can Fix It.
Synthetic monitoring depends on scripts engineers maintain by hand. Capturing and replaying real production transactions turns customer behavior into tests.
The Pod Was Cheaper. The Service Wasn’t.
Use OpenCost and proxymock to prove Kubernetes rightsizing lowers cost per successful request without hiding behavior or throughput regressions in testing.
Flamegraphs Find It. Replay Proves It.
Use Grafana Pyroscope and proxymock with an AI coding agent to find a Go CPU hotspot, preserve API behavior, and verify the performance fix.
eBPF: Preventing Garbage HTTP Payloads When Reading Kernel Scatter-Gather Buffers
How our nettap eBPF agent produced garbage HTTP bodies from kernel iov_iter scatter-gather buffers, and the CO-RE plus task-local storage fix that made it correct.
AI writes code in seconds, but delivery still takes days
AI pushed throughput up 59%, yet median delivery got worse. The bottleneck moved to validation. How replaying production traffic in CI closes the gap.
Trace without traces
The trace was sampled out. I found the bug anyway — by filtering recorded traffic on the customer's email instead of a trace ID. Here's how to follow one request across four services with no trace IDs and no OpenTelemetry.
Logs told me something broke. Traffic showed me what.
Capture production traffic and store it in your own Elasticsearch with Speedscale BYOC. Pull it locally with es-gather.py and reproduce bugs with proxymock.
When Your Observability Literally Stops Traffic
Observability tells you what failed—but not how to recreate it. Why reproducibility is the missing fourth pillar, and what that means for incident response.
Try Speedscale free
Choose the desktop proxymock or the hosted cloud trial to get started.