AWS re:Invent happy hour: Dec 1, 6 PM PST at MGM Grand RSVP

Capture Encrypted Microservice Traffic with Speedscale eBPF

Capture Encrypted Microservice Traffic with Speedscale eBPF

(Updated: )

Production bugs that only reproduce in actual traffic can be some of the most frustrating bugs in software development. You can stare at your logs, add traces to your code, add instrumentation – and still not be able to see the actual requests that went over the wire. And that gets even harder when the requests are encrypted and the system is a black box. You can use tools like Wireshark or Kubeshark to capture the requests. But they require storage for the dump of the packets or may not be enabled in the first place. You need to be able to see the actual requests and responses without modifying code, without dealing with certificates, with measured resource impact. In this post, I will show you how to set up Speedscale’s eBPF collector and use it to capture actual requests and responses from a running Java application in a few minutes with a fresh install of Helm.

What the eBPF Collector Does Differently

Observability solutions tend to follow a progression. You start with telemetry agents that can provide you with metrics, logs, and traces. That lets you know when something goes wrong and where you’re spending your time. But once you start to actually debug your system, rather than simply alerting people to the problem, you need full fidelity. You need to see the actual requests and responses that went over the wire. Full fidelity used to require a sidecar in the path of every request. Sidecars add latency to your system. They add complexity to your system. And they add another point of failure to your system. eBPF bypasses that problem.

The collector observes plaintext TCP at kernel probes. Supported TLS capture uses library-specific probes, or a Java instrumentation agent for JVM traffic. The Java demo in this walkthrough uses that agent, not OpenSSL hooks. Loading it requires new pods and a JVM restart. Check the capture matrix and Java compatibility guide before installation.

Benefits

  • No Cert Management. You don’t have to worry about installing a CA or rotating certs.
  • No Application Source Changes. JVM capture still adds an agent through startup configuration.
  • No Capture Sidecar Per Pod. The collector runs as a DaemonSet with separate capture and ingest containers.
  • Measurable Overhead. Monitor application CPU and latency, collector capacity, and dropped observation events.

eBPF capture point between application and OpenSSL before encryption

Prerequisites

You need a Kubernetes cluster whose nodes meet the kernel, BTF, host-access, and capability requirements. You can install minikube if you don’t have one. You don’t have to install an application, as we include one as part of the operator install.

Four-step workflow: Install, Activate, Inspect, Archive

Step 1: Install the Speedscale Operator with the Demo App

First, get your API key from your Speedscale profile page. Haven’t signed up? Start a free trial. For local capture and replay, install proxymock and follow its current activation and license instructions.

Add the Helm repo and install the operator. The deployDemo: "java" flag, which is enabled by default, means the operator will automatically install a Java Spring Boot demo application alongside itself. This means you can immediately observe the demo application after installation.

# Set SPEEDSCALE_API_KEY and CAPTURE_CLUSTER in this shell first.
helm repo add speedscale https://speedscale.github.io/operator-helm/
helm repo update

helm install speedscale-operator speedscale/speedscale-operator \
  -n speedscale \
  --create-namespace \
  --set ebpf.enabled=true \
  --set-string apiKey="$SPEEDSCALE_API_KEY" \
  --set-string clusterName="$CAPTURE_CLUSTER"

deployDemo is enabled by default, so the demo application is automatically deployed. You can override this with --set deployDemo="java" or --set deployDemo="" if you want. ebpf.enabled must be enabled to record eBPF data. It is required unless you want to disable eBPF and use sidecars instead.

Check the operator and forwarder pods are running:

kubectl -n speedscale get pods

Expected output:

NAME                                    READY   STATUS    RESTARTS   AGE
speedscale-forwarder-xxxxxxxxxx-xxxxx   1/1     Running   0          5s
speedscale-operator-xxxxxxxxxx-xxxxx    1/1     Running   0          15s

The java-server demo pod should appear in the same namespace. It’s a simple Spring Boot web server with authenticated endpoints, making HTTP requests.

Step 2: Enable eBPF Monitoring via the Infrastructure UI

With the operator running, go to the Infrastructure section of the Speedscale web UI at app.speedscale.com/infrastructure.

You should see your cluster listed under the name you chose during installation. Clicking on it opens the Cluster Inspector, a tabbed view of all Speedscale monitored workloads for this cluster.

Activate eBPF on the java-server

  1. Go to the Workloads tab. You should see all deployments in the namespace, including java-server.
  2. For eBPF, you need to flip the Injected switch. This is where you register workloads with the eBPF collector. No sidecar is injected into the pod itself.
  3. Toggle Injected to true for java-server.

For the Java demo, enable the Java agent as well. Replace the deployment name if yours differs:

kubectl annotate deployment java-server -n speedscale \
  capture.speedscale.com/enabled=true \
  capture.speedscale.com/java-agent=true --overwrite
kubectl rollout status deployment/java-server -n speedscale

This rolls new pods and restarts the JVM. For supported native probe paths, selecting capture targets does not itself require that Java-agent rollout. Verify decoded requests and responses after enabling capture.

The Workloads tab offers additional information and actions for each workload:

  • TLS Out: Whether unwrapping of outgoing TLS is enabled (useful for services calling other APIs over HTTPS)
  • Container Logs and Sidecar Logs: Clicking these buttons allows you to view logs directly in the UI without needing kubectl access
  • Sidecar Health: An alert icon is displayed for any issues related to capture health and configuration
  • Request CPU/Memory: Current resource utilization for the workload

Step 3: Inspect Traffic in the Traffic Viewer

When you’ve done this, you should see live traffic from java-server appear in the Traffic Viewer.

What you can see

The Traffic Viewer offers a detailed log of all captured transactions. Each request/response pair offers:

  • Info: Response codes, duration, URL, timestamp
  • Request: Headers and body sent
  • Response: Headers and body received

Supported outgoing calls from java-server can include TLS plaintext captured inside the JVM by the Java agent. Transport and provider coverage varies; inspect a known exchange rather than assuming every connection was decoded.

Filtering down to what matters

The filter features allow you to filter your view of the traffic:

  • Time Range: Zoom in on a specific time window, which is useful if you are correlating a bug report with a specific timestamp.
  • Inbound/Outbound: Switch between incoming traffic to java-server and outgoing traffic from it to other services.
  • Status Codes, URLs, and Headers: Filter on error responses, specific URLs, and specific header values.
  • Full Text: This is probably one of the most important features and is a filter on text anywhere in the traffic set. This will even find text in the payload of binary protocols such as gRPC, databases, and other sources.

Click on a row to expand it into the Request/Response Pair view. This is where you will find the complete picture of what is going on. The exact bytes sent and received will be shown in a human-readable form. For JSON payloads, you will even be able to toggle between raw and pretty views. Schema mismatches, missing fields, and unexpected values will all be available without having to instrument anything on your end.

The Traffic Viewer will also show a Service Map, which will display the dependency graph constructed from the actual traffic. The inbound callers to java-server and its outbound dependencies will be represented as nodes in the graph, with request volumes and error rates on the edges.


Step 4: Archive Traffic as a Snapshot

After you have identified a time window or requests that you want to save as a snapshot, you can do so. A snapshot will save the selected traffic as a self-contained artifact that includes:

  • Raw Traffic - the actual requests and responses
  • Transform Rules - the rules that define how to transform the requests during the replay process
  • Dynamic Values - review and configure correlation for tokens and generated identifiers; saving a snapshot does not guarantee that every value is refreshed. For more on how to keep your sensitive data safe, please see DLP, Traffic Replay, and the Missing Link to Software Quality.

How to create a snapshot

  1. In the Traffic Viewer, select the traffic that you want to save as a snapshot using the time picker and filters.
  2. Click Save to Tests/Mocks.

Snapshots are reusable between replay runs. Each run generates its own report with its own latency graphs, assertion results, and captured traffic, but all runs share the same source snapshot. This means that the snapshot is a stable test fixture, with the traffic you saw in production today becoming the load you can run against the code deployed tomorrow.


Resource Considerations

Resource defaults vary by chart version. Inspect your pinned chart’s values and use the resource utilization guide for sizing. For nodes that need to handle high volumes of traffic, you can adjust these values in your values file:

ebpf:
  enabled: true
  resources:
    requests:
      cpu: "200m"
      memory: "512M"
    limits:
      cpu: "1000m"
      memory: "2G"

We strongly advise against running the collector with the debug log level under production load, however, as this will greatly increase CPU utilization in the ingest container and may lead to dropped traffic under moderate request rates.


What’s Next

The workflow is install, activate, inspect, and archive. Setup time depends on node compatibility, permissions, and the workload’s capture path. The demo app provides you with some real traffic to play with right away, but the same workflow applies to your own applications.

Now that you have your archive, the obvious next steps are:

  • Replay it with your new version of the service to catch regressions before they reach production
  • Add assertions to verify response codes, response payloads, or latency budgets
  • Use recorded outbound responses to mock the dependencies java-server calls during an isolated test

This is the full-fidelity production context that closes the Observability Gap. Sampled traces tell you something went wrong. Full traffic archives let you test code changes against exactly what production experienced , including supported encrypted payloads that ordinary traces may omit.

For more information on any of the above, see the Speedscale documentation or drop us a line at support@speedscale.com.

Frequently asked questions

Which runtimes and TLS libraries are supported?

Plaintext TCP capture and TLS plaintext visibility have different requirements. The documented TLS paths cover OpenSSL 3.x, supported Go and rustls builds, and supported JVM transports through a Java agent. Older OpenSSL, BoringSSL, and LibreSSL are not covered by that OpenSSL path. Verify the current runtime matrix and a decoded exchange from your own workload.

What node access and capabilities are required?

Use supported Linux nodes with BTF enabled and the host paths and capabilities listed in the security reference. The documented kernel baseline is 5.15 for x86_64 and 6.0 for arm64, with RHEL 9.2 or newer. The non-root capture container still needs elevated capabilities; review that contract with the cluster owner.

When does collection require an agent or restart?

The JVM path loads a Java instrumentation agent through the startup configuration, so it requires new pods and a JVM restart. Supported native-library probes can attach without that Java rollout. Existing long-lived connections may still be incomplete, especially multiplexed HTTP/2 or gRPC connections. Confirm fresh connections and decoded payloads after enabling capture.

Sources: eBPF capture requirements and limits, Java agent setup and compatibility, Kubernetes security requirements.

Capture and replay your production traffic

Speedscale records live traffic from your Kubernetes services and turns it into tests, mocks, and load tests automatically. 30-day free trial, full feature set.