eBPF: Correlating rustls Plaintext to TCP Connections Without a File Descriptor

In Under the Hood with Go TLS and eBPF, I left socket tracking as an exercise for later. The example used bpf_get_current_pid_tgid() and explicitly excluded concurrent TLS operations. Capturing plaintext was enough for that post. With rustls, later arrived: I could read the HTTP payload perfectly and still attach it to the wrong TCP connection.

That’s a frustratingly convincing failure. The request looks right. The response looks right. The application works. Then you follow the recorded connection and discover that the bytes supposedly came from a different service.

A single pooled keep-alive connection scored 80% in an early test. Adding a second subgraph dropped that to 20%, without changing the capture code. A subgraph is one of the backend GraphQL services that Apollo Router calls to assemble a response. We’d added another destination, and suddenly the correlation looked like it had fallen apart. Really, we’d finally given the test enough connections to expose what it had been doing all along.

I built and threw away three designs during this investigation. One detour involved a Tokio function that existed in the symbol table, looked exactly right in disassembly, had five real call sites, and fired our probe zero times during the workload. I spent far too long treating those facts as incompatible.

No socket to follow

With a conventional socket-backed OpenSSL connection, there’s a straightforward chain from the SSL * to its BIO, OpenSSL’s I/O abstraction, and then to the file descriptor. OpenSSL even exposes SSL_get_fd for that association. This depends on the BIO actually being descriptor-backed; a memory BIO doesn’t magically acquire a socket. But for the socket-backed case, the relationship we’re trying to discover is already represented in the library.

Go’s TLS example didn’t give our probe that relationship directly. A goroutine, Go’s lightweight unit of concurrent execution, offers another possible identity to track. But that isn’t what the earlier post implemented. Its thread-keyed example deliberately stopped before handling concurrent TLS operations. Carrying enough context to identify a stream remained unfinished work.

rustls made the missing relationship harder to ignore. It’s designed to keep TLS processing separate from transport I/O, often called sans-io. The caller owns the transport, feeds encrypted bytes into the TLS library, and takes encrypted bytes back out to send. The TLS state doesn’t have to know whether those bytes came from a socket, a pipe, or an adapter around something else.

You can see the boundary in rustls 0.23.35’s rustls/src/conn.rs:739-791: read_tls accepts a reader, and write_tls accepts a writer. Those interfaces provide byte I/O without providing a TCP identity. It’s a good separation of responsibilities. For someone standing outside the process trying to reconstruct that identity, it’s inconvenient.

There are, of course, connection objects in rustls. ConnectionCore holds TLS state, as rustls/src/conn.rs:858-869 shows. What it doesn’t hold is the socket relationship we need. Finding a TLS object and finding the TCP connection behind it are separate jobs.

At the plaintext capture point, I had bytes and a thread identity, but no established socket binding. At the kernel TCP probe, I had a socket and encrypted bytes. Correlation had to connect those observations so the plaintext landed on the correct 5-tuple: source address, source port, destination address, destination port, and protocol.

Remembering the most recent socket on a thread sounded reasonable until multiple connections shared that thread. Tokio can run many tasks on its worker threads, and tasks can move between workers, as its task documentation explains. A thread ID describes where work is executing. It doesn’t tell you which connection owns the next plaintext buffer.

Suppose one thread reads from connection A, then reads from connection B, then handles plaintext buffered for A. A map containing only that thread’s most recent socket says B. The lookup succeeds. The socket exists. The association is wrong, and there’s nothing about readable HTTP that will reveal the mistake.

I wanted something that followed the logical work instead.

Five calls, zero hits

Tokio’s task identity looked like the obvious answer. If Go instrumentation could follow a goroutine, perhaps we could follow an async task through the rustls operation. That still required proving the task-to-connection relationship, but first I needed to observe the task reliably.

The source was encouraging. Tokio installs a task ID around execution of the future, the object representing an async computation. For example, Tokio 1.48.0’s tokio/src/runtime/task/core.rs:318-373 passes the ID into TaskIdGuard::enter before polling the future. There’s a plain function argument at the point where the runtime knows which task it’s running. That’s an appealing place to put a probe.

I found the candidate symbol with nm. I disassembled its address. The instructions did what I expected, and the field offset agreed with DWARF, the binary’s debugging information describing things like type layouts. This wasn’t a case of matching a vaguely similar name and hoping the arguments lined up.

There were also five real call sites. That seemed to settle the question of whether the compiler had left behind something unused. The function existed, the code matched the source, and other code called it. I attached the probe and ran the workload.

Zero hits.

Meanwhile, the plaintext probes kept producing data. The application was executing async work. The traffic I wanted to attribute was passing through the process. I had a very convincing static explanation of where the task ID should be and no runtime evidence that execution ever passed through my probe.

Here’s the inspection I’d start with on an unstripped Linux Tokio binary. Use GNU binutils with Rust demangling support and replace ./router with your executable:

nm -an --demangle=rust ./router | rg 'TaskIdGuard|set_current_task_id|Core.*poll'
objdump -d --demangle=rust ./router | rg -n -B 12 -A 20 'TaskIdGuard|set_current_task_id|Core.*poll'

The first command finds named candidates. The second gives you the function bodies and surrounding call instructions to investigate. Follow the actual target addresses, then inspect the enclosing callers. Don’t stop at counting references to the name. For a larger function, expand the disassembly beyond the displayed context.

Those commands reproduce the inspection, not a promise that your binary will have five callers or the same optimization choices. A stripped binary may expose none of these names. And neither command tells you whether a probe fires; that requires attaching it and exercising the path you care about. The distinction was precisely what I’d overlooked.

Rust compiles generic code through monomorphization: it generates implementations for the concrete types a program uses. The Rust book’s explanation uses simple generic functions, but the same principle applies to Tokio’s generic task machinery. One source definition can lead to several pieces of machine code.

Then inlining makes the correspondence less convenient. The compiler can put a callee’s instructions directly into a caller, so execution doesn’t visit a separately named function entry. Different concrete versions of the generic caller can end up with different decisions about which calls remain out of line.

That’s what happened in the Router binary I was inspecting. The async Core::poll paths had inlined the call I wanted to observe. The out-of-line copies that retained an attachable symbol were on the blocking thread pool paths doing DNS work. Their callers were real. They just weren’t the async paths carrying the TLS work in this test.

nm had correctly told me that the symbol existed. objdump had correctly shown the instructions and call sites. I’d supplied the unsupported conclusion that the workload must execute through that address.

The check I’d add now is small: put a hit counter on the candidate before building anything that depends on its arguments. Exercise connection setup and steady traffic separately, then compare the hits with the known TLS activity. A probe that observes startup work can still miss every request afterward. Disassembly helps explain a hit count; it can’t substitute for one.

The task ID wasn’t fictional, and the source wasn’t wrong. But the convenient attachment point didn’t cover the work I needed. Supporting that approach meant accounting for the concrete compiled paths, not just locating one attractive symbol. Before expanding the design again, I went back to the socket information we already collected.

An empty map

I’d also tried two opposite cache policies. One kept a connection’s existing binding while the associated flow was still alive. The other refreshed the binding using the thread’s recent socket activity. Both produced bad attribution, which seemed like evidence that this whole source of correlation information was unreliable.

It wasn’t evidence of that. The map supplying the information wasn’t being populated for the TLS flows.

Our kernel-side probe skipped payload capture once a flow was marked TLS. That made sense for payload handling: we’d get the plaintext from the TLS probes. But the early return also skipped the line recording which connection the thread had just read from.

The bug reduced to this ordering. These are abbreviated C sketches of the control flow, not standalone probe programs:

/* Broken: TLS flows never reach the correlation update. */
if (flow->is_tls)
    return 0;

bpf_map_update_elem(&last_read, &pid_tgid, &flow_id, BPF_ANY);

The map that was supposed to connect the kernel observation to the plaintext observation had no entry for exactly the traffic under investigation. I’d been changing the reader’s policy while the writer never ran.

Moving that update ahead of the TLS check let the bookkeeping happen even when we didn’t capture the kernel payload:

/* After a successful TCP read, record its flow before skipping payloads. */
bpf_map_update_elem(&last_read, &pid_tgid, &flow_id, BPF_ANY);

if (flow->is_tls)
    return 0;

This reminded me of the scatter-gather buffer investigation. There, the captured data looked wrong because I’d misunderstood what the kernel probe could read at a particular point. Here, the plaintext was fine. The failure was in bookkeeping that I’d assumed had already happened.

Four counters, added late, answered in one read what a week of dumping captured chunks hadn’t. The debugging question should have been whether the correlation mechanism was running and receiving data. I’d been asking whether its final output looked better after another change.

If you’re investigating something similar, count the producer updates, consumer entries, successful lookups, and installed bindings. Those are suggested observation points, not an invitation to build another telemetry system. You need enough evidence to distinguish a missing producer from a bad association. A hundred more payload dumps won’t make an empty map less empty.

Fixing the early return gave us data. It didn’t make the thread’s most recent socket correct at every point in the application. I still needed a place where that temporary observation meant something definite.

Bind at packet processing

That place was ConnectionCore::process_new_packets. rustls’s public wrapper delegates to the core method in rustls/src/conn.rs:436-440. The useful property came from how tokio-rustls called it after receiving TLS bytes.

Look at tokio-rustls 0.26.5’s src/common/mod.rs:99-118. Inside read_io, it calls read_tls and then process_new_packets. If the read returns WouldBlock, it returns Pending before packet processing. There’s no async suspension between the successful read and the processing call in that function.

For the direct TCP read path we were observing, this gave us a useful window: the kernel probe had just recorded the socket, and the processing probe now had the rustls connection object consuming those bytes. That was where to establish the binding.

The source proves the call ordering. Applying it to socket attribution also requires that the read actually reached the observed TCP transport. An arbitrary buffered reader can return bytes without a new socket read, and rustls can return zero after a close notification. The order alone doesn’t turn stale thread state into fresh evidence. That’s why this is a description of the observed tokio-rustls TCP path, not a guarantee for every caller of a sans-io library.

At that boundary, the cache operation was simple:

/* Run only at the verified boundary after a fresh, successful TCP read.
 * conn_key identifies this process and ConnectionCore object. */
struct flow_id *flow = bpf_map_lookup_elem(&last_read, &pid_tgid);
if (flow)
    bpf_map_update_elem(&conn_to_flow, &conn_key, flow, BPF_ANY);

The important change was when that lookup happened. Binding at a later plaintext read could accidentally use another connection’s recent socket activity. Binding while processing freshly received TLS data captured the association before that thread-local clue lost its meaning.

The abbreviated snippet assumes that receive-boundary evidence has already been established. If you adapt it, keep that condition explicit instead of interpreting every successful map lookup as permission to bind. Likewise, scope an object address to its process and account for object lifetime in the cache. An address alone isn’t a permanent identity. Those requirements don’t disappear just because the binding operation fits on three lines.

Subsequent plaintext reads and writes could look up the connection object’s binding. They didn’t need the thread to retain the same most recent socket, or even to be the thread that established the binding. The durable association belonged to the TLS connection. The thread map supplied evidence at the receive boundary.

That distinction also explained why the keep-the-first-binding policy was dangerous. Checking that a socket remained open could validate the lifetime of a cache entry without validating its origin. If A had been incorrectly bound to B and B stayed alive in a connection pool, the liveness check would keep approving the mistake.

Writes didn’t need a separate binding mechanism in our normal client handshake path. Before ordinary application writes, the client received the server’s handshake messages, including ServerHello, through the same read-and-process sequence. That seeded the cache. The handshake loop uses read_io in src/common/mod.rs:132-183.

That statement has a boundary too: TLS early data can precede those handshake reads, as the separate early-data write path in src/client.rs:525-566 illustrates. Attaching after a connection’s handshake also means we didn’t observe those reads. This investigation didn’t establish coverage for early data. Late attachment did show up in the measurements.

Against a real Apollo Router on GKE, Google’s managed Kubernetes service, the test used a federated query that fanned out to two TLS subgraphs. Before the fix, 10 of 36 plaintext chunks landed on the correct 5-tuple. Afterward, 94 of 96 did, across 6 sequential and 10 parallel requests.

The two misses belonged to a connection that already existed when the probe attached and whose next operation was a write. There hadn’t been an observed read to seed its binding. It corrected itself on that connection’s next read. That’s a specific limitation of the observation history, and the remaining two chunks belong in the result.

These counts describe captured plaintext chunks, not requests or TCP packets. The before-and-after samples also had different sizes. They’re useful evidence about this workload; they aren’t a general accuracy claim for every rustls application.

What I’d change first about the investigation is the measurement. I spent a week measuring outcomes when a few counters could have told me whether the proposed mechanism had any input. The empty map made two opposite cache experiments look like meaningful failures. Neither had tested what I thought it had.

I’d also stop treating a symbol as proof of execution. Generics and inlining let a perfectly legitimate named function survive in a binary while the relevant workload takes different compiled paths. Five callers didn’t answer which callers ran.

And I’d keep correctness separate from liveness. An open socket can be the wrong socket for as long as the pool keeps it open. Once a binding is wrong, checking that its target still exists can make the error remarkably persistent.

Finally, I’d add the second connection at the start. The single-connection test couldn’t distinguish working correlation from lucky correlation. rustls didn’t owe us a file descriptor, and Tokio didn’t owe us a convenient function entry. We had to find the point where the evidence actually connected the plaintext to the socket, then test with enough sockets to know the difference.