---
title: "Stream BYOC API Traffic into Apache Iceberg | Speedscale"
description: "Stream captured API traffic into Apache Iceberg with Speedscale BYOC. Set up AWS S3 Tables and query deduplicated HTTP errors in DuckDB."
canonical: "https://speedscale.com/blog/byoc-api-traffic-apache-iceberg/"
---

[Site index](https://speedscale.com/blog/llms.txt) | [Markdown](https://speedscale.com/blog/byoc-api-traffic-apache-iceberg/index.md) | [Canonical page](https://speedscale.com/blog/byoc-api-traffic-apache-iceberg/)

![Aerial view of a layered iceberg with teal currents flowing around it](https://speedscale.com/_astro/featured-image.DJW6uwsu_1Wod9D.webp)

# Stream BYOC API Traffic into Apache Iceberg

Nov 19, 2026

[By Speedscale Team](https://speedscale.com/blog/author/speedscale-team/)

[Observability](https://speedscale.com/blog/category/observability/) [Cloud native](https://speedscale.com/blog/category/cloud-native/) [Security](https://speedscale.com/blog/category/security/)

* * *

If your team already investigates incidents in a lakehouse, captured API traffic belongs there too. You can join response failures with deployment events in the same query and inspect the captured exchange when a count needs explaining. Speedscale Bring Your Own Cloud (BYOC) can now stream that traffic into Apache Iceberg tables in your cloud account, under your catalog, storage, and access policies. The export path keeps the captured traffic in your environment.

We extended the OpenTelemetry path BYOC already uses. The new `iceberg-sink` service was the easy part to draw on a whiteboard. Getting concurrent writers to commit reliably and keeping the metadata from growing faster than the data took more work. In our live AWS S3 Tables spike, four writers committed 72,228 rows with zero commit failures. DuckDB read back all 72,228, with no duplicate UUIDs.

## The OpenTelemetry path we already had

The Speedscale Forwarder already sends request/response pairs (RRPairs) over OTLP/gRPC to BYOC channels such as S3, GCS, Azure Blob, Datadog, Loki, and Elasticsearch. Iceberg uses that same exporter interface. Add a named entry under `forwarder.exporters` and point it at the sink. The Forwarder applies the entry’s filter rule and data loss prevention (DLP) configuration before sending the traffic. You can redact payload fields more aggressively for the lakehouse while keeping the replay copy on its own policy. See the [BYOC architecture](https://docs.speedscale.com/byoc/how-it-works/) and [Forwarder configuration](https://docs.speedscale.com/byoc/configure-kubernetes/#4-configure-named-forwarder-exporters).

The OpenTelemetry Collector contrib distribution had no Iceberg exporter when we evaluated it. The community exporters we reviewed write generic log schemas, leaving the RRPair in a JSON body. To count errors by service or filter by cluster, an engine would have to pull those fields out of JSON. We wanted those fields available as columns for direct filters and aggregation. See the [Collector registry](https://opentelemetry.io/ecosystem/registry/?language=collector&component=exporter) and the [EDB](https://github.com/enterprisedb/icebergexporter) and [terakoya76](https://github.com/terakoya76/opentelemetry-iceberg-exporter) exporters.

The sink decodes each OTLP record and writes one typed row per request/response pair, partitioned by `day(ts)` and `cluster`. The columns are `ts`, `uuid`, `cluster`, `namespace`, `service`, `direction`, `l7protocol`, `tech`, `command`, `location`, `status`, `elapsed_us`, `is_tls`, `tags` (a map), `dlp_modified`, and `ingested_at`. The `rrpair` JSON column retains the full pair as received by the sink, after the channel’s DLP processing.

Choose AWS S3 Tables, AWS Glue, or an Iceberg REST catalog such as Polaris, Lakekeeper, Nessie, or Unity Catalog. The sink supports S3, Google Cloud Storage, and Azure Blob Storage as storage backends. Use the Iceberg query engine that already fits your catalog and permissions, such as Athena, Trino, Snowflake, Databricks, or DuckDB. See the [Iceberg BYOC channel documentation](https://docs.speedscale.com/byoc/backends/#iceberg).

## Set up the Iceberg channel with AWS S3 Tables

### 1\. Deploy the sink

Start with an existing Speedscale BYOC deployment on Kubernetes, an AWS S3 table bucket, and an EKS workload identity with access to that bucket and its tables. This example uses IAM Roles for Service Accounts (IRSA). The role must trust the `byoc-iceberg-iceberg` service account in the `byoc-iceberg` namespace. Save the following as `values-iceberg.yaml`, replacing the sample region, account, bucket, and role. Use the table bucket ARN as the warehouse. The [chart README](https://github.com/speedscale/speedscale-byoc/tree/main/charts/iceberg#aws-s3-tables) lists the required permissions.

```
iceberg:
  catalog:
    type: s3tables
    uri: ""
    warehouse: arn:aws:s3tables:us-east-1:111122223333:bucket/speedscale-traffic
awsRegion: us-east-1
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/byoc-iceberg
```

From the directory containing that file, install the chart in the Kubernetes context that runs your BYOC Forwarder. On startup, the sink creates the default namespace and table, `speedscale.rrpairs`, if they are missing.

```
helm repo add speedscale-byoc https://speedscale.github.io/speedscale-byoc/
helm repo update speedscale-byoc
helm upgrade --install byoc-iceberg speedscale-byoc/iceberg \
  --namespace byoc-iceberg --create-namespace -f values-iceberg.yaml
```

LLM prompt for this step

```
Help me deploy the Speedscale BYOC Iceberg sink to my existing EKS cluster. Ask for my Kubernetes context, AWS region, S3 table bucket ARN, and IRSA role ARN. Use the AWS S3 Tables example and the published iceberg chart README. Generate values-iceberg.yaml, confirm the role trusts the chart's service account, and render the chart to verify the warehouse and workload identity. Show me the Helm install command for the byoc-iceberg namespace and verify the deployment becomes ready after I approve applying it.
```

### 2\. Add a Forwarder exporter

Add the entry below to your existing Speedscale operator Helm values and apply the update through your usual release workflow. Keep your existing exporters in that file. The hostname matches the release and namespace from step 1. Replace the sample filter and DLP policy IDs with the ones approved for your lakehouse channel. If your Forwarder runs outside the default `speedscale` namespace, include its namespace in `networkPolicy.allowedNamespaces` in the sink chart values.

```
forwarder:
  exporters:
    byoc_iceberg:
      otel_endpoint: "http://byoc-iceberg-iceberg.byoc-iceberg.svc.cluster.local:4317"
      filter_rule: standard
      dlp_config_id: standard
```

Send a request to a captured service that matches the filter, then check the sink’s `iceberg_sink_rows_committed_total` metric on port 8080. The default commit interval is 60 seconds, and the sink starts a commit sooner when the batch reaches 8 MiB.

LLM prompt for this step

```
Find the Helm values that manage my Speedscale BYOC Forwarder. Add a byoc_iceberg exporter using the sink Service DNS name and OTLP/gRPC port 4317. Preserve all existing exporters and ask which approved filter_rule and dlp_config_id to use for the lakehouse copy. Check that the sink's NetworkPolicy admits my Forwarder's namespace. Show the values diff and apply command. After I send a captured request that matches the filter, verify the sink's committed-row counter increases.
```

### 3\. Count HTTP errors in DuckDB

Run the following in DuckDB with an AWS identity that can read the table. Use the same table bucket ARN and region as in step 1. The sink buffers pending rows in memory. Delivery is at least once, so a retried export can land twice. The query keeps the earliest-ingested row for each `uuid` in the last hour, then counts HTTP 5xx responses by service. The `status` column is a string, so the query converts it to an integer before comparing it. See the [DuckDB S3 Tables reference](https://duckdb.org/docs/current/core_extensions/iceberg/amazon_s3_tables).

```
INSTALL iceberg;
INSTALL aws;
LOAD iceberg;
LOAD aws;

CREATE SECRET byoc_s3_tables (
  TYPE s3,
  PROVIDER credential_chain,
  REGION 'us-east-1'
);

ATTACH 'arn:aws:s3tables:us-east-1:111122223333:bucket/speedscale-traffic'
  AS lake (TYPE iceberg, ENDPOINT_TYPE s3_tables);

WITH recent AS (
  SELECT service, l7protocol, TRY_CAST(status AS INTEGER) AS status_code
  FROM lake.speedscale.rrpairs
  WHERE ts >= CURRENT_TIMESTAMP - INTERVAL '1 hour'
    AND ts <= CURRENT_TIMESTAMP
  QUALIFY ROW_NUMBER() OVER (PARTITION BY uuid ORDER BY ingested_at) = 1
)
SELECT service, COUNT(*) AS http_5xx
FROM recent
WHERE LOWER(l7protocol) = 'http' AND status_code BETWEEN 500 AND 599
GROUP BY service
ORDER BY http_5xx DESC, service;
```

LLM prompt for this step

```
Use DuckDB's Iceberg extension and my approved AWS read identity to attach my S3 table bucket as lake. Ask me for the region and ARN if needed, and use the AWS credential chain. Run the article's last-hour query against lake.speedscale.rrpairs. Keep the earliest-ingested row per uuid before counting HTTP 5xx responses by service. Return the aggregate counts and, if they are empty, check whether any rows have a capture timestamp in the last hour.
```

## Four lessons from streaming into Iceberg

The November 19, 2026 spike streamed synthetic HTTP captures through an Iceberg REST catalog on S3-compatible storage, then exercised AWS S3 Tables live. The local commit latencies below come from the REST setup. They describe the test environment and workload.

### Commit retries decide whether writers can share a table

With iceberg-go v0.7.0 and four concurrent writers, three writers failed every commit. About 176,000 of 235,000 rows never landed. All four writers were trying to update the same table, and that version defaults `commit.retry.num-retries` to zero. A writer that lost the race got no second attempt at the catalog commit.

With retries set to 10, the four- and eight-writer tests committed every generated row. The eight-writer readback found no duplicate UUIDs. Local p95 commit latency was 388 ms and 410 ms, respectively. The sink explicitly configures 10 catalog retries and gives each batch an ID so it can check whether an append landed before resubmitting after an ambiguous failure. Upstream has since changed its default to four retries. See the [iceberg-go retry change](https://github.com/apache/iceberg-go/pull/2462).

### Fast commits can make metadata bigger than the data

At one commit per second per writer, our two-writer test accumulated 1,008 snapshots in 10 minutes. Each new table metadata file carried the growing snapshot list. Keeping those files around left us with 435 MB of metadata for 365 MB of data.

In a separate five-minute run with snapshot expiry and old-metadata deletion enabled, we wrote 187 MB of data and retained 8.4 MB of metadata. Commit latency stayed around 160 ms. Snapshot expiry bounds the list in the current metadata file, and old-metadata cleanup removes superseded files. Both matter when commits are frequent.

### Catalog choice determines who maintains the table

S3 Tables compacts data files and expires snapshots itself. During our live test, it accepted a client-side expiry commit, then denied the client’s attempt to delete old files. The sink’s `auto` maintenance mode leaves S3 Tables to handle that work. See [AWS S3 Tables maintenance](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html).

For Glue and REST catalogs, `auto` enables snapshot expiry and old-metadata cleanup in the sink. Glue optimizers need to be enabled for new tables. If you enable Glue’s snapshot retention optimizer, turn off the sink’s expiry with `iceberg.maintenance: off`. For a REST catalog that leaves compaction to you, or a Glue table without the compaction optimizer enabled, set `compaction.enabled: true` in the chart to schedule `iceberg-sink compact`. See the [Glue table optimizer documentation](https://docs.aws.amazon.com/glue/latest/dg/table-optimizers.html) and the [chart’s compaction settings](https://github.com/speedscale/speedscale-byoc/tree/main/charts/iceberg#compaction).

### A separate service keeps compaction out of the capture path

Linking the Iceberg libraries for the supported catalogs and storage backends added 44.5 MB to our stripped Linux AMD64 Forwarder test binary. In a separate test, compacting 4,032 small data files into four peaked at 653 MiB of heap. That heap alone was above the Forwarder’s 512M memory limit.

The sink runs as a separate deployment with its own workload identity and memory limit. The compaction job runs from the same image with a separate memory budget. The existing Forwarder keeps its OTLP interface, binary, and credentials.

## Frequently asked questions

### How soon does captured traffic show up in the Iceberg table?

By default, the sink starts a commit every 60 seconds or when its batch reaches 8 MiB, whichever comes first. Rows become visible after the catalog accepts the commit.

### Can I keep a separate copy of traffic for replay?

Yes. Keep a supported object-storage replay channel alongside Iceberg. Each channel has its own filter rule and DLP configuration, so you can retain different fields for SQL analysis and replay.

### Does the chart provision the cloud resources?

Provision the table bucket or warehouse and workload identity first. The sink creates the namespace and table if they are missing and its identity has permission.

## Table of Contents

## See Speedscale in action

Book a 30-minute demo to see how Speedscale handles your specific stack — microservices, Kubernetes, or legacy Java services.

[Book a demo](https://speedscale.com/company/demo/) [Get started](https://app.speedscale.com/signup)
