perf: reduce TAP framer per-packet I/O #465

Closed

caution-bot wants to merge 49 commits from perf/tap-framer-io into fix/tap-vsock-framing

caution-bot commented

2026-10-03 07:43:52 +00:00

(Migrated from codeberg.org)

Summary

Stacked on #464, targeting fix/tap-vsock-framing, not main.

Emit each existing length prefix and Ethernet frame together in one stream write when the writer accepts the whole record. Keep short-write retries and immediate per-record flushing; no batch-fill delay.
Decode through a bounded 8 KiB BufReader, amortizing stream reads without changing the four-byte wire format, frame bounds, one-write-per-TAP-packet rule, synchronous TAP startup, or independent duplex workers.
Add RED→GREEN regression tests for combined writes and cross-record read amortization. One source file changed; no routing, MTU, egress policy, dependencies, or deployment changes. The internal send-error operation label is now write frame for the combined operation; underlying errors still propagate.

Base: a12a7f0f2a1c5ef4bbc321825fb298aa7fde82cb.
Candidate: 028031cabbdc6a2b5c20f74d2842128b27d02287.
Tested/reviewed tree: 2a70372093d180c2532f772ba637221b70fc3148.

Measured local performance — not Nitro

Two real TAPs in separate, owned user/network namespaces, real TCP, AF_UNIX replacing vsock, pinned production-build static helpers. MTU 1500; 512 MiB per direction; five interleaved repetitions; both mixed old/new pairings. All 24 GiB of bulk payload passed byte-count/SHA-256 and EOF checks. All owned process groups were removed; parent interfaces were unchanged.

Workload	Baseline MiB/s	Optimized MiB/s	Median throughput gain	Helper CPU/GiB reduction
Forward	124.9	217.3	74.0%	46.2%
Reverse	140.4	237.9	69.4%	42.1%
Simultaneous duplex, sum of directions	186.4	358.2	92.2%	48.0%

These are local comparison results, not native-vsock capacity, internet transfer rates, or promised production gains. Python endpoints, hashing, kernel TCP, backpressure and uncontrolled host scheduling contribute. CPU is both helper processes' user+system time per delivered GiB, excluding endpoints.

A separate release-mode socket microbenchmark verified 60 samples / 8.2 million frames against unchanged production functions: send throughput +33–34%, receive throughput +122–134%; bulk forwarding CPU/frame down about 25% and 55–57%, respectively. It uses a datagram TAP surrogate, not real TAP.

Limitations / counter-evidence:

Both versions recorded kernel TAP TX queue drops under load: baseline 1,373, candidate 1,686, across equal bulk payloads. Payload verification passed, but this is not zero-loss evidence. TCP retransmission counts were not captured; queue behavior/tuning is deferred. No intentional packet-drop policy was introduced.
Latency is not a blanket win: local TCP echo median-of-run p99 was 161.591 → 149.692 µs, but the synthetic 1,514-byte serial roundtrip p50 was 30.941 → 35.550 µs (p99 43.680 → 42.061 µs). Placement/scheduling noise remains; the change is supported primarily by bulk throughput and CPU evidence.
Native-vsock loopback was unavailable locally (ENODEV). No new Nitro deployment or native-vsock performance validation was performed. Production gains require a separate matched managed-Nitro comparison using the established dev workflow.
Optimized and baseline framed helpers interoperated locally. This does not make the original raw/framed protocols compatible or remove #464's paired host/EIF rollout requirement.

Test plan and results

make test-unit, invoked via /tmp/tap-framer-io-evidence/regular_tests.py with a fresh, disposable PostgreSQL 17 container and cloud credentials removed: 910 passed, 0 failed, 3 ignored. Test container removed and absence verified.
make test-cli-install: passed.

Standalone helper: 21 passed, 0 ignored, including actual TAP lifecycle, in an owned user/network namespace:

unshare --user --map-root-user --net env RUSTUP_TOOLCHAIN=1.96.1 \
  cargo test --locked --manifest-path src/enclave-builder/templates/tap-framer/Cargo.toml \
  --target-dir /tmp/platform-tap-framer-target -- --include-ignored

RUSTUP_TOOLCHAIN=1.96.1 CARGO_TARGET_DIR=/tmp/platform-tap-framer-target make test-tap-framer: 8 passed against the normal binary.
unshare --user --map-root-user --net python3 tests/test_tap_framer.py /tmp/tap-framer-io-evidence/static/tap-framer: 8 passed against the pinned static binary.
cargo fmt --manifest-path src/enclave-builder/templates/tap-framer/Cargo.toml -- --check and cargo clippy --locked --manifest-path src/enclave-builder/templates/tap-framer/Cargo.toml --all-targets -- -D warnings, both with RUSTUP_TOOLCHAIN=1.96.1: passed.
Pinned production tap-framer-builder stage exported with Docker BuildKit for both immutable source snapshots. Contexts compared byte-for-byte to Git. Rebuilt baseline matches the prior framing-fix static artifact; benchmark source/executable hashes and raw logs independently verified.
Independent frozen-tree review: no blocking security/correctness findings; two optional test-hardening suggestions retained in the evidence. Secret scan and git diff --check passed. Local merge-tree against the actual base equals the tested tree.

Reproducible benchmark runners, raw samples, production build contexts, artifact checksums and review are provided in the attached evidence archive. Its README contains the exact benchmark/build commands. Hostnames were omitted from public metadata; no credentials or production inputs are included.

Merge/review verdict: small, validated performance change on top of #464. Production verdict: not deployed; native-vsock/Nitro qualification remains separate. No issue auto-close directive is appropriate for this user-requested optimization.

Reproducible evidence

Download benchmark evidence (tap-framer-io-028031c-evidence.zip, 163,853 bytes).

SHA-256: 6584f0565c689f69aa8bc853eeea986fae7a02587cc429104a6804e97251a1ce. Uploaded asset was downloaded and checksum-verified.

## Summary **Stacked on #464**, targeting `fix/tap-vsock-framing`, not `main`. - Emit each existing length prefix and Ethernet frame together in one stream write when the writer accepts the whole record. Keep short-write retries and immediate per-record flushing; no batch-fill delay. - Decode through a bounded **8 KiB `BufReader`**, amortizing stream reads without changing the four-byte wire format, frame bounds, one-write-per-TAP-packet rule, synchronous TAP startup, or independent duplex workers. - Add RED→GREEN regression tests for combined writes and cross-record read amortization. One source file changed; no routing, MTU, egress policy, dependencies, or deployment changes. The internal send-error operation label is now `write frame` for the combined operation; underlying errors still propagate. Base: `a12a7f0f2a1c5ef4bbc321825fb298aa7fde82cb`. Candidate: `028031cabbdc6a2b5c20f74d2842128b27d02287`. Tested/reviewed tree: `2a70372093d180c2532f772ba637221b70fc3148`. ## Measured local performance — **not Nitro** Two real TAPs in separate, owned user/network namespaces, real TCP, **AF_UNIX replacing vsock**, pinned production-build static helpers. MTU 1500; 512 MiB per direction; five interleaved repetitions; both mixed old/new pairings. All **24 GiB** of bulk payload passed byte-count/SHA-256 and EOF checks. All owned process groups were removed; parent interfaces were unchanged. | Workload | Baseline MiB/s | Optimized MiB/s | Median throughput gain | Helper CPU/GiB reduction | |---|---:|---:|---:|---:| | Forward | 124.9 | 217.3 | 74.0% | 46.2% | | Reverse | 140.4 | 237.9 | 69.4% | 42.1% | | Simultaneous duplex, **sum of directions** | 186.4 | 358.2 | 92.2% | 48.0% | These are local comparison results, **not native-vsock capacity, internet transfer rates, or promised production gains**. Python endpoints, hashing, kernel TCP, backpressure and uncontrolled host scheduling contribute. CPU is both helper processes' user+system time per delivered GiB, excluding endpoints. A separate release-mode socket microbenchmark verified **60 samples / 8.2 million frames** against unchanged production functions: send throughput +33–34%, receive throughput +122–134%; bulk forwarding CPU/frame down about 25% and 55–57%, respectively. It uses a datagram TAP surrogate, not real TAP. **Limitations / counter-evidence:** - Both versions recorded kernel TAP TX queue drops under load: baseline **1,373**, candidate **1,686**, across equal bulk payloads. Payload verification passed, but this is **not zero-loss evidence**. TCP retransmission counts were not captured; queue behavior/tuning is deferred. No intentional packet-drop policy was introduced. - Latency is not a blanket win: local TCP echo median-of-run p99 was 161.591 → 149.692 µs, but the synthetic 1,514-byte serial roundtrip p50 was **30.941 → 35.550 µs** (p99 43.680 → 42.061 µs). Placement/scheduling noise remains; the change is supported primarily by bulk throughput and CPU evidence. - Native-vsock loopback was unavailable locally (`ENODEV`). **No new Nitro deployment or native-vsock performance validation** was performed. Production gains require a separate matched managed-Nitro comparison using the established dev workflow. - Optimized and baseline **framed** helpers interoperated locally. This does not make the original raw/framed protocols compatible or remove #464's paired host/EIF rollout requirement. ## Test plan and results - `make test-unit`, invoked via `/tmp/tap-framer-io-evidence/regular_tests.py` with a fresh, disposable PostgreSQL 17 container and cloud credentials removed: **910 passed, 0 failed, 3 ignored**. Test container removed and absence verified. - `make test-cli-install`: passed. - Standalone helper: **21 passed, 0 ignored**, including actual TAP lifecycle, in an owned user/network namespace: ```sh unshare --user --map-root-user --net env RUSTUP_TOOLCHAIN=1.96.1 \ cargo test --locked --manifest-path src/enclave-builder/templates/tap-framer/Cargo.toml \ --target-dir /tmp/platform-tap-framer-target -- --include-ignored ``` - `RUSTUP_TOOLCHAIN=1.96.1 CARGO_TARGET_DIR=/tmp/platform-tap-framer-target make test-tap-framer`: **8 passed** against the normal binary. - `unshare --user --map-root-user --net python3 tests/test_tap_framer.py /tmp/tap-framer-io-evidence/static/tap-framer`: **8 passed** against the pinned static binary. - `cargo fmt --manifest-path src/enclave-builder/templates/tap-framer/Cargo.toml -- --check` and `cargo clippy --locked --manifest-path src/enclave-builder/templates/tap-framer/Cargo.toml --all-targets -- -D warnings`, both with `RUSTUP_TOOLCHAIN=1.96.1`: passed. - Pinned production `tap-framer-builder` stage exported with Docker BuildKit for both immutable source snapshots. Contexts compared byte-for-byte to Git. Rebuilt baseline matches the prior framing-fix static artifact; benchmark source/executable hashes and raw logs independently verified. - Independent frozen-tree review: no blocking security/correctness findings; two optional test-hardening suggestions retained in the evidence. Secret scan and `git diff --check` passed. Local merge-tree against the actual base equals the tested tree. Reproducible benchmark runners, raw samples, production build contexts, artifact checksums and review are provided in the attached evidence archive. Its README contains the exact benchmark/build commands. Hostnames were omitted from public metadata; no credentials or production inputs are included. **Merge/review verdict:** small, validated performance change on top of #464. **Production verdict:** not deployed; native-vsock/Nitro qualification remains separate. No issue auto-close directive is appropriate for this user-requested optimization. ## Reproducible evidence [Download benchmark evidence](https://codeberg.org/attachments/d5a833f4-6ddf-4809-9cf7-549deed3c306) (`tap-framer-io-028031c-evidence.zip`, 163,853 bytes). SHA-256: `6584f0565c689f69aa8bc853eeea986fae7a02587cc429104a6804e97251a1ce`. Uploaded asset was downloaded and checksum-verified.

caution-bot commented

2026-10-07 01:43:13 +00:00

(Migrated from codeberg.org)

Closing as incorporated into #464 at 5cd86be51695f60f9c76015f2d5211f9d77775f0. That PR now contains this optimization and its regression tests, together with the matching reviewed host-helper checksum pin. Verified the helper source matches this PR byte-for-byte after relocation to src/tap-framer/.

This is a superseded-PR closure, not a merge into main; #464 remains open. The source branch is retained.

Closing as incorporated into #464 at `5cd86be51695f60f9c76015f2d5211f9d77775f0`. That PR now contains this optimization and its regression tests, together with the matching reviewed host-helper checksum pin. Verified the helper source matches this PR byte-for-byte after relocation to `src/tap-framer/`. This is a superseded-PR closure, not a merge into `main`; #464 remains open. The source branch is retained.

Pull request closed

This pull request cannot be reopened because the branch was deleted.

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

public/platform!465

No description provided.

Rows
Columns