public/platform

Fork 0

Fix Ethernet framing in the TAP/vsock egress tunnel #464

Merged

caution-bot merged 0 commits from fix/tap-vsock-framing into main

2026-10-07 22:46:23 +00:00

caution-bot commented

2026-10-02 23:34:34 +00:00

(Migrated from codeberg.org)

Summary

Fix the TAP/vsock packet/stream mismatch, with the optimization and regression tests from #465 now incorporated here (#465 was closed as incorporated). Signed head 3321efdbd641b46b343cbe641fee7a313dcbdcec is one commit on main (456e558c8222cf72c9ca425def5b38efe5acd29b).

Preserve Ethernet boundaries with bounded length framing, partial/interrupted I/O handling, independent directions, TAP-UP handling and optimized frame forwarding. Unrelated ingress/egress policy is unchanged.
Managed and BYOC bootstrap verifies the host helper against the independently reviewed src/tap-framer/host.sha256 before root installation or execution. A builder-generated checksum is not a trust anchor.
containerfiles/Containerfile.tap-framer is the canonical pinned build/export recipe. The standalone verifier invokes it directly; EIF rendering includes the whole file from the same selected framework revision as the helper source. The API image packages the matching source layout.
No-egress builds omit the helper and do not require its recipe/source. Historical inline recipes and source layouts remain supported. CLI reproduction errors retain selected-framework identity and upgrade guidance.
The standalone target exports a binary and candidate checksum; neither it nor the verifier updates the trusted pin or publishes a release. Host/guest pairing and per-EIF sidecar delivery are retained. Platform-wide distribution and #468's cache are outside scope.

These preserve the integrity, egress gating and reproduction fixes discussed in https://codeberg.org/caution/platform/pulls/464#issuecomment-24787674. The CLI README compatibility section remains removed. The TAP README now uses thin make build-tap-framer / make verify-tap-framer wrappers; substantive tests and verification remain.

README and Make wrappers at `3321efd`

The component README is reduced to 17 lines. make build-tap-framer exports through the existing canonical Containerfile; make verify-tap-framer invokes the existing release-pin verifier. Neither updates the trusted pin.

make -n build-tap-framer and make -n verify-tap-framer passed; their argument lists were checked against the previously documented commands and match exactly. git diff --check 643f41e2f26f3c8a0a018fc3c51a0a16cd3f89f8..3321efdbd641b46b343cbe641fee7a313dcbdcec passed. This follow-up changes only Makefile/README; no actual build or runtime suite was rerun.

Alias-only follow-up at `643f41e`

The standalone helper imports are named rust and ca-certificates; their references were updated. The enclosing EIF template uses eif-ca-certificates for its own certificate stage to avoid duplicate names when the recipe is included.
Static comparison against 5339c04 confirms both Containerfiles differ only by consistent stage-alias renames. Production Rust, helper source, lockfiles, image pins and the trusted binary pin are unchanged. Standalone and combined stage graphs (including no-egress inspection) have unique names, resolved references and no cycles; a naive-rename negative control correctly detects the duplicate certificate stage.
rustfmt --edition 2021 --check src/enclave-builder/src/build.rs and git diff --check 5339c049ca92b569d9e1f3d13db8d4e383b09e46..643f41e2f26f3c8a0a018fc3c51a0a16cd3f89f8 passed. The staging test now checks for duplicate stage aliases and expects the new names.
No fresh Rust tests or container builds were run for this alias-only follow-up. The runtime and independent-review evidence below applies to 5339c04, not the current head.

Runtime validation at `5339c04` (before alias cleanup)

On an owned, role-free disposable EC2 host; no builds, tests or installs on the maintainer/live-dev host. Exact Git commit/tree were checked before/after, with a clean source worktree. Native Rust 1.96.1 used locked/offline cached dependencies; helper container builds used pinned images, locked fetches and network-disabled frozen compilation.

cargo test --locked -p enclave-builder --lib: passed, including selected-recipe/missing-input/no-egress/historical-layout/archive regressions. The selected-recipe regression was first observed failing on the old implementation.
make test: 924 Rust passed, 0 failed, 3 ignored; CLI installer tests passed.
unshare --user --map-root-user --net python3 tests/test_tap_framer.py <export>/tap-framer: 8 real Linux TAP tests passed.
unshare --user --map-root-user --net cargo test --locked --manifest-path src/tap-framer/Cargo.toml -- --ignored: 1 TAP FD-lifetime test passed, covering one regular-gate ignore. Database projection and live-Nitro tests remain unrun here.
python3 tests/test_tap_framer_release.py: passed. A disposable copy with a valid-looking wrong pin failed specifically with host release pin mismatch and left its pin unchanged.
docker buildx build --progress=plain --platform linux/amd64 --no-cache -f containerfiles/Containerfile.tap-framer --target tap-framer-export --output type=local,dest=<export> .: passed.
The standalone export, the helper target through the actual EIF staging API, and the same path using source copied by the actual API Containerfile COPY instructions produced byte-identical static x86-64 executables: 827,328 bytes, mode 0755, SHA-256 e50265d7f57ce2ade393714b2c52f3173d288b348598f5a85aad1a775f49f283. The reviewed pin and helper source/lockfiles were not changed.

Native dependency fetching used retained source-only Git objects checked against locked commit/tree identities and checksum-verified registry packages. Direct Codeberg dependency transport was not tested in this run. This does not weaken the lockfiles or substitute precompiled helper artifacts.

Evidence archive: execution-evidence.tar.gz, SHA-256 6b7fe54a08726d377094e25bbcff04780f3451198512b6acd0a30941c12d9721 (1,097,994 bytes). Command logs, failed/successful attempts, recipes, driver and all three outputs are retained; transfer/member hashes and artifact equality were independently rechecked. The first candidate's archive-test fixture mistake was corrected and the final head rerun.

The owned instance was terminated; its EBS volume, ENI and security group were confirmed absent. The watchdog stopped and temporary SSH credentials were removed. Protected configurations/live services matched the before inventory; only separately recorded IAM usage telemetry was excluded from that comparison. Original worktrees and unrelated branches were preserved.

Independent exact-head source review: no actionable findings in the seven-file delta at 5339c049ca92b569d9e1f3d13db8d4e383b09e46. The earlier archive-fixture finding in 395cf8c was fixed and the final candidate rerun. This review was source-only; execution evidence above is separately retained.

Rollout and evidence limits

No merge or deployment. These are helper-target builds, not full EIF/PCR equality, a full API-image build, native-vsock Nitro qualification, production performance or independent attestation qualification. The staging driver used explicitly synthetic application directories and the real production staging implementation.

Raw and framed peers are incompatible: replace/roll back host and guest together and approve changed measurements normally. The earlier PR description records managed-Nitro tests at 153c9f7fcdbfc1370b2916124ce5d780997cc155 on 2026-10-03; those are historical, not execution of this head. That run's first-use-pinned host key (independent console verification was permission-blocked) and intermediate queue drops preclude independent host-identity and zero-loss/soak claims. Full BYOC/customer IAM, long-duration behavior and production workloads remain outside this fresh validation. Historical versioned-S3 test-object cleanup remains permission-blocked and unrelated to this runner's verified cleanup.

## Summary Fix the TAP/vsock packet/stream mismatch, with the optimization and regression tests from #465 now incorporated here (#465 was closed as incorporated). Signed head `3321efdbd641b46b343cbe641fee7a313dcbdcec` is one commit on `main` (`456e558c8222cf72c9ca425def5b38efe5acd29b`). - Preserve Ethernet boundaries with bounded length framing, partial/interrupted I/O handling, independent directions, TAP-UP handling and optimized frame forwarding. Unrelated ingress/egress policy is unchanged. - Managed and BYOC bootstrap verifies the host helper against the independently reviewed `src/tap-framer/host.sha256` **before root installation or execution**. A builder-generated checksum is not a trust anchor. - `containerfiles/Containerfile.tap-framer` is the canonical pinned build/export recipe. The standalone verifier invokes it directly; EIF rendering includes the whole file from the **same selected framework revision** as the helper source. The API image packages the matching source layout. - No-egress builds omit the helper and do not require its recipe/source. Historical inline recipes and source layouts remain supported. CLI reproduction errors retain selected-framework identity and upgrade guidance. - The standalone target exports a binary and candidate checksum; neither it nor the verifier updates the trusted pin or publishes a release. Host/guest pairing and per-EIF sidecar delivery are retained. Platform-wide distribution and #468's cache are outside scope. These preserve the integrity, egress gating and reproduction fixes discussed in https://codeberg.org/caution/platform/pulls/464#issuecomment-24787674. The CLI README compatibility section remains removed. The TAP README now uses thin `make build-tap-framer` / `make verify-tap-framer` wrappers; substantive tests and verification remain. ## README and Make wrappers at `3321efd` The component README is reduced to 17 lines. `make build-tap-framer` exports through the existing canonical Containerfile; `make verify-tap-framer` invokes the existing release-pin verifier. Neither updates the trusted pin. `make -n build-tap-framer` and `make -n verify-tap-framer` passed; their argument lists were checked against the previously documented commands and match exactly. `git diff --check 643f41e2f26f3c8a0a018fc3c51a0a16cd3f89f8..3321efdbd641b46b343cbe641fee7a313dcbdcec` passed. This follow-up changes only Makefile/README; **no actual build or runtime suite was rerun**. ## Alias-only follow-up at `643f41e` - The standalone helper imports are named `rust` and `ca-certificates`; their references were updated. The enclosing EIF template uses `eif-ca-certificates` for its own certificate stage to avoid duplicate names when the recipe is included. - Static comparison against `5339c04` confirms both Containerfiles differ only by consistent stage-alias renames. Production Rust, helper source, lockfiles, image pins and the trusted binary pin are unchanged. Standalone and combined stage graphs (including no-egress inspection) have unique names, resolved references and no cycles; a naive-rename negative control correctly detects the duplicate certificate stage. - `rustfmt --edition 2021 --check src/enclave-builder/src/build.rs` and `git diff --check 5339c049ca92b569d9e1f3d13db8d4e383b09e46..643f41e2f26f3c8a0a018fc3c51a0a16cd3f89f8` passed. The staging test now checks for duplicate stage aliases and expects the new names. - **No fresh Rust tests or container builds were run for this alias-only follow-up.** The runtime and independent-review evidence below applies to `5339c04`, not the current head. ## Runtime validation at `5339c04` (before alias cleanup) On an owned, role-free disposable EC2 host; **no builds, tests or installs on the maintainer/live-dev host**. Exact Git commit/tree were checked before/after, with a clean source worktree. Native Rust 1.96.1 used locked/offline cached dependencies; helper container builds used pinned images, locked fetches and network-disabled frozen compilation. - `cargo test --locked -p enclave-builder --lib`: passed, including selected-recipe/missing-input/no-egress/historical-layout/archive regressions. The selected-recipe regression was first observed failing on the old implementation. - `make test`: **924 Rust passed, 0 failed, 3 ignored**; CLI installer tests passed. - `unshare --user --map-root-user --net python3 tests/test_tap_framer.py <export>/tap-framer`: **8 real Linux TAP tests passed**. - `unshare --user --map-root-user --net cargo test --locked --manifest-path src/tap-framer/Cargo.toml -- --ignored`: **1 TAP FD-lifetime test passed**, covering one regular-gate ignore. Database projection and live-Nitro tests remain unrun here. - `python3 tests/test_tap_framer_release.py`: passed. A disposable copy with a valid-looking wrong pin failed specifically with `host release pin mismatch` and left its pin unchanged. - `docker buildx build --progress=plain --platform linux/amd64 --no-cache -f containerfiles/Containerfile.tap-framer --target tap-framer-export --output type=local,dest=<export> .`: passed. - The standalone export, the helper target through the **actual EIF staging API**, and the same path using source copied by the **actual API Containerfile COPY instructions** produced byte-identical static x86-64 executables: **827,328 bytes, mode 0755**, SHA-256 `e50265d7f57ce2ade393714b2c52f3173d288b348598f5a85aad1a775f49f283`. The reviewed pin and helper source/lockfiles were not changed. Native dependency fetching used retained source-only Git objects checked against locked commit/tree identities and checksum-verified registry packages. **Direct Codeberg dependency transport was not tested in this run.** This does not weaken the lockfiles or substitute precompiled helper artifacts. Evidence archive: `execution-evidence.tar.gz`, SHA-256 `6b7fe54a08726d377094e25bbcff04780f3451198512b6acd0a30941c12d9721` (1,097,994 bytes). Command logs, failed/successful attempts, recipes, driver and all three outputs are retained; transfer/member hashes and artifact equality were independently rechecked. The first candidate's archive-test fixture mistake was corrected and the final head rerun. The owned instance was terminated; its EBS volume, ENI and security group were confirmed absent. The watchdog stopped and temporary SSH credentials were removed. Protected configurations/live services matched the before inventory; only separately recorded IAM usage telemetry was excluded from that comparison. Original worktrees and unrelated branches were preserved. Independent exact-head source review: no actionable findings in the seven-file delta at `5339c049ca92b569d9e1f3d13db8d4e383b09e46`. The earlier archive-fixture finding in `395cf8c` was fixed and the final candidate rerun. This review was source-only; execution evidence above is separately retained. ## Rollout and evidence limits No merge or deployment. These are helper-target builds, **not full EIF/PCR equality, a full API-image build, native-vsock Nitro qualification, production performance or independent attestation qualification**. The staging driver used explicitly synthetic application directories and the real production staging implementation. Raw and framed peers are incompatible: replace/roll back host and guest together and approve changed measurements normally. The earlier PR description records managed-Nitro tests at `153c9f7fcdbfc1370b2916124ce5d780997cc155` on 2026-10-03; those are historical, not execution of this head. That run's first-use-pinned host key (independent console verification was permission-blocked) and intermediate queue drops preclude independent host-identity and zero-loss/soak claims. Full BYOC/customer IAM, long-duration behavior and production workloads remain outside this fresh validation. Historical versioned-S3 test-object cleanup remains permission-blocked and unrelated to this runner's verified cleanup.

vkobel commented

2026-10-06 12:53:01 +00:00

(Migrated from codeberg.org)

Review of a12a7f0f + 2d18b25f.

High

terraform/modules/aws/nitro-enclave/user-data.sh:57: the parent host downloads ${eif_s3_path}.tap-framer and runs it as root with no integrity check. The object is written by the per-build builder, whose role has PutObject on eifs/* for every org (infra-bootstrap/main.tf:483-491). A compromised builder can therefore plant root code on another org's host, and unlike the EIF, attestation doesn't cover it. The IMDS hop limit of 1 (src/api/src/ec2.rs:130) blocks the direct path from a customer docker build, so this needs a compromised builder. #468 adds a sha256 check only when a component set is pinned; source builds still run the helper unchecked. Fix: pin it the way remote-build-helper is pinned (HELPER_SHA256 + sha256sum -c), or build it in the API image, publish it from the platform, and pin it.

Medium

src/enclave-builder/templates/Containerfile.eif:122: the framework template now does COPY components/tap-framer/, and that directory only exists if the CLI's own code stages it (tap_framer::stage_sources). For reproduction, the CLI uses the framework archive's templates (lib.rs:703-715), so any CLI released before this PR fails caution verify on every new deploy with "/components/tap-framer: not found". Fix: decide on a minimum CLI version for verify and report a clear error for older CLIs; the next item limits the damage.
src/enclave-builder/templates/Containerfile.eif:115: the tap-framer-builder stage, the COPY into binaries (l.145), the initramfs install (l.229) and the output COPY (l.321) have no # {EGRESS markers. Every EIF builds and ships the helper even when egress is off: an extra cargo fetch + release build on every uncached build (builders have no layer cache), an unused root binary in the initramfs, and the old-CLI break above extends to apps without egress. Fix: gate them with # {EGRESS.

Finding	Severity	Impact	Effort
Unverified root sidecar	High	Root on another org's host if a builder is compromised	Low–Medium (pin the hash)
Old-CLI verify break	Medium	Every verify run with a CLI from before this PR	Low (minimum version + clear error)
No EGRESS gate	Medium	Every build and every app	Low (add markers)

Effect on #468: CLIs from before #464 can't verify any new deploy anyway. So the #468 High about manifest field order only affects CLIs released between #464 and #468

Review of `a12a7f0f` + `2d18b25f`. **High** - `terraform/modules/aws/nitro-enclave/user-data.sh:57`: the parent host downloads `${eif_s3_path}.tap-framer` and runs it as root with no integrity check. The object is written by the per-build builder, whose role has `PutObject` on `eifs/*` for every org (`infra-bootstrap/main.tf:483-491`). A compromised builder can therefore plant root code on another org's host, and unlike the EIF, attestation doesn't cover it. The IMDS hop limit of 1 (`src/api/src/ec2.rs:130`) blocks the direct path from a customer `docker build`, so this needs a compromised builder. #468 adds a sha256 check only when a component set is pinned; source builds still run the helper unchecked. Fix: pin it the way `remote-build-helper` is pinned (`HELPER_SHA256` + `sha256sum -c`), or build it in the API image, publish it from the platform, and pin it. **Medium** - `src/enclave-builder/templates/Containerfile.eif:122`: the framework template now does `COPY components/tap-framer/`, and that directory only exists if the CLI's own code stages it (`tap_framer::stage_sources`). For reproduction, the CLI uses the framework archive's templates (`lib.rs:703-715`), so any CLI released before this PR fails `caution verify` on every new deploy with "/components/tap-framer: not found". Fix: decide on a minimum CLI version for verify and report a clear error for older CLIs; the next item limits the damage. - `src/enclave-builder/templates/Containerfile.eif:115`: the `tap-framer-builder` stage, the COPY into binaries (l.145), the initramfs install (l.229) and the output COPY (l.321) have no `# {EGRESS` markers. Every EIF builds and ships the helper even when egress is off: an extra `cargo fetch` + release build on every uncached build (builders have no layer cache), an unused root binary in the initramfs, and the old-CLI break above extends to apps without egress. Fix: gate them with `# {EGRESS`. --- | Finding | Severity | Impact | Effort | | -------------------------------- | ----------------- | ------------------------------------------------------ | ----------------------------------- | | Unverified root sidecar | High | Root on another org's host if a builder is compromised | Low–Medium (pin the hash) | | Old-CLI verify break | Medium | Every verify run with a CLI from before this PR | Low (minimum version + clear error) | | No EGRESS gate | Medium | Every build and every app | Low (add markers) | **Effect on #468:** CLIs from before #464 can't verify any new deploy anyway. So the #468 High about manifest field order only affects CLIs released between #464 and #468

ryan referenced this pull request

2026-10-16 10:08:57 +00:00

feat: reuse digest-pinned enclave components from S3 #468

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

public/platform!464

No description provided.

Rows
Columns

Fix Ethernet framing in the TAP/vsock egress tunnel #464

Summary

README and Make wrappers at 3321efd

Alias-only follow-up at 643f41e

Runtime validation at 5339c04 (before alias cleanup)

Rollout and evidence limits

README and Make wrappers at `3321efd`

Alias-only follow-up at `643f41e`

Runtime validation at `5339c04` (before alias cleanup)