fix: publish enclave metadata atomically without clobbering #18

Closed

anton wants to merge 0 commits from fix/atomic-metadata-publication into main

anton commented

2026-10-19 00:27:00 +00:00

Owner

Summary

Fix the normal STEVE-enabled enclave startup race in metadata publication, independently of the platform E2E-speed changes.

write_metadata_to previously created the final /metadata.json pathname before streaming JSON into it. Normal enclave startup launches STEVE and bootproofd asynchronously; bootproofd can open the existing but empty/partial file and exit on the JSON error.

Serialize into a RAII-managed temporary file in the destination directory; ordinary error paths clean it up without exposing partial destination bytes.
Publish the completed file with persist_noclobber: no partial final pathname and no replacement of an existing destination, including a concurrently created file or dangling symlink.
Preserve public metadata bytes and the original Unix 0666 & !umask permissions; support both basename-only and nested relative paths.
Keep startup ordering, reader error policy, timeouts, test scheduling and wire format unchanged.

Regression evidence

The deterministic regression calls the actual serialization/publication helper, checks during serialization after the first JSON field, and requires the final pathname to remain absent without inspecting temporary-file internals. No timing sleeps or probability of a racing reader are needed.

Before: failed with metadata destination became visible before serialization completed (exit 101).
After: passes. Additional coverage checks serialization-failure cleanup, a destination created during serialization, dangling symlinks, basename-only paths, and Unix permissions.

Validation

cargo test --offline --locked --workspace --all-targets --all-features: 144 passed, 0 failed, 2 ignored. Ignored tests are not counted as executed.
Native STEVE binary + the unchanged bootproofd binary from the failing image: 30/30 offline startup-order cases passed (reader before publication, after publication, and concurrent start). Each checks daemon health and the real /attestation handler's metadata-read path using an invalid nonce. This is not NSM/cryptographic attestation.
Full strict workspace Clippy has the same pre-existing await_holding_lock diagnostic at crates/steve-sdk/src/tests.rs:966 as untouched main; no unrelated SDK cleanup is included.
Independent review: approved, no blocking findings, bound to the exact published diff. Scoped formatting/strict Clippy passed. Workspace formatting findings were also reproduced on unchanged SDK code.
Separate platform adoption: public/platform#486.

Test-only cleanup

Removed 52 net lines: the redundant threaded-writers test, exact partial temporary-file byte assertions, and the supplementary nested-relative-path case. Production code is byte-identical to d0703497e5c7b04871d43fed8407d1d1a09ea892. The trimmed atomicity regression still rejects the old create-before-serialize behavior and passes with the fix after a clean rebuild. Fresh workspace tests, scoped formatting, strict STEVE Clippy, and independent diff-bound review passed. Cleanup evidence: /home/tuna/git/metadata-publication-fix-evidence/test-trim-93qh7kbc/.

The live results below remain bound to their original revision; live/cloud tests were not rerun for this test-only change.

Completed live dev validation

The unchanged make -j1 test-e2e-platform-ports passed: 8 steps, 0 failures, exit 0, on the normal dev host/account. Platform efc9562f9987b8fe0f282dc67818f4c61c370ec5; STEVE d0703497e5c7b04871d43fed8407d1d1a09ea892.

Normal make prepare-components built STEVE for x86_64-unknown-linux-musl, published the matching dev prebuilt set, and passed the API component preflight. The live Make invocation selected that exact set: 139a69cc01ced50ba04b70da515ab4c2676d9c3c576589d75599758b6e91e081.
All suite steps passed: gateway health; login/account setup; SSH key; E2E fixture; deployment running; AWS/host routing; direct STEVE E2P endpoint; health, attestation response, fail-closed routing and Caddy E2P parity.
The attestation assertion specifically requires HTTP 200 and a nonempty attestation-document string. This is not an independent certificate/COSE-signature validation or full successful encrypted application round trip.
The app was destroyed and test containers/database volume were removed. Independent post-run checks confirmed all five original service images running with active units, API/gateway/email HTTP 200, unchanged configuration/unit hashes, and no new live instances, VPCs, security groups or internet gateways in either checked dev region (us-west-2, ap-south-1). Prepared component artifacts are intentionally retained.
An initial attempt stopped during component preparation before suite assertions: the local validation checkout borrowed Git objects from an unmounted alternate directory. Its original services were restored. Making the owned checkout self-contained fixed the container-visible source read without changing tracked source. A retry-only external-runner procfs disappearance guard was also corrected and tested before launch (40 offline runner tests passed). No platform assertions, timeouts, startup ordering or concurrency settings were altered.

The other component pins were the existing dev selections. This is one successful live correctness run, not a production failure-rate estimate, exhaustive default-pin matrix, full E2E suite, or speed benchmark. The successful Make command took 629.433 seconds, including its setup and test teardown, but excluding separate component qualification and external host restoration; no timing comparison is claimed.

Evidence is preserved locally under /home/tuna/git/metadata-publication-fix-evidence/: component-preparation.log, qualified-components.lock.json, retry-02/live-ports/ports/{console.log,result.json}, retry-02/restoration-independent.json, and the before/after AWS inventories with retry-02/aws-resource-additions.json. Public source archives were independently downloaded and matched to every tracked committed file.

## Summary Fix the normal STEVE-enabled enclave startup race in metadata publication, independently of the platform E2E-speed changes. `write_metadata_to` previously created the final `/metadata.json` pathname before streaming JSON into it. Normal enclave startup launches STEVE and bootproofd asynchronously; bootproofd can open the existing but empty/partial file and exit on the JSON error. - Serialize into a RAII-managed temporary file **in the destination directory**; ordinary error paths clean it up without exposing partial destination bytes. - Publish the completed file with `persist_noclobber`: no partial final pathname and no replacement of an existing destination, including a concurrently created file or dangling symlink. - Preserve public metadata bytes and the original Unix `0666 & !umask` permissions; support both basename-only and nested relative paths. - Keep startup ordering, reader error policy, timeouts, test scheduling and wire format unchanged. ## Regression evidence The deterministic regression calls the actual serialization/publication helper, checks during serialization after the first JSON field, and requires the final pathname to remain absent without inspecting temporary-file internals. No timing sleeps or probability of a racing reader are needed. - **Before:** failed with `metadata destination became visible before serialization completed` (exit 101). - **After:** passes. Additional coverage checks serialization-failure cleanup, a destination created during serialization, dangling symlinks, basename-only paths, and Unix permissions. ## Validation - `cargo test --offline --locked --workspace --all-targets --all-features`: **144 passed, 0 failed, 2 ignored**. Ignored tests are not counted as executed. - Native STEVE binary + the unchanged bootproofd binary from the failing image: **30/30 offline startup-order cases passed** (reader before publication, after publication, and concurrent start). Each checks daemon health and the real `/attestation` handler's metadata-read path using an invalid nonce. **This is not NSM/cryptographic attestation.** - Full strict workspace Clippy has the same pre-existing `await_holding_lock` diagnostic at `crates/steve-sdk/src/tests.rs:966` as untouched main; no unrelated SDK cleanup is included. - Independent review: **approved, no blocking findings**, bound to the exact published diff. Scoped formatting/strict Clippy passed. Workspace formatting findings were also reproduced on unchanged SDK code. - Separate platform adoption: https://git.caution.co/public/platform/pulls/486. ## Test-only cleanup Removed 52 net lines: the redundant threaded-writers test, exact partial temporary-file byte assertions, and the supplementary nested-relative-path case. Production code is byte-identical to `d0703497e5c7b04871d43fed8407d1d1a09ea892`. The trimmed atomicity regression still rejects the old create-before-serialize behavior and passes with the fix after a clean rebuild. Fresh workspace tests, scoped formatting, strict STEVE Clippy, and independent diff-bound review passed. Cleanup evidence: `/home/tuna/git/metadata-publication-fix-evidence/test-trim-93qh7kbc/`. The live results below remain bound to their original revision; live/cloud tests were not rerun for this test-only change. ## Completed live dev validation The unchanged `make -j1 test-e2e-platform-ports` **passed: 8 steps, 0 failures, exit 0**, on the normal dev host/account. Platform `efc9562f9987b8fe0f282dc67818f4c61c370ec5`; STEVE `d0703497e5c7b04871d43fed8407d1d1a09ea892`. - Normal `make prepare-components` built STEVE for `x86_64-unknown-linux-musl`, published the matching dev prebuilt set, and passed the API component preflight. The live Make invocation selected that exact set: `139a69cc01ced50ba04b70da515ab4c2676d9c3c576589d75599758b6e91e081`. - All suite steps passed: gateway health; login/account setup; SSH key; E2E fixture; deployment running; AWS/host routing; direct STEVE E2P endpoint; health, attestation response, fail-closed routing and Caddy E2P parity. - The attestation assertion specifically requires HTTP 200 and a nonempty attestation-document string. This is **not** an independent certificate/COSE-signature validation or full successful encrypted application round trip. - The app was destroyed and test containers/database volume were removed. Independent post-run checks confirmed all five original service images running with active units, API/gateway/email HTTP 200, unchanged configuration/unit hashes, and no new live instances, VPCs, security groups or internet gateways in either checked dev region (`us-west-2`, `ap-south-1`). Prepared component artifacts are intentionally retained. - An initial attempt stopped during component preparation before suite assertions: the local validation checkout borrowed Git objects from an unmounted alternate directory. Its original services were restored. Making the owned checkout self-contained fixed the container-visible source read without changing tracked source. A retry-only external-runner procfs disappearance guard was also corrected and tested before launch (**40 offline runner tests passed**). No platform assertions, timeouts, startup ordering or concurrency settings were altered. The other component pins were the existing dev selections. This is one successful live correctness run, not a production failure-rate estimate, exhaustive default-pin matrix, full E2E suite, or speed benchmark. The successful Make command took **629.433 seconds**, including its setup and test teardown, but excluding separate component qualification and external host restoration; no timing comparison is claimed. Evidence is preserved locally under `/home/tuna/git/metadata-publication-fix-evidence/`: `component-preparation.log`, `qualified-components.lock.json`, `retry-02/live-ports/ports/{console.log,result.json}`, `retry-02/restoration-independent.json`, and the before/after AWS inventories with `retry-02/aws-resource-additions.json`. Public source archives were independently downloaded and matched to every tracked committed file.

anton added 1 commit

2026-10-19 00:27:01 +00:00