[WIP] [EXPERIMENTAL] builder: add opt-in S3 BuildKit layer cache for EIF builds #353

Closed

vkobel wants to merge 3 commits from fix/eif-s3-layer-cache into main

vkobel commented

2026-07-02 10:47:14 +00:00

(Migrated from codeberg.org)

LLM Experiment to see if this is an actual way to speed things up a bit

Should test in dev.

Adds an opt-in S3 BuildKit layer cache on remote builders, cutting the "Docker image built → EIF built" phase (~2m of a plain-app deploy) by reusing cached stages across builds — including the steve/locksmith/bootproof cargo builds when their pinned commits are unchanged.

Scoped & safe: cache runs only on the remote builder via a dedicated eifcache docker-container builder targeted with --builder (not --use), so the customer's docker build stays on the default driver — IMDS-isolated and loaded into the image store.
Zero impact on caution verify / local builds: they never set the cache env vars, so they take the byte-identical plain docker build path. Cache changes build speed, never output bytes → PCRs unaffected.
Bypass switch: build { layer_cache = false } in HCL forces a fully self-contained uncached build (escape hatch for a poisoned cache).

Requires: deploy updated api (new user-data) + apply the infra-bootstrap IAM change (buildcache/* S3 perms). Without the IAM grant, cache writes fail silently (ignore-error=true) and builds still succeed uncached.

Relates to #250 (Prebuild EnclaveOS components) — lighter-weight approach, no ECR image dependency.

Verified by compile + code review only; needs a real deploy trace to confirm the phase-time drop.

LLM Experiment to see if this is an actual way to speed things up a bit Should test in dev. --- Adds an opt-in S3 BuildKit layer cache on remote builders, cutting the "Docker image built → EIF built" phase (~2m of a plain-app deploy) by reusing cached stages across builds — including the steve/locksmith/bootproof cargo builds when their pinned commits are unchanged. - Scoped & safe: cache runs only on the remote builder via a dedicated eifcache docker-container builder targeted with --builder (not --use), so the customer's docker build stays on the default driver — IMDS-isolated and loaded into the image store. - Zero impact on caution verify / local builds: they never set the cache env vars, so they take the byte-identical plain docker build path. Cache changes build speed, never output bytes → PCRs unaffected. - Bypass switch: build { layer_cache = false } in HCL forces a fully self-contained uncached build (escape hatch for a poisoned cache). Requires: deploy updated api (new user-data) + apply the infra-bootstrap IAM change (buildcache/* S3 perms). Without the IAM grant, cache writes fail silently (ignore-error=true) and builds still succeed uncached. Relates to #250 (Prebuild EnclaveOS components) — lighter-weight approach, no ECR image dependency. Verified by compile + code review only; needs a real deploy trace to confirm the phase-time drop.

vkobel commented

2026-07-06 19:51:22 +00:00

(Migrated from codeberg.org)

Should we keep this as a temporary bandaid?