public/platform

Fork 0

Stable app ingress and node rollouts for redeploys/scaling #408

New issue

Open

opened 2026-08-05 05:06:02 +00:00 by caution-bot · 0 comments

caution-bot commented

2026-08-05 05:06:02 +00:00

(Migrated from codeberg.org)

Executive summary

Give each app a stable public ingress and perform real blue/green redeploys: boot and prove a candidate node before moving traffic, retain the active node for rollback, then drain and retire it.
Gate secrets and traffic on the intended workload, not merely valid Nitro evidence. Every rollout is bound to reviewed source/build inputs and an exact PCR0/1/2 profile, authorized before activation, and captured in a signed transition record.
Treat STEVE v2 identity policy as rollout state. During overlap it may contain only complete, pre-authorized active and candidate PCR profiles; activate and retire profiles atomically, never mix PCR slots or use TOFU as rollout authorization.

Problem

Redeploying an app can replace the underlying enclave/instance and change its public endpoint. That breaks DNS records, allowlists, webhook callbacks, and customer integrations that depend on a stable endpoint.

The product requirement is broader than preserving a single Elastic IP: Caution should model an application as having a stable public ingress and one or more backing nodes. Redeploys should add a new node behind that stable ingress, prove/prepare it, move traffic to it only after it is ready, and then drain/terminate the old node.

This also gives us the right foundation for future horizontal scaling: multiple healthy/reconstituted enclave nodes can sit behind the same application ingress instead of treating every redeploy as a single-instance replacement.

Desired behavior

Use an app-level load-balancing ingress, preferably an AWS ALB for managed AWS deployments, rather than making the public endpoint equal to a replaceable instance address.

High-level rollout model:

App has a stable ingress object and public URL.
First deploy creates the ingress and registers the initial enclave instance as an app node.
Redeploy creates a new enclave instance/node without immediately removing the old one.
The new node appears under the application in API/UI/CLI with a lifecycle status.
The new node becomes eligible for traffic only after the enclave is booted, attested, healthy, and the app secret/quorum material has been reconstituted to it.
Once the new node is active and serving, traffic shifts to it through the load balancer.
After traffic moves, the old node is drained and kept available through a bounded observation/rollback window, then spun down.
Failed redeploys leave the old active node and stable ingress in place.

Rollout authorization and verified target state

A valid Nitro attestation alone is insufficient: it proves what is running, not that the node matches the intended deployment.

Before a candidate node receives secrets or traffic:

bind the rollout to the target source commit, build inputs/manifest, configuration identity, image/EIF digest, and expected PCR0/1/2;
obtain fresh attestation from the candidate and require an exact match against the expected PCRs produced by the target build and independently reproducible with caution verify;
reject debug/all-zero PCRs, stale evidence, and valid attestations for an unintended workload;
record who or what authorized the transition from the current active identity to the target identity;
persist an append-only, signed transition record binding the complete current and target identities, authorization, activation/retirement times, and final outcome or revocation.

During blue/green overlap, ingress and STEVE v2 clients may accept only complete PCR profiles for the current active identity and explicitly authorized candidate identities. Policy activation and retirement must be atomic: never mix PCR slots across versions, use TOFU as rollout authorization, or accept an arbitrary valid enclave. Retire the old profile only after the rollback window closes.

Failed or revoked rollouts must not alter the active identity policy or release application secrets to an unapproved candidate.

Suggested node statuses:

provisioning: cloud resources are being created.
booting: host/enclave is starting.
attesting: enclave evidence/metadata is being produced or checked.
awaiting_secret: node is healthy enough to receive the app secret/quorum material but is not allowed to serve yet.
warming: secret has been reconstituted and app health checks are passing, but traffic has not fully shifted.
active: node is registered for traffic.
draining: node is no longer receiving new traffic and is waiting for shutdown.
terminated: node resources have been removed.
failed: node rollout failed; the previous active node remains serving if one existed.

Static endpoint / EIP implications

For DNS-based customers, an ALB gives the application a stable managed endpoint and supports blue/green registration/draining cleanly.

For customers that require a literal static IPv4 address or firewall allowlisting, ALB alone may not satisfy the requirement because ALBs expose DNS names rather than fixed EIPs. The implementation should explicitly choose one of these policies:

document that the stable endpoint is DNS/ALB-based for v1 and literal static IP allowlisting is deferred;
front the ALB with AWS Global Accelerator/static Anycast IPs;
use an NLB/EIP front door where L4 behavior is required;
or retain BYOC/provider-specific EIP allocation support for cases that cannot use ALB DNS.

Do not silently promise a stable IP if the actual managed endpoint is only an ALB DNS name.

TLS / enclave ingress interaction

This issue should compose with the enclave-terminated TLS work:

In STEVE mode, the host/load-balancer path may continue to route to the STEVE-compatible ingress as designed.
In enclave-terminated TLS mode, the host/load-balancer path must preserve raw TLS semantics to the enclave-side terminator. If ALB terminates TLS, that is not enclave-terminated TLS; use an L4/NLB-style path or an ALB mode only for modes where host/LB TLS termination is acceptable.
Node readiness for TLS mode should include the attested metadata/certificate-fingerprint readiness check before a node is marked active.

UI / CLI expectations

The application detail view and CLI should show node-level rollout state, for example:

App: example
Ingress: https://example.caution.app
Nodes:
  node-20260806-a  active    healthy, serving traffic
  node-20260806-b  awaiting_secret  attested, waiting for quorum reconstitution

Users should be able to tell:

which node is currently active;
which node is being prepared;
whether a rollout is blocked on health, attestation, or secret reconstitution;
whether an old node is draining;
whether the stable ingress is preserved across the rollout.

Scaling implications

The same node model can support scaling later:

desired_nodes / min-max capacity can create multiple active nodes behind the same app ingress.
Each node must independently satisfy attestation, health, and secret reconstitution before receiving traffic.
Scale-in should drain selected nodes before termination.
Secret reconstitution policy must define whether every active node receives the same app secret, a node-specific derivative, or another shard/epoch-specific credential.
Metering/billing should account for multiple active/warming nodes.

Scaling does not have to ship in the first blue/green implementation, but the data model and UI naming should avoid assuming exactly one live instance forever.

Implementation notes

There is already some EIP-related machinery in the deployment path, including eip_allocation_id Terraform output handling and associate_eip_with_instance(...) in src/api/src/deployment.rs. This issue should evolve that product/API/state behavior from “remember an instance IP” into “remember a stable app ingress and tracked backing nodes.”

Likely platform state additions:

app ingress record: provider, URL/DNS name, optional static IP allocation/global accelerator identifiers, listener/target group IDs, TLS mode constraints;
app node record: node ID, app/resource ID, instance/enclave identifiers, status, health, attestation metadata pointer, created/activated/drained timestamps;
rollout record: desired target version/config, old active nodes, new candidate nodes, failure/rollback state.
rollout verification fields: expected source commit, build/manifest digest, EIF digest, expected PCR0/1/2, authorization identity, verification result, and policy activation/retirement timestamps.

Acceptance criteria

App redeploy does not require changing the public endpoint when the app uses the managed stable-ingress mode.
A redeploy creates a new node and keeps the old active node serving until the new node is ready.
The new node is not marked active until health, attestation, and secret reconstitution requirements pass.
Failed redeploys keep the old active node and stable ingress intact.
Old nodes are drained before termination after a successful rollout.
API, UI, and CLI expose node status under an app.
The design explicitly distinguishes ALB/DNS stability from literal static IPv4/EIP requirements.
Provider-specific EIP/static-IP cleanup semantics are documented so app deletion does not create dangling domains or silently release addresses users still need.
The design is compatible with future horizontal scaling of multiple active nodes.
Candidate readiness requires fresh attestation matching the rollout's source/build-linked expected PCR policy; valid attestation alone is insufficient.
During overlap, only the old active identity and explicitly authorized candidate identities are accepted; the old identity is retired after drain.
Failed or revoked rollouts do not change the active identity policy or release secrets to the candidate.
Tests reject mismatched PCRs, stale evidence, debug/all-zero PCRs, and valid evidence for an unintended workload.
Tests cover rollout state transitions, failed-rollout rollback, stable ingress preservation, and node visibility in API/CLI/UI at the appropriate layers.

Non-goals / follow-ups

Shipping full horizontal autoscaling in the initial blue/green implementation.
Requiring every deployment mode/provider to support ALB in v1.
Pretending ALB DNS is a literal static IP for customers that require IP allowlisting.
Weakening enclave-terminated TLS by terminating TLS at the ALB in modes where the enclave must own the TLS private key.

## Executive summary - Give each app a stable public ingress and perform real blue/green redeploys: boot and prove a candidate node before moving traffic, retain the active node for rollback, then drain and retire it. - Gate secrets and traffic on the intended workload, not merely valid Nitro evidence. Every rollout is bound to reviewed source/build inputs and an exact PCR0/1/2 profile, authorized before activation, and captured in a signed transition record. - Treat STEVE v2 identity policy as rollout state. During overlap it may contain only complete, pre-authorized `active` and `candidate` PCR profiles; activate and retire profiles atomically, never mix PCR slots or use TOFU as rollout authorization. ## Problem Redeploying an app can replace the underlying enclave/instance and change its public endpoint. That breaks DNS records, allowlists, webhook callbacks, and customer integrations that depend on a stable endpoint. The product requirement is broader than preserving a single Elastic IP: Caution should model an application as having a stable public ingress and one or more backing **nodes**. Redeploys should add a new node behind that stable ingress, prove/prepare it, move traffic to it only after it is ready, and then drain/terminate the old node. This also gives us the right foundation for future horizontal scaling: multiple healthy/reconstituted enclave nodes can sit behind the same application ingress instead of treating every redeploy as a single-instance replacement. ## Desired behavior Use an app-level load-balancing ingress, preferably an AWS ALB for managed AWS deployments, rather than making the public endpoint equal to a replaceable instance address. High-level rollout model: 1. App has a stable ingress object and public URL. 2. First deploy creates the ingress and registers the initial enclave instance as an app **node**. 3. Redeploy creates a new enclave instance/node without immediately removing the old one. 4. The new node appears under the application in API/UI/CLI with a lifecycle status. 5. The new node becomes eligible for traffic only after the enclave is booted, attested, healthy, and the app secret/quorum material has been reconstituted to it. 6. Once the new node is active and serving, traffic shifts to it through the load balancer. 7. After traffic moves, the old node is drained and kept available through a bounded observation/rollback window, then spun down. 8. Failed redeploys leave the old active node and stable ingress in place. ## Rollout authorization and verified target state A valid Nitro attestation alone is insufficient: it proves what is running, not that the node matches the intended deployment. Before a candidate node receives secrets or traffic: - bind the rollout to the target source commit, build inputs/manifest, configuration identity, image/EIF digest, and expected PCR0/1/2; - obtain fresh attestation from the candidate and require an exact match against the expected PCRs produced by the target build and independently reproducible with `caution verify`; - reject debug/all-zero PCRs, stale evidence, and valid attestations for an unintended workload; - record who or what authorized the transition from the current active identity to the target identity; - persist an append-only, signed transition record binding the complete current and target identities, authorization, activation/retirement times, and final outcome or revocation. During blue/green overlap, ingress and STEVE v2 clients may accept only complete PCR profiles for the current active identity and explicitly authorized candidate identities. Policy activation and retirement must be atomic: never mix PCR slots across versions, use TOFU as rollout authorization, or accept an arbitrary valid enclave. Retire the old profile only after the rollback window closes. Failed or revoked rollouts must not alter the active identity policy or release application secrets to an unapproved candidate. Suggested node statuses: - `provisioning`: cloud resources are being created. - `booting`: host/enclave is starting. - `attesting`: enclave evidence/metadata is being produced or checked. - `awaiting_secret`: node is healthy enough to receive the app secret/quorum material but is not allowed to serve yet. - `warming`: secret has been reconstituted and app health checks are passing, but traffic has not fully shifted. - `active`: node is registered for traffic. - `draining`: node is no longer receiving new traffic and is waiting for shutdown. - `terminated`: node resources have been removed. - `failed`: node rollout failed; the previous active node remains serving if one existed. ## Static endpoint / EIP implications For DNS-based customers, an ALB gives the application a stable managed endpoint and supports blue/green registration/draining cleanly. For customers that require a literal static IPv4 address or firewall allowlisting, ALB alone may not satisfy the requirement because ALBs expose DNS names rather than fixed EIPs. The implementation should explicitly choose one of these policies: - document that the stable endpoint is DNS/ALB-based for v1 and literal static IP allowlisting is deferred; - front the ALB with AWS Global Accelerator/static Anycast IPs; - use an NLB/EIP front door where L4 behavior is required; - or retain BYOC/provider-specific EIP allocation support for cases that cannot use ALB DNS. Do not silently promise a stable IP if the actual managed endpoint is only an ALB DNS name. ## TLS / enclave ingress interaction This issue should compose with the enclave-terminated TLS work: - In STEVE mode, the host/load-balancer path may continue to route to the STEVE-compatible ingress as designed. - In enclave-terminated TLS mode, the host/load-balancer path must preserve raw TLS semantics to the enclave-side terminator. If ALB terminates TLS, that is not enclave-terminated TLS; use an L4/NLB-style path or an ALB mode only for modes where host/LB TLS termination is acceptable. - Node readiness for TLS mode should include the attested metadata/certificate-fingerprint readiness check before a node is marked `active`. ## UI / CLI expectations The application detail view and CLI should show node-level rollout state, for example: ```text App: example Ingress: https://example.caution.app Nodes: node-20260806-a active healthy, serving traffic node-20260806-b awaiting_secret attested, waiting for quorum reconstitution ``` Users should be able to tell: - which node is currently active; - which node is being prepared; - whether a rollout is blocked on health, attestation, or secret reconstitution; - whether an old node is draining; - whether the stable ingress is preserved across the rollout. ## Scaling implications The same node model can support scaling later: - `desired_nodes` / min-max capacity can create multiple active nodes behind the same app ingress. - Each node must independently satisfy attestation, health, and secret reconstitution before receiving traffic. - Scale-in should drain selected nodes before termination. - Secret reconstitution policy must define whether every active node receives the same app secret, a node-specific derivative, or another shard/epoch-specific credential. - Metering/billing should account for multiple active/warming nodes. Scaling does not have to ship in the first blue/green implementation, but the data model and UI naming should avoid assuming exactly one live instance forever. ## Implementation notes There is already some EIP-related machinery in the deployment path, including `eip_allocation_id` Terraform output handling and `associate_eip_with_instance(...)` in `src/api/src/deployment.rs`. This issue should evolve that product/API/state behavior from “remember an instance IP” into “remember a stable app ingress and tracked backing nodes.” Likely platform state additions: - app ingress record: provider, URL/DNS name, optional static IP allocation/global accelerator identifiers, listener/target group IDs, TLS mode constraints; - app node record: node ID, app/resource ID, instance/enclave identifiers, status, health, attestation metadata pointer, created/activated/drained timestamps; - rollout record: desired target version/config, old active nodes, new candidate nodes, failure/rollback state. - rollout verification fields: expected source commit, build/manifest digest, EIF digest, expected PCR0/1/2, authorization identity, verification result, and policy activation/retirement timestamps. ## Acceptance criteria - App redeploy does not require changing the public endpoint when the app uses the managed stable-ingress mode. - A redeploy creates a new node and keeps the old active node serving until the new node is ready. - The new node is not marked `active` until health, attestation, and secret reconstitution requirements pass. - Failed redeploys keep the old active node and stable ingress intact. - Old nodes are drained before termination after a successful rollout. - API, UI, and CLI expose node status under an app. - The design explicitly distinguishes ALB/DNS stability from literal static IPv4/EIP requirements. - Provider-specific EIP/static-IP cleanup semantics are documented so app deletion does not create dangling domains or silently release addresses users still need. - The design is compatible with future horizontal scaling of multiple active nodes. - Candidate readiness requires fresh attestation matching the rollout's source/build-linked expected PCR policy; valid attestation alone is insufficient. - During overlap, only the old active identity and explicitly authorized candidate identities are accepted; the old identity is retired after drain. - Failed or revoked rollouts do not change the active identity policy or release secrets to the candidate. - Tests reject mismatched PCRs, stale evidence, debug/all-zero PCRs, and valid evidence for an unintended workload. - Tests cover rollout state transitions, failed-rollout rollback, stable ingress preservation, and node visibility in API/CLI/UI at the appropriate layers. ## Non-goals / follow-ups - Shipping full horizontal autoscaling in the initial blue/green implementation. - Requiring every deployment mode/provider to support ALB in v1. - Pretending ALB DNS is a literal static IP for customers that require IP allowlisting. - Weakening enclave-terminated TLS by terminating TLS at the ALB in modes where the enclave must own the TLS private key.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

public/platform#408

No description provided.

Rows
Columns