Deployment progress timings underreport elapsed time during EIF build #458

New issue

Open

opened 2026-09-17 07:52:54 +00:00 by vkobel · 1 comment

vkobel commented

2026-09-17 07:52:54 +00:00

(Migrated from codeberg.org)

On dev, git push caution HEAD:main reported:

Building application Docker image... Complete! (2m2s)
Docker image built, building EIF... Complete! (3m24s)
EIF built, uploading to S3... Complete! (10.2s)

The EIF build step appeared to take 9–10 minutes, despite reporting 3m24s. Progress messages or duration accounting may be delayed or attributed to the wrong stage; the cause is unconfirmed.

Expected: Stage messages reflect the current deployment state, and reported durations match elapsed time.

Investigation: Compare API status-transition logs with timestamped client output (e.g. asciinema), checking for missing transitions, polling delays, or incorrect duration boundaries. Preserve the current dev instance and logs before restarting.

On dev, `git push caution HEAD:main` reported: ```text Building application Docker image... Complete! (2m2s) Docker image built, building EIF... Complete! (3m24s) EIF built, uploading to S3... Complete! (10.2s) ``` The EIF build step appeared to take **9–10 minutes**, despite reporting **3m24s**. Progress messages or duration accounting may be delayed or attributed to the wrong stage; the cause is unconfirmed. **Expected:** Stage messages reflect the current deployment state, and reported durations match elapsed time. **Investigation:** Compare API status-transition logs with timestamped client output (e.g. asciinema), checking for missing transitions, polling delays, or incorrect duration boundaries. Preserve the current dev instance and logs before restarting.

vkobel commented

2026-09-23 14:52:04 +00:00

(Migrated from codeberg.org)

The bug is how time gets assigned to stages:

The API polls S3 every 10 seconds. When it misses intermediate stages, it emits all their messages immediately. src/api/src/builder.rs (line 1343)
The gateway measures time between receiving messages, rather than actual builder phase durations. src/gateway/src/ssh_server.rs (line 1268)
In this run, polling missed both launching-builder and uploading-enclave. Consequently, booting shows 0.0s with its time absorbed into provisioning; upload shows 0.0s with its time absorbed into EIF building.

The bug is how time gets assigned to stages: - The API polls S3 every 10 seconds. When it misses intermediate stages, it emits all their messages immediately. `src/api/src/builder.rs` (line 1343) - The gateway measures time between receiving messages, rather than actual builder phase durations. `src/gateway/src/ssh_server.rs` (line 1268) - In this run, polling missed both launching-builder and uploading-enclave. Consequently, booting shows 0.0s with its time absorbed into provisioning; upload shows 0.0s with its time absorbed into EIF building.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

public/platform#458

No description provided.

Rows
Columns