public/platform

Fork 0

perf(tests): default live E2E to six parallel suites #487

Manually merged

anton merged 1 commit from test/live-e2e-workers-3-4 into main

2026-10-19 06:09:43 +00:00

anton commented

2026-10-19 05:22:31 +00:00

Owner

Summary

Speed up the existing full make test-e2e lifecycle without changing live assertions or adding a second runner.

Run the six independent whole-suite targets against one shared test stack, defaulting to E2E_JOBS=6. Supported values are 1–6; E2E_JOBS=1 remains the serial diagnostic fallback. Setup remains serial; only the suite sub-Make is parallel.
Give each suite a private CLI configuration. Preserve one setup/teardown, drain active suites and their EXIT cleanup before teardown, stop queued work after failure, and ignore inherited -k/-j values for the suite scheduler.
Exclude generated E2E directories from Docker context and replace fixed startup sleeps with bounded readiness probes. Keep genuine live assertions unchanged.
Remove one-off local benchmark scaffolding from the earlier offline-speed work; retain the helper-import regression in tests/.

Merge platform #486 first to adopt the separately fixed STEVE metadata-publication race. This PR contains no runtime workaround or STEVE pin change.

Fresh local validation

make -j1 test-e2e-contracts: 68 passed, including 27 scheduling tests.
python3 -B tests/test_e2e_startup.py -v: 24 passed.
Default-six regression failed against the old default two, then passed after the change. A five-worker case failed the intermediate 1,2,3,4,6 allowlist before completing support for all widths 1–6.
Failure/interrupt tests exercise six active suites, held cleanup and teardown ordering; lower-width cases retain queued-work cancellation, Make flag isolation and explicit caps.
A broader check exposed a test-observer race: simultaneous exclusive lock probes could falsely report lifecycle ownership. The fixture now probes with a shared lock while checking the real exclusive owner. A deterministic held-reader regression failed before the fix; 20 repeats of both shared-reader and exclusive-owner tests passed afterward. This does not change production lock ownership.
git diff --check: passed. No tests skipped or assertions relaxed to hide a runtime failure.

Prior live evidence — not a new run of this head

The frozen six-worker experiment completed 6/6 live suites, zero failures, on the normal dev host/account, using platform 518585fccd2ad93205b80b6fd2ad510df87c0b57, STEVE da6d1ca85ddf15fec9864c9a588a046d77c32a85, and a run-owned Makefile width-guard delta. Original services/configuration were independently verified restored and no new live AWS resources remained in the checked regions.

The new landed STEVE pin in #486 is 694b5afbd6cf6f9ec743e27266097921d4102b09: all 92 tracked files in its canonical commit-addressed archive match da6d1ca exactly. Source identity does not replace normal component qualification for the new commit pair.

That experiment's full Make wall time was 14m01s; the preceding four-worker run was 18m06s. Most of that difference was setup time; suite-plus-teardown was slightly slower at six. These are correctness observations with different preparation durations, not a matched causal claim that six outperforms four. Six is the explicitly requested default.

This PR refresh was verified offline only: no fresh full live suite, deployment, service restart or component activation. No benchmark output or temporary runner is added to the repository.

Local evidence: /home/tuna/git/platform-pin-six-workers-ycluxg8y/VALIDATION.md; frozen live evidence: /home/tuna/git/metadata-publication-six-e2e-ez4fo3_i/RESULTS.md.

## Summary Speed up the existing full `make test-e2e` lifecycle without changing live assertions or adding a second runner. - Run the six independent whole-suite targets against one shared test stack, **defaulting to `E2E_JOBS=6`**. Supported values are 1–6; `E2E_JOBS=1` remains the serial diagnostic fallback. Setup remains serial; only the suite sub-Make is parallel. - Give each suite a private CLI configuration. Preserve one setup/teardown, drain active suites and their EXIT cleanup before teardown, stop queued work after failure, and ignore inherited `-k`/`-j` values for the suite scheduler. - Exclude generated E2E directories from Docker context and replace fixed startup sleeps with bounded readiness probes. Keep genuine live assertions unchanged. - Remove one-off local benchmark scaffolding from the earlier offline-speed work; retain the helper-import regression in `tests/`. **Merge platform [#486](https://git.caution.co/public/platform/pulls/486) first** to adopt the separately fixed STEVE metadata-publication race. This PR contains no runtime workaround or STEVE pin change. ## Fresh local validation - `make -j1 test-e2e-contracts`: **68 passed**, including **27 scheduling tests**. - `python3 -B tests/test_e2e_startup.py -v`: **24 passed**. - Default-six regression failed against the old default two, then passed after the change. A five-worker case failed the intermediate 1,2,3,4,6 allowlist before completing support for all widths 1–6. - Failure/interrupt tests exercise six active suites, held cleanup and teardown ordering; lower-width cases retain queued-work cancellation, Make flag isolation and explicit caps. - A broader check exposed a test-observer race: simultaneous exclusive lock probes could falsely report lifecycle ownership. The fixture now probes with a shared lock while checking the real exclusive owner. A deterministic held-reader regression failed before the fix; **20 repeats of both shared-reader and exclusive-owner tests passed** afterward. This does not change production lock ownership. - `git diff --check`: passed. No tests skipped or assertions relaxed to hide a runtime failure. ## Prior live evidence — not a new run of this head The frozen six-worker experiment completed **6/6 live suites, zero failures**, on the normal dev host/account, using platform `518585fccd2ad93205b80b6fd2ad510df87c0b57`, STEVE `da6d1ca85ddf15fec9864c9a588a046d77c32a85`, and a run-owned Makefile width-guard delta. Original services/configuration were independently verified restored and no new live AWS resources remained in the checked regions. The new landed STEVE pin in #486 is `694b5afbd6cf6f9ec743e27266097921d4102b09`: all 92 tracked files in its canonical commit-addressed archive match `da6d1ca` exactly. Source identity does not replace normal component qualification for the new commit pair. That experiment's full Make wall time was **14m01s**; the preceding four-worker run was **18m06s**. Most of that difference was setup time; suite-plus-teardown was slightly slower at six. These are correctness observations with different preparation durations, **not a matched causal claim that six outperforms four**. Six is the explicitly requested default. This PR refresh was verified offline only: no fresh full live suite, deployment, service restart or component activation. No benchmark output or temporary runner is added to the repository. Local evidence: `/home/tuna/git/platform-pin-six-workers-ycluxg8y/VALIDATION.md`; frozen live evidence: `/home/tuna/git/metadata-publication-six-e2e-ez4fo3_i/RESULTS.md`.

anton added 5 commits

2026-10-19 05:22:31 +00:00

perf(tests): avoid generated build context and bound startup readiness 24fc631a25

test: remove one-off local benchmark scaffolding 14f4e31cf0

Keep the lightweight ports helper and its startup regression in the normal test tree and run that regression from test-e2e-contracts. Remove experimental runner-only tests and machine-specific benchmark artifacts. Independent cleanup delta review passed; no new live speed claim.

perf(tests): [verified] run live E2E suites with two workers c77c456c00

Use bounded GNU Make suite jobs with E2E_JOBS=1 fallback, isolated CLI configurations, and distinct log prefixes. Preserve all six original live suites and assertion/cleanup bodies.

Avoid re-locking the external observer and arm teardown only after successful CLI compilation. Add actual GNU Make orchestration, failure, lock-ownership, and fixture-cleanup regressions.

Verification: 56 offline contracts and 24 startup tests pass; independent final review passed and reran all 15 scheduling tests. This is not a measured live performance improvement. Local-only; no source publication or cloud run.

test(e2e): [verified] allow opt-in three and four workers 518585fccd

test(e2e): default live suites to six workers d9a99e00be

anton referenced this pull request

2026-10-19 06:01:06 +00:00

fix: adopt atomic STEVE metadata publication #486

anton commented

2026-10-19 06:01:08 +00:00

Author

Owner

Six-worker live E2E recheck — PASS

One complete batch ran on the existing dev host/account, with no overlapping live experiment and no follow-on batch. Both PRs remain unmerged.

Exact inputs and scope

Platform: d9a99e00be0a42f49c2c711615cb90817fe2b7fb (published PR #487).
STEVE: 694b5afbd6cf6f9ec743e27266097921d4102b09 (landed revision, selected through the supported Make override).
Normal source Makefile, make -j1 test-e2e E2E_JOBS=6, including build, component preparation, all suites and test teardown. No experimental Makefile and no separate prebuild.
This qualifies that exact platform/STEVE pair. It is not execution of PR #486's framework head or a hypothetical merged tree.
Exact public platform and STEVE archives matched their Git trees. Forty offline runner checks and 27 scheduler checks passed before launch; these are separate from the live results below.

Live result

Full Make: 1151.300 seconds (19m11.3s), exit 0.
All six suites started within 0.004 seconds of one another.
6/6 suites passed, 56 reported checks passed, 0 failed.

Suite	Passed	Failed
Environment parity	9	0
Happy path	10	0
PGP units	11	0
PGP audit constraints	9	0
SSH units	9	0
Platform ports	8	0

The previous six-worker batch took 841.090 seconds (14m01.1s). This recheck took 310.210 seconds longer. Build/setup before suite launch grew from 51.618 to 372.368 seconds; suites plus teardown took 778.932 seconds. Different framework/component revisions and retained build caches make this a correctness recheck, not a controlled performance comparison. Full Make timing excludes preflight and external original-service restoration.

Component verification

Normal Make component preparation and startup preflight passed. Post-run, read-only verification confirmed the selected canonical S3 manifest matched its accepted local lock and all five source pins. The downloaded STEVE ELF matched its declared size and digest:

Component set: 03327d5f458efc2e045d46c09cc177bcec42fc3c71ec4e4af171c047dacbc93f.
STEVE SHA-256: 4f6aa867e3a7068904acc0b1a86a8266f4d854e82cbf7f5f1b8595c84919f8bf.
STEVE bytes: 6458584.

Cleanup and restoration

Launcher, independent restoration check, and AWS inventory check: all exit 0.
Original postgres, API, gateway, email and metering services active, with original images.
Direct-IP API/gateway/email health: HTTP 200.
Original PostgreSQL persistent mounts retained; config and unit files unchanged.
No test containers or test volume remain; Make process group is quiescent.
Before/after nonterminated EC2 instances, VPCs, security groups and internet gateways have identical IDs in us-west-2 and ap-south-1. This inventory is not a claim about every AWS resource class. Immutable component objects remain intentionally retained.
Source, runner, frozen run inputs and predecessor evidence hashes unchanged.

Evidence

FINAL-VERIFICATION.json, component-provenance.json, live-ports/ports/result.json, live-ports/result.json, live-ports/restored.json, restoration-independent.json, six source/tests/e2e/logs/*.log files, AWS before/after inventories, and completion-sha256.json.

The historical live-ports/ports folder name is retained by the reused runner; its receipt records the full six-suite Make command, not a standalone platform-ports run.

Evidence SHA-256 receipts:

FINAL-VERIFICATION.json: 355646dbb62564098d5dbe84af2d3f471a347c5fc132c10db880120dd647444a
component-provenance.json: 25cf0e66c458e66e97435c8348b7da8733d274ca4f83924fb8f2ae12a46e3ccd
RESULTS.md: c362f4a1406106b2f98afd44806a7c2680c5de581403c935d48ebb750f3020a0

# Six-worker live E2E recheck — PASS One complete batch ran on the existing dev host/account, with no overlapping live experiment and no follow-on batch. Both PRs remain unmerged. ## Exact inputs and scope - Platform: `d9a99e00be0a42f49c2c711615cb90817fe2b7fb` (published PR #487). - STEVE: `694b5afbd6cf6f9ec743e27266097921d4102b09` (landed revision, selected through the supported Make override). - Normal source Makefile, `make -j1 test-e2e E2E_JOBS=6`, including build, component preparation, all suites and test teardown. No experimental Makefile and no separate prebuild. - This qualifies that exact platform/STEVE pair. It is not execution of PR #486's framework head or a hypothetical merged tree. - Exact public platform and STEVE archives matched their Git trees. Forty offline runner checks and 27 scheduler checks passed before launch; these are separate from the live results below. ## Live result - Full Make: **1151.300 seconds (19m11.3s), exit 0**. - All six suites started within 0.004 seconds of one another. - **6/6 suites passed, 56 reported checks passed, 0 failed.** | Suite | Passed | Failed | | --- | ---: | ---: | | Environment parity | 9 | 0 | | Happy path | 10 | 0 | | PGP units | 11 | 0 | | PGP audit constraints | 9 | 0 | | SSH units | 9 | 0 | | Platform ports | 8 | 0 | The previous six-worker batch took 841.090 seconds (14m01.1s). This recheck took 310.210 seconds longer. Build/setup before suite launch grew from 51.618 to 372.368 seconds; suites plus teardown took 778.932 seconds. Different framework/component revisions and retained build caches make this a correctness recheck, **not a controlled performance comparison**. Full Make timing excludes preflight and external original-service restoration. ## Component verification Normal Make component preparation and startup preflight passed. Post-run, read-only verification confirmed the selected canonical S3 manifest matched its accepted local lock and all five source pins. The downloaded STEVE ELF matched its declared size and digest: - Component set: `03327d5f458efc2e045d46c09cc177bcec42fc3c71ec4e4af171c047dacbc93f`. - STEVE SHA-256: `4f6aa867e3a7068904acc0b1a86a8266f4d854e82cbf7f5f1b8595c84919f8bf`. - STEVE bytes: `6458584`. ## Cleanup and restoration - Launcher, independent restoration check, and AWS inventory check: all exit 0. - Original postgres, API, gateway, email and metering services active, with original images. - Direct-IP API/gateway/email health: HTTP 200. - Original PostgreSQL persistent mounts retained; config and unit files unchanged. - No test containers or test volume remain; Make process group is quiescent. - Before/after nonterminated EC2 instances, VPCs, security groups and internet gateways have identical IDs in `us-west-2` and `ap-south-1`. This inventory is not a claim about every AWS resource class. Immutable component objects remain intentionally retained. - Source, runner, frozen run inputs and predecessor evidence hashes unchanged. ## Evidence `FINAL-VERIFICATION.json`, `component-provenance.json`, `live-ports/ports/result.json`, `live-ports/result.json`, `live-ports/restored.json`, `restoration-independent.json`, six `source/tests/e2e/logs/*.log` files, AWS before/after inventories, and `completion-sha256.json`. The historical `live-ports/ports` folder name is retained by the reused runner; its receipt records the full six-suite Make command, not a standalone platform-ports run. Evidence SHA-256 receipts: - `FINAL-VERIFICATION.json`: `355646dbb62564098d5dbe84af2d3f471a347c5fc132c10db880120dd647444a` - `component-provenance.json`: `25cf0e66c458e66e97435c8348b7da8733d274ca4f83924fb8f2ae12a46e3ccd` - `RESULTS.md`: `c362f4a1406106b2f98afd44806a7c2680c5de581403c935d48ebb750f3020a0`

anton force-pushed test/live-e2e-workers-3-4 from d9a99e00be to 2cb44ae846

2026-10-19 06:09:01 +00:00

Compare

anton manually merged commit 48d82351d0 into main

2026-10-19 06:09:43 +00:00

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

public/platform!487

No description provided.

Rows
Columns