open-platform/mitosis
3
0
Fork 0
Code Issues 27 Pull requests 9 Releases 4 Packages 1 Wiki Activity Actions

ci: parallel per-ref gates + nightly app-E2E variant (#67 second half, #21) #75

Merged
plat merged 4 commits from feat/gate-parallel-nightly into main 2026-07-02 17:10:47 +00:00
Conversation 0 Commits 4 Files changed 4 +417 -37
plat commented 2026-07-02 16:50:37 +00:00
Owner
Copy link

What

Second half of #67 + the CI-seam follow-ups from #21: the cold-start gate now runs in parallel across refs and grows a nightly app-E2E variant that ships a real canary app unattended — default-SKIP until an operator seals a credential. Originally based on fix/gate-a4-detection (#74); rebased onto main after #74 merged.

1. Parallel gates

  • Runner capacity on vxrail raised 1 → 2 (~/act_runner/config.yml, user unit restarted, re-registered cleanly, stayed registered and serving).
  • Workflow concurrency is now per-ref (coldstart-${{ github.ref }}, cancel-in-progress: true): different PRs gate in parallel; a push to the same ref supersedes its in-flight run. Total load is capped by runner capacity, not the group.
  • Orphan sweep at run start: a superseded run can be cancelled before its teardown, so each run removes ^k3d-ci[0-9]+- containers older than 150 min (a live run fits the 110-min job cap). Match is anchored — prod (k3d-platgold2) and live dev clusters are never touched.
  • FJ_PORT in bin/germinate (one surgical knob, default :3000 unchanged): the first parallel attempt killed run 11 with admin auth failing after handoff — germinate's forgejo port-forward binds a FIXED 127.0.0.1:3000, so the second germination's curls silently reached the FIRST cluster's forgejo (different admin password). The workflow now exports a per-run port (30000 + run_number % 10000). CAVEAT: PR branches forked before this fix still bind :3000 — at most one such gate at a time until rebased.

2. Nightly app-E2E (#67 second half)

On the nightly cron or an e2e=true dispatch, after the standard gate, scripts/coldstart-assert.sh --e2e runs against the same CI daughter:

  • E1 claude credential sealed in plat-agents-secrets (fail fast)
  • E2 canary repo generated from _app-template (Forgejo API, 201)
  • E3 dispatcher bootstraps it: prime PR opens
  • E4 prime PR auto-merges (builder → preview → reviewer verdict → merge)
  • E5 prod serves 200 in-cluster (Host <app>-plat.ci<N>.local via the traefik svc)
  • E6 dispatcher log contains no preview liveness unconfirmed (#57 fallback)
  • E7 builder-finish → reviewer-queued gap < 120 s
  • E8 app prod namespace carries plat.sh/app + ResourceQuota + NetworkPolicies

Credential: the germination step reads repo Actions secret E2E_CLAUDE_TOKEN and seals it as CLAUDE_CODE_OAUTH_TOKEN (values never echoed). The secret is NOT created by this PR (the staged host token is compromised-pending-rotation and was deliberately not wired in). When absent, e2e-flagged runs print one loud E2E SKIP line (host-persisted) and exit 0 — nightly stays green-by-default. Sealing recipe: docs/ops/coldstart-ci.md + comment on #67.

Phase is hard-capped (timeout 35m step; E2E_TIMEOUT knob, default 1800 s). Humans run the identical thing: scripts/coldstart-assert.sh --e2e <ctx> <domain>.

3. Gate-probe hardening (found live during validation)

A4's redirect probe used kubectl run -i --rm, whose attach can MISS output printed before it connects — observed live as a no-XFP=? false-FAIL under 3-way host load. A4 and the new E5 probe now share pod_probe(): run the pod detached, wait for a terminal phase, read pod logs (durable), delete. Re-ran the full standard gate on the same platform after the fix: PASS.

Evidence

Parallel proof (runs 12 + 13, dispatched simultaneously)

# 16:07:15 UTC, ~25s after the two dispatches — both clusters germinating side by side:
k3d-ci12-serverlb    24 seconds ago
k3d-ci12-server-0    24 seconds ago
k3d-ci13-serverlb    25 seconds ago
k3d-ci13-server-0    25 seconds ago

# both green, distinct clusters, overlapping ~/ci-gate-logs timestamps:
run 12 (feat/gate-parallel-nightly): success — "COLDSTART GATE PASS (93s) ... context=k3d-ci12" (run12-gate.log mtime 16:25:24)
run 13 (main):                       success — "COLDSTART GATE PASS (94s) ... context=k3d-ci13" (run13-gate.log mtime 16:24:37)

# zero leaks after both teardowns:
docker ps -a --format '{{.Names}}' | grep -E 'k3d-ci1[23]-'   -> (empty)

(First attempt, runs 10+11: run 10 green, run 11 died on the :3000 cross-talk described above — that failure is what the FJ_PORT commit fixes; runs 12+13 are the post-fix proof.)

E2E default-SKIP proof (runs 10 + 12, e2e=true, no secret — both runs green)

$ cat ~/ci-gate-logs/run12-e2e.log
#### E2E SKIP: repo Actions secret E2E_CLAUDE_TOKEN is not sealed — app-E2E did NOT run (the standard gate above is this run's result); seal it per docs/ops/coldstart-ci.md (Nightly app E2E) to enable ####

Manual validation of the full --e2e path (marked manual-validation)

Scratch germination ci9999 on the vxrail host (PORTS=none, isolated KUBECONFIG, FJ_PORT=31999), credential taken from the host shell for this one run and never persisted into Forgejo/CI. All eight assertions passed; the canary shipped unattended in 793 s:

== coldstart E2E gate: context=k3d-ci9999 domain=ci9999.local app=plat/canary07021611 budget=1800s ==
PASS [E1-credential] (   1s) claude credential sealed in plat-agents-secrets (ns agents; value never printed)
PASS [E2-canary-created] (   3s) POST /repos/plat/_app-template/generate -> 201 (plat/canary07021611)
PASS [E3-prime-pr] (  96s) dispatcher bootstrapped plat/canary07021611: prime PR #1 open
PASS [E4-auto-merge] ( 610s) prime PR #1 merged ('v1 auto-merged after validation' log lines: 1)
PASS [E5-prod-200] ( 793s) https://canary07021611-plat.ci9999.local/ -> 200 from an in-cluster pod
PASS [E6-no-liveness-fallback] ( 793s) zero 'preview liveness unconfirmed' lines in the dispatcher log
PASS [E7-review-latency] ( 793s) builder-finish 2026-07-02T16:19:23.933Z -> reviewer-queued 2026-07-02T16:19:57.238Z: 34s (< 120s)
PASS [E8-ns-governance] ( 793s) ns plat--canary07021611--prod: plat.sh/app=canary07021611, 2 ResourceQuota, 6 NetworkPolicies (allow-egress allow-egress-postgres allow-ingress-from-workflows allow-ingress-postgres allow-ingress-traefik default-deny)
COLDSTART E2E GATE PASS ( 793s): app plat/canary07021611 shipped builder->reviewer->prod unattended

# dispatcher log lines behind E6/E7:
2026-07-02T16:19:23.933Z info run finished runId=builder-plat-canary07021611-... role=builder exitCode=0 durationMs=364063
2026-07-02T16:19:57.238Z info run queued  runId=reviewer-plat-canary07021611-... role=reviewer item=#1        (gap: 34s)
2026-07-02T16:21:52.412Z info v1 auto-merged after validation pr=plat/canary07021611#1

Files

  • .forgejo/workflows/coldstart.yml — per-ref concurrency, orphan sweep, per-run FJ_PORT, e2e input + phase, step-level timeouts (40m germinate / 20m gate / 35m e2e; job cap 110m)
  • scripts/coldstart-assert.sh — --e2e mode (E1–E8), pod_probe() logs-based probes (A4 + E5), same PASS/FAIL/SKIP evidence-line contract
  • bin/germinate — FJ_PORT knob (default 3000, behavior unchanged for single-host use)
  • docs/ops/coldstart-ci.md — parallelism + superseding, FJ_PORT, nightly app-E2E, secret recipe, log locations

🤖 Generated with Claude Code

## What Second half of #67 + the CI-seam follow-ups from #21: the cold-start gate now runs **in parallel across refs** and grows a **nightly app-E2E variant** that ships a real canary app unattended — default-SKIP until an operator seals a credential. Originally based on `fix/gate-a4-detection` (#74); rebased onto main after #74 merged. ### 1. Parallel gates - Runner capacity on vxrail raised 1 → 2 (`~/act_runner/config.yml`, user unit restarted, re-registered cleanly, stayed registered and serving). - Workflow concurrency is now **per-ref** (`coldstart-${{ github.ref }}`, `cancel-in-progress: true`): different PRs gate in parallel; a push to the same ref supersedes its in-flight run. Total load is capped by runner capacity, not the group. - **Orphan sweep** at run start: a superseded run can be cancelled before its teardown, so each run removes `^k3d-ci[0-9]+-` containers older than 150 min (a live run fits the 110-min job cap). Match is anchored — prod (`k3d-platgold2`) and live dev clusters are never touched. - **`FJ_PORT` in `bin/germinate`** (one surgical knob, default `:3000` unchanged): the first parallel attempt killed run 11 with `admin auth failing after handoff` — germinate's forgejo port-forward binds a FIXED `127.0.0.1:3000`, so the second germination's curls silently reached the FIRST cluster's forgejo (different admin password). The workflow now exports a per-run port (`30000 + run_number % 10000`). CAVEAT: PR branches forked before this fix still bind `:3000` — at most one such gate at a time until rebased. ### 2. Nightly app-E2E (#67 second half) On the nightly cron or an `e2e=true` dispatch, after the standard gate, `scripts/coldstart-assert.sh --e2e` runs against the same CI daughter: - E1 claude credential sealed in `plat-agents-secrets` (fail fast) - E2 canary repo generated from `_app-template` (Forgejo API, 201) - E3 dispatcher bootstraps it: prime PR opens - E4 prime PR auto-merges (builder → preview → reviewer verdict → merge) - E5 prod serves 200 **in-cluster** (`Host <app>-plat.ci<N>.local` via the traefik svc) - E6 dispatcher log contains **no** `preview liveness unconfirmed` (#57 fallback) - E7 builder-finish → reviewer-queued gap **< 120 s** - E8 app prod namespace carries `plat.sh/app` + ResourceQuota + NetworkPolicies Credential: the germination step reads repo Actions secret `E2E_CLAUDE_TOKEN` and seals it as `CLAUDE_CODE_OAUTH_TOKEN` (values never echoed). **The secret is NOT created by this PR** (the staged host token is compromised-pending-rotation and was deliberately not wired in). When absent, e2e-flagged runs print one loud `E2E SKIP` line (host-persisted) and exit 0 — nightly stays green-by-default. Sealing recipe: `docs/ops/coldstart-ci.md` + comment on #67. Phase is hard-capped (`timeout 35m` step; `E2E_TIMEOUT` knob, default 1800 s). Humans run the identical thing: `scripts/coldstart-assert.sh --e2e <ctx> <domain>`. ### 3. Gate-probe hardening (found live during validation) A4's redirect probe used `kubectl run -i --rm`, whose attach can MISS output printed before it connects — observed live as a `no-XFP=?` false-FAIL under 3-way host load. A4 and the new E5 probe now share `pod_probe()`: run the pod detached, wait for a terminal phase, read **pod logs** (durable), delete. Re-ran the full standard gate on the same platform after the fix: PASS. ## Evidence ### Parallel proof (runs 12 + 13, dispatched simultaneously) ``` # 16:07:15 UTC, ~25s after the two dispatches — both clusters germinating side by side: k3d-ci12-serverlb 24 seconds ago k3d-ci12-server-0 24 seconds ago k3d-ci13-serverlb 25 seconds ago k3d-ci13-server-0 25 seconds ago # both green, distinct clusters, overlapping ~/ci-gate-logs timestamps: run 12 (feat/gate-parallel-nightly): success — "COLDSTART GATE PASS (93s) ... context=k3d-ci12" (run12-gate.log mtime 16:25:24) run 13 (main): success — "COLDSTART GATE PASS (94s) ... context=k3d-ci13" (run13-gate.log mtime 16:24:37) # zero leaks after both teardowns: docker ps -a --format '{{.Names}}' | grep -E 'k3d-ci1[23]-' -> (empty) ``` (First attempt, runs 10+11: run 10 green, run 11 died on the `:3000` cross-talk described above — that failure is what the `FJ_PORT` commit fixes; runs 12+13 are the post-fix proof.) ### E2E default-SKIP proof (runs 10 + 12, `e2e=true`, no secret — both runs green) ``` $ cat ~/ci-gate-logs/run12-e2e.log #### E2E SKIP: repo Actions secret E2E_CLAUDE_TOKEN is not sealed — app-E2E did NOT run (the standard gate above is this run's result); seal it per docs/ops/coldstart-ci.md (Nightly app E2E) to enable #### ``` ### Manual validation of the full `--e2e` path (marked manual-validation) Scratch germination `ci9999` on the vxrail host (PORTS=none, isolated KUBECONFIG, `FJ_PORT=31999`), credential taken from the host shell for this one run and never persisted into Forgejo/CI. All eight assertions passed; the canary shipped **unattended** in 793 s: ``` == coldstart E2E gate: context=k3d-ci9999 domain=ci9999.local app=plat/canary07021611 budget=1800s == PASS [E1-credential] ( 1s) claude credential sealed in plat-agents-secrets (ns agents; value never printed) PASS [E2-canary-created] ( 3s) POST /repos/plat/_app-template/generate -> 201 (plat/canary07021611) PASS [E3-prime-pr] ( 96s) dispatcher bootstrapped plat/canary07021611: prime PR #1 open PASS [E4-auto-merge] ( 610s) prime PR #1 merged ('v1 auto-merged after validation' log lines: 1) PASS [E5-prod-200] ( 793s) https://canary07021611-plat.ci9999.local/ -> 200 from an in-cluster pod PASS [E6-no-liveness-fallback] ( 793s) zero 'preview liveness unconfirmed' lines in the dispatcher log PASS [E7-review-latency] ( 793s) builder-finish 2026-07-02T16:19:23.933Z -> reviewer-queued 2026-07-02T16:19:57.238Z: 34s (< 120s) PASS [E8-ns-governance] ( 793s) ns plat--canary07021611--prod: plat.sh/app=canary07021611, 2 ResourceQuota, 6 NetworkPolicies (allow-egress allow-egress-postgres allow-ingress-from-workflows allow-ingress-postgres allow-ingress-traefik default-deny) COLDSTART E2E GATE PASS ( 793s): app plat/canary07021611 shipped builder->reviewer->prod unattended # dispatcher log lines behind E6/E7: 2026-07-02T16:19:23.933Z info run finished runId=builder-plat-canary07021611-... role=builder exitCode=0 durationMs=364063 2026-07-02T16:19:57.238Z info run queued runId=reviewer-plat-canary07021611-... role=reviewer item=#1 (gap: 34s) 2026-07-02T16:21:52.412Z info v1 auto-merged after validation pr=plat/canary07021611#1 ``` ## Files - `.forgejo/workflows/coldstart.yml` — per-ref concurrency, orphan sweep, per-run `FJ_PORT`, `e2e` input + phase, step-level timeouts (40m germinate / 20m gate / 35m e2e; job cap 110m) - `scripts/coldstart-assert.sh` — `--e2e` mode (E1–E8), `pod_probe()` logs-based probes (A4 + E5), same PASS/FAIL/SKIP evidence-line contract - `bin/germinate` — `FJ_PORT` knob (default 3000, behavior unchanged for single-host use) - `docs/ops/coldstart-ci.md` — parallelism + superseding, `FJ_PORT`, nightly app-E2E, secret recipe, log locations 🤖 Generated with [Claude Code](https://claude.com/claude-code) 
Parallel gates: concurrency group is now per-ref (coldstart-${{ github.ref }},
cancel-in-progress) so different PRs germinate in parallel — runner capacity
(2) is the cap, not the group. Same-ref pushes supersede their in-flight run;
an orphan sweep at run start removes k3d-ci<N> leftovers older than 150 min
(anchored match — prod and live clusters on the shared host are untouchable).

Nightly app-E2E (#67 second half): on schedule or e2e=true dispatch, after the
standard gate, coldstart-assert.sh --e2e generates a canary app from
_app-template on the CI daughter and asserts it ships UNATTENDED:
builder -> reviewer -> auto-merge -> prod 200 in-cluster, zero 'preview
liveness unconfirmed' lines, builder-finish -> reviewer-queued < 120s, and the
prod namespace carries plat.sh/app + quota + netpols. Credential comes from
the repo Actions secret E2E_CLAUDE_TOKEN sealed at germination; when absent
the phase prints one loud E2E SKIP line and exits 0 (default-safe: no secret
is created here, and the standard gate already ran).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 11 (main, germinating beside run 10) died on 'admin auth failing after
handoff': germinate's forgejo port-forward binds a FIXED 127.0.0.1:3000, so
the second germination's curls silently reach the FIRST cluster's forgejo
(different admin password). FJ_PORT parameterizes the local port (default
3000 — manual single-host use unchanged); the workflow now exports a
per-run port (30000 + run_number % 10000).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gate: read probes from pod logs — kubectl run -i attach loses early output
All checks were successful
coldstart / coldstart (pull_request) Successful in 18m18s
Details
753af68b8b 
A4 false-FAILed live (no-XFP=?) under 3-way host load: `kubectl run -i --rm`
can miss lines printed before the attach connects. A4 and the e2e E5 probe
now share pod_probe(): run detached, wait for a terminal phase, read the pod
LOGS (durable), delete. Re-ran the full gate on the same platform: PASS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
plat merged commit 82f5c29623 into main 2026-07-02 17:10:47 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
Something is not working

  
discussion
An idea that is still being discussed

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
goal

   
help wanted
Need some help

  
horizon:backlog

   
horizon:governance

   
horizon:mvp

   
invalid
Something is wrong

  
operator-decision
Blocked on the owner's call — no agent work until decided

  
question
More information is needed

  
roadmap

   
wontfix
This won't be fixed

No labels
bug
discussion
duplicate
enhancement
goal
help wanted
horizon:backlog
horizon:governance
horizon:mvp
invalid
operator-decision
question
roadmap
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
open-platform/mitosis!75
No description provided.