CI-1: automatic cold-start validation — shared gate script + coldstart workflow + vxrail runner (#21) #71

Merged

plat merged 2 commits from feat/ci-1-coldstart-gate into main

2026-07-02 14:16:15 +00:00

plat commented

2026-07-02 04:12:26 +00:00

Owner

Delivers CI-1 (#21): every change to plat/mitosis can now prove, in CI, that a FRESH platform germinates and passes the acceptance assertions. This session's two release-blockers (germinate tag-regex abort, caddy file-caps EPERM) were invisible to static checks and running platforms — only fresh germinations caught them. This makes that loop push-button and automatic.

What's in the PR

scripts/coldstart-assert.sh — THE acceptance gate, one source of truth for humans and CI (encodes the #67 assertions):
- A1: all Flux Kustomizations Ready=True (>=10, incl. kyverno + kyverno-policies); >=12 Kyverno ClusterPolicies
- A2: exactly ONE Forgejo system webhook, probed by id /api/v1/admin/hooks/{1..8} (the list endpoint returns [] while hooks exist — #67); admin creds read from the in-cluster forgejo-admin secret, not logs
- A3: webhook idempotency — rollout restart deploy/agents, wait Available, STILL exactly one hook; system webhook verified log check is version-aware (SKIPs on agents < 0.7.6 so old seeds can't false-fail)
- A4: https redirect matrix from an in-cluster pod (no-XFP → 301/308, XFP:https → 200, https → 200, zero does not exist middleware errors) — SKIPs wholesale on pre-#64 platforms without redirect-to-https
- A5: registry mirror — forwarder Running, node netns 127.0.0.1:31100 listener, real crictl pull git.<domain>/plat/ci-builder:0.2.0 (mechanism-agnostic: covers hostPort AND hostNetwork variants)
- A6: dispatcher + mcp Available, chart tags printed as evidence
.forgejo/workflows/coldstart.yml — triggers: pull_request (paths: bin/**, genesis/**, recipe/**, flake.* + the gate + itself), nightly cron, workflow_dispatch with optional gitops_ref (builds an RC seed via nix run .#seed using the new PLAT_PAT repo Actions secret; the default path needs zero secrets). Germinates ci<run_number> with PORTS=none FORK_KEY_ACK=1 under a 40-min hard timeout, runs the gate, then an if: always() teardown that force-removes and FAILS on any leaked k3d-ci* container. Concurrency group coldstart, cancel-in-progress: false, runner capacity 1 → one cold-start at a time.
docs/ops/coldstart-ci.md — runbook: runner provisioning/re-registration, manual gate usage, RC-seed dispatch recipe, future work (nightly app-E2E with a sealed Claude credential, parallel clusters, #14 seed-build automation on this same runner).

Isolation invariants (shared prod host)

per-run KUBECONFIG file — bin/up's kubectl config use-context never touches the host default context (which is prod)
FORK_KEY + kubeconfig live OUTSIDE the clone so nix flake tree copies can never sweep secrets into the world-readable store
PORTS=none always — host 80/443 never bound

Infra stood up (live, not repo state)

Repo-scoped runner vxrail-coldstart (label vxrail-coldstart:host, capacity 1) on the vxrail host: forgejo-runner v12.5.2 via nix profile, user systemd unit forgejo-runner.service with linger — survives restarts, no root daemon. Host has no node, so the workflow uses a manual git clone instead of actions/checkout.
Repo Actions secret PLAT_PAT for the gitops_ref RC-seed path.

Proof

Green end-to-end workflow_dispatch on this branch against the committed genesis/seed.tar.gz: run https://git.open-platform.sh/plat/mitosis/actions/runs/1 (timings in the run; docker ps verified clean of k3d-ci* afterwards). The committed seed's pre-train state exercises the version-aware SKIP paths legitimately.

No push/PR protection blocked this delivery — no bypass was needed. Not merging: leaving the merge to a human per team-ci scope.

🤖 Generated with Claude Code

Delivers **CI-1 (#21)**: every change to plat/mitosis can now prove, in CI, that a FRESH platform germinates and passes the acceptance assertions. This session's two release-blockers (germinate tag-regex abort, caddy file-caps EPERM) were invisible to static checks and running platforms — only fresh germinations caught them. This makes that loop push-button and automatic. ## What's in the PR - **`scripts/coldstart-assert.sh`** — THE acceptance gate, one source of truth for humans and CI (encodes the #67 assertions): - A1: all Flux Kustomizations Ready=True (>=10, incl. kyverno + kyverno-policies); >=12 Kyverno ClusterPolicies - A2: exactly ONE Forgejo system webhook, probed **by id** `/api/v1/admin/hooks/{1..8}` (the list endpoint returns `[]` while hooks exist — #67); admin creds read from the in-cluster `forgejo-admin` secret, not logs - A3: webhook idempotency — `rollout restart deploy/agents`, wait Available, STILL exactly one hook; `system webhook verified` log check is **version-aware** (SKIPs on agents < 0.7.6 so old seeds can't false-fail) - A4: https redirect matrix from an in-cluster pod (no-XFP → 301/308, XFP:https → 200, https → 200, zero `does not exist` middleware errors) — SKIPs wholesale on pre-#64 platforms without `redirect-to-https` - A5: registry mirror — forwarder Running, node netns `127.0.0.1:31100` listener, real `crictl pull git.<domain>/plat/ci-builder:0.2.0` (mechanism-agnostic: covers hostPort AND hostNetwork variants) - A6: dispatcher + mcp Available, chart tags printed as evidence - **`.forgejo/workflows/coldstart.yml`** — triggers: `pull_request` (paths: `bin/**, genesis/**, recipe/**, flake.*` + the gate + itself), nightly cron, `workflow_dispatch` with optional `gitops_ref` (builds an RC seed via `nix run .#seed` using the new `PLAT_PAT` repo Actions secret; the default path needs zero secrets). Germinates `ci<run_number>` with `PORTS=none FORK_KEY_ACK=1` under a 40-min hard timeout, runs the gate, then an `if: always()` teardown that force-removes and FAILS on any leaked `k3d-ci*` container. Concurrency group `coldstart`, `cancel-in-progress: false`, runner capacity 1 → one cold-start at a time. - **`docs/ops/coldstart-ci.md`** — runbook: runner provisioning/re-registration, manual gate usage, RC-seed dispatch recipe, future work (nightly app-E2E with a sealed Claude credential, parallel clusters, #14 seed-build automation on this same runner). ## Isolation invariants (shared prod host) - per-run `KUBECONFIG` file — `bin/up`'s `kubectl config use-context` never touches the host default context (which is prod) - `FORK_KEY` + kubeconfig live OUTSIDE the clone so nix flake tree copies can never sweep secrets into the world-readable store - `PORTS=none` always — host 80/443 never bound ## Infra stood up (live, not repo state) - Repo-scoped runner `vxrail-coldstart` (label `vxrail-coldstart:host`, capacity 1) on the vxrail host: `forgejo-runner` v12.5.2 via nix profile, user systemd unit `forgejo-runner.service` with linger — survives restarts, no root daemon. Host has no node, so the workflow uses a manual `git clone` instead of actions/checkout. - Repo Actions secret `PLAT_PAT` for the `gitops_ref` RC-seed path. ## Proof - Green end-to-end `workflow_dispatch` on this branch against the committed `genesis/seed.tar.gz`: run https://git.open-platform.sh/plat/mitosis/actions/runs/1 (timings in the run; `docker ps` verified clean of `k3d-ci*` afterwards). The committed seed's pre-train state exercises the version-aware SKIP paths legitimately. No push/PR protection blocked this delivery — no bypass was needed. Not merging: leaving the merge to a human per team-ci scope. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

plat added 1 commit

2026-07-02 04:12:26 +00:00

ci: CI-1 cold-start gate — shared assertion script + coldstart workflow + runbook

coldstart / coldstart (pull_request) Failing after 18m4s

Details

1282db7a57

One gate, two callers: scripts/coldstart-assert.sh runs the acceptance matrix
(flux/kyverno, exactly-one-webhook by id, restart idempotency version-aware,
redirect matrix, registry mirror, dispatcher+mcp) after ANY germination —
manual or CI. coldstart.yml germinates ci<run> on the vxrail host runner
(PORTS=none, per-run KUBECONFIG, always-teardown with leak check) on PR paths,
nightly cron, and dispatch (optional gitops_ref RC seed via PLAT_PAT secret).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

plat referenced this pull request

2026-07-02 13:07:47 +00:00

/goal /ultracode-workflows [CI-1] platform cold-start CI in plat/mitosis — Forgejo Actions + vcluster ephemeral germination #21

plat referenced this pull request

2026-07-02 13:07:47 +00:00

release train: agents 0.7.6 (webhook idempotency + in-cluster preview probe) → gitops pin bump → genesis reseed #66

plat force-pushed feat/ci-1-coldstart-gate from 1282db7a57

coldstart / coldstart (pull_request) Failing after 18m4s

Details

to 99f8303d52

coldstart / coldstart (pull_request) Successful in 17m24s

Details