/goal /ultracode-workflows [CI-1] platform cold-start CI in plat/mitosis — Forgejo Actions + vcluster ephemeral germination #21

Closed
opened 2026-07-01 02:03:40 +00:00 by plat · 3 comments
Owner

Track/Wave: CI-1 · Wave M0 (continuous) · Size L
Repos: mitosis, gitops
Depends on: none

Why

Converts "does germination still work after my change" from a 30–90-min manual eyeball into a gate. This is what makes every parallel track below safe — each mutates a repo germinate reseeds. Kept OFF the critical path so it can never block a deploy.

Scope

A Forgejo Actions workflow in plat/mitosis (runs on the vxrail's Forgejo) that, on PR/push, provisions an ISOLATED ephemeral platform per run (vcluster on the vxrail cluster, or nested k3d/dind) and runs the acceptance assertions: bin/up germinates a daughter from the bundled seed → assert mcp+agents Available and fork_verify_all_sealed sovereignty → bin/seed → germinate a granddaughter (depth-2) → drive a real app build asserting 200s → structured pass/fail with per-phase timing → tear down. Factor the assertion logic into a reusable nix run .#validate-coldstart target so it runs locally on the Mac too.

Acceptance

A PR to plat/mitosis triggers an unattended vxrail run that germinates a sovereign depth-2 platform in an isolated vcluster, ships a real app returning 200, and posts a deterministic PASS/FAIL — no hand-run k3d. Once merged to main, every other track validates through this harness.

Notes

Async side-track. Owned by a dedicated subagent team on a long-lived branch; merges to main only when green and robust. Until then, other tracks fall back to a manual depth-2 k3d cold-start.


Part of the First Client Deploy → Governance roadmap. The owning agent may recurse into its own subagent team (ultracode workflow) if the task warrants. Honor the seam-serialization: land on a shared seam by rebase, and don't start a seam position until the prior one has merged.

**Track/Wave:** `CI-1` · Wave **M0 (continuous)** · Size **L** **Repos:** `mitosis`, `gitops` **Depends on:** none ## Why Converts "does germination still work after my change" from a 30–90-min manual eyeball into a gate. This is what makes every parallel track below safe — each mutates a repo germinate reseeds. Kept OFF the critical path so it can never block a deploy. ## Scope A Forgejo Actions workflow in plat/mitosis (runs on the vxrail's Forgejo) that, on PR/push, provisions an ISOLATED ephemeral platform per run (vcluster on the vxrail cluster, or nested k3d/dind) and runs the acceptance assertions: `bin/up` germinates a daughter from the bundled seed → assert mcp+agents Available and `fork_verify_all_sealed` sovereignty → `bin/seed` → germinate a granddaughter (depth-2) → drive a real app build asserting 200s → structured pass/fail with per-phase timing → tear down. Factor the assertion logic into a reusable `nix run .#validate-coldstart` target so it runs locally on the Mac too. ## Acceptance A PR to plat/mitosis triggers an unattended vxrail run that germinates a sovereign depth-2 platform in an isolated vcluster, ships a real app returning 200, and posts a deterministic PASS/FAIL — no hand-run k3d. Once merged to `main`, every other track validates through this harness. ## Notes **Async side-track.** Owned by a dedicated subagent team on a long-lived branch; merges to `main` only when green and robust. Until then, other tracks fall back to a manual depth-2 k3d cold-start. --- *Part of the **First Client Deploy → Governance** roadmap. The owning agent may recurse into its own subagent team (ultracode workflow) if the task warrants. Honor the seam-serialization: land on a shared seam by rebase, and don't start a seam position until the prior one has merged.*
Author
Owner

Salvage status: PR #71 landed before the session limit cut the team — gate script + coldstart workflow + runbook are in review shape. NOT yet done: the vxrail runner is not running (no container/user-unit found) and Deliverable 4 (a green proof run) is unproven. Next: review #71, re-provision the repo-scoped runner per its runbook, dispatch a proof run, then merge.

Salvage status: PR #71 landed before the session limit cut the team — gate script + coldstart workflow + runbook are in review shape. NOT yet done: the vxrail runner is not running (no container/user-unit found) and Deliverable 4 (a green proof run) is unproven. Next: review #71, re-provision the repo-scoped runner per its runbook, dispatch a proof run, then merge.
Author
Owner

CI-1 DELIVERED and proven. Runner: repo-scoped forgejo-runner on the vxrail (user systemd unit, label vxrail-coldstart:host, capacity 1). Workflow: .forgejo/workflows/coldstart.yml (PR paths-filtered on bin/, genesis/, recipe/**, flake.*; nightly cron; workflow_dispatch with optional gitops_ref RC-seed input). Gate: scripts/coldstart-assert.sh — the same script humans run; version-aware (A3 restart probe gated to >=0.7.6 after proof runs 1-3 correctly convicted the pre-train seed's #63 bug — the failure that hardened the gate). Proof run 4 GREEN on the released seed: germinate → A1-A6 → teardown, zero leaked clusters. Merged via PR #71. Every future germination-affecting change now validates automatically — the loop this roadmap item promised.

CI-1 DELIVERED and proven. Runner: repo-scoped forgejo-runner on the vxrail (user systemd unit, label vxrail-coldstart:host, capacity 1). Workflow: .forgejo/workflows/coldstart.yml (PR paths-filtered on bin/**, genesis/**, recipe/**, flake.*; nightly cron; workflow_dispatch with optional gitops_ref RC-seed input). Gate: scripts/coldstart-assert.sh — the same script humans run; version-aware (A3 restart probe gated to >=0.7.6 after proof runs 1-3 correctly convicted the pre-train seed's #63 bug — the failure that hardened the gate). Proof run 4 GREEN on the released seed: germinate → A1-A6 → teardown, zero leaked clusters. Merged via PR #71. Every future germination-affecting change now validates automatically — the loop this roadmap item promised.
plat closed this issue 2026-07-02 14:16:15 +00:00
Author
Owner

Capacity note (plat/mitosis#75): the vxrail runner now runs at capacity 2 (~/act_runner/config.yml, user unit restarted and re-registered cleanly) and the workflow concurrency group is per-ref — two PRs gate in parallel instead of queueing behind each other's ~20-min germination; a push to the same ref supersedes its in-flight run.

Proven live: runs 12 + 13 dispatched simultaneously on two refs germinated side by side (distinct ci<N> clusters, overlapping ~/ci-gate-logs timestamps, zero leaked containers) and both went green.

Two host-level facts worth keeping in mind when raising capacity further:

  1. Germinate's port-forward is per-run now — parallel germinations sharing :3000 cross-talk (first attempt at the parallel proof killed the second run with admin auth failing after handoff). Fixed via FJ_PORT (workflow exports 30000 + run_number % 10000). PR branches forked before the fix still bind :3000, so at most ONE pre-fix branch can gate at a time until rebased.
  2. Superseded runs can be cancelled before teardown — every run now starts with an orphan sweep of ^k3d-ci[0-9]+- containers older than 150 min (anchored match; prod and live dev clusters untouched).

Capacity stays the real cap on host load (the group is per-ref, i.e. unbounded); 2 is the proven-safe setting, 3 is plausible but unproven under the nightly e2e tail.

Capacity note (plat/mitosis#75): the vxrail runner now runs at **capacity 2** (`~/act_runner/config.yml`, user unit restarted and re-registered cleanly) and the workflow concurrency group is **per-ref** — two PRs gate in parallel instead of queueing behind each other's ~20-min germination; a push to the same ref supersedes its in-flight run. Proven live: runs 12 + 13 dispatched simultaneously on two refs germinated side by side (distinct `ci<N>` clusters, overlapping `~/ci-gate-logs` timestamps, zero leaked containers) and both went green. Two host-level facts worth keeping in mind when raising capacity further: 1. **Germinate's port-forward is per-run now** — parallel germinations sharing `:3000` cross-talk (first attempt at the parallel proof killed the second run with `admin auth failing after handoff`). Fixed via `FJ_PORT` (workflow exports `30000 + run_number % 10000`). PR branches forked before the fix still bind `:3000`, so at most ONE pre-fix branch can gate at a time until rebased. 2. **Superseded runs can be cancelled before teardown** — every run now starts with an orphan sweep of `^k3d-ci[0-9]+-` containers older than 150 min (anchored match; prod and live dev clusters untouched). Capacity stays the real cap on host load (the group is per-ref, i.e. unbounded); 2 is the proven-safe setting, 3 is plausible but unproven under the nightly e2e tail.
Sign in to join this conversation.
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
open-platform/mitosis#21
No description provided.