Cold-start: fresh germination can stall on the ci-builder CI build (orphaned run, KEDA won't re-scale a runner) #19

New issue

Closed

opened 2026-06-30 22:02:28 +00:00 by plat · 0 comments

plat commented

2026-06-30 22:02:28 +00:00

Owner

Found while validating clone-and-go on a fresh colima cluster. A fresh nix run .#up germination intermittently stalls in build_and_verify waiting for plat/ci-builder:0.1.0 to land. The ci-builder CI run wedges in a "running" state with no runner pod executing it, and KEDA's forgejo-runner ScaledJob stays Active=False (it only scales for queued jobs, not orphaned "running" ones), so nothing re-picks it up. germinate retries the build 4× (35m budget each) and each attempt re-orphans — ~2h then FATAL.

Symptoms observed (cold.local, this run):

KEDA forgejo-runner ScaledJob: Active=False, lastActiveTime frozen ~when the early CI steps finished; 0 runner pods.
germinate log: …waiting for plat/ci-builder:0.1.0 (Nm left, CI=running) counting down across attempts 1→3.
Node NOT resource-starved (CPU 3%, mem 17%) — so it's scheduling, not capacity.

Suspected cause: the ephemeral runner pod that claims the ci-builder job exits/terminates mid-build (heavy toolchain image), leaving the Forgejo run marked "running" forever; KEDA's postgres trigger counts only queued tasks, so it never spawns a replacement runner. germinate's re-trigger creates a run that's immediately claimed-then-orphaned again.

Fix directions:

germinate build_and_verify: detect an orphaned/stuck "running" CI run (no progress for N min) and cancel + re-queue it (so KEDA sees a queued job), rather than just waiting out the 35m timeout.
Make the KEDA trigger count stuck/stale "running" tasks (or reap runs whose runner is gone) so a replacement runner spawns.
Ensure the runner pod outlives a long ci-builder build (job activeDeadline / no premature pod completion).

Note: intermittent — germination succeeded cleanly multiple times this session (plat.local, gc.local). The secrets stack itself is unaffected (the re-seeded bundle verifiably carries mcp 0.23.15 + secrets-cli + the gitops pins). This is cold-start reliability hardening, tracked toward #10.

**Found while validating clone-and-go on a fresh colima cluster.** A fresh `nix run .#up` germination intermittently stalls in `build_and_verify` waiting for `plat/ci-builder:0.1.0` to land. The ci-builder CI run wedges in a "running" state with **no runner pod executing it**, and KEDA's `forgejo-runner` ScaledJob stays `Active=False` (it only scales for *queued* jobs, not orphaned "running" ones), so nothing re-picks it up. germinate retries the build 4× (35m budget each) and each attempt re-orphans — ~2h then FATAL. **Symptoms observed (cold.local, this run):** - KEDA `forgejo-runner` ScaledJob: `Active=False`, `lastActiveTime` frozen ~when the early CI steps finished; 0 runner pods. - germinate log: `…waiting for plat/ci-builder:0.1.0 (Nm left, CI=running)` counting down across attempts 1→3. - Node NOT resource-starved (CPU 3%, mem 17%) — so it's scheduling, not capacity. **Suspected cause:** the ephemeral runner pod that claims the ci-builder job exits/terminates mid-build (heavy toolchain image), leaving the Forgejo run marked "running" forever; KEDA's postgres trigger counts only queued tasks, so it never spawns a replacement runner. germinate's re-trigger creates a run that's immediately claimed-then-orphaned again. **Fix directions:** - germinate `build_and_verify`: detect an orphaned/stuck "running" CI run (no progress for N min) and **cancel + re-queue** it (so KEDA sees a queued job), rather than just waiting out the 35m timeout. - Make the KEDA trigger count stuck/stale "running" tasks (or reap runs whose runner is gone) so a replacement runner spawns. - Ensure the runner pod outlives a long ci-builder build (job activeDeadline / no premature pod completion). **Note:** intermittent — germination succeeded cleanly multiple times this session (plat.local, gc.local). The secrets stack itself is unaffected (the re-seeded bundle verifiably carries mcp 0.23.15 + secrets-cli + the gitops pins). This is cold-start *reliability* hardening, tracked toward #10.

plat referenced this issue

2026-07-01 00:20:01 +00:00

genesis: refresh bundled seed to carry the secrets stack (toward #14) #20

plat referenced this issue from a commit

2026-07-01 00:20:02 +00:00

genesis: refresh seed to carry the secrets stack (mcp 0.23.15 + secrets-cli + agents 0.7.3 + sync-env)

plat referenced this issue

2026-07-01 00:22:18 +00:00

self-serve secrets #2

plat referenced this issue

2026-07-01 00:22:19 +00:00

Automate the genesis seed build (auditable Forgejo workflow, not a hand-committed blob) #14