🐤 App-building health canary — continuous validation + regression fixes (orchestration epic) #69
Labels
No labels
bug
discussion
duplicate
enhancement
goal
help wanted
horizon:backlog
horizon:governance
horizon:mvp
invalid
operator-decision
question
roadmap
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
open-platform/mitosis#69
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Mission: keep open-platform.sh app-building + preview generation reliable while another agent loads the vxrail heavily.
Model: an orchestrator (me) runs a self-paced validation loop; each cycle an ultracode Workflow team builds a canary app in
app-canary/and asserts the full path (create_app → CI build on ci-builder → crew opens PR → preview env deploys + serves 200). On any failure, a team diagnoses + fixes it (recursively), opening a PR that I proof-gate + merge. All incidents tracked as linked issues here.Baseline resolved: ci-builder:0.2.0 version-skew (new _app-template needs 0.2.0; in-place Flux upgrades do not rebuild it) — fixed by tagging plat/ci-builder v0.2.0. See the wheels-nice-auto-sales/www incident.
Running incident log: (comments below)
✅ Canary probe #1 — PASS (app-canary/canary-313c)
Full path validated: create_app provisioned (no failure), CI build green on ci-builder:0.2.0, crew opened PR #1, preview env app-canary--canary-313c--pr-1 (app pod + Postgres 1/1 Running), preview served HTTP 200 rendering the spec ("canary ok" + build timestamp, version 0.0.0-pr.1.5cae93e).
Health snapshot: core pods Ready (mcp/agents/forgejo/kyverno/flux); KEDA runners scaled 0→2→1; node MemoryPressure/DiskPressure/PIDPressure all False; peak 7% CPU / 5% mem — massive headroom. Only contention effect from the other agent: ~1-2 min HelmRelease reconcile lag (slowness, not failure). No fix needed.
Probe duration ~11.5 min. Next probe after a cooldown (one at a time, to not starve the other agent).
✅ Canary probe #2 — PASS (app-canary/canary-7712). Full path green: create_app→CI on ci-builder:0.2.0→crew PR #1→preview pod+Postgres Running→HTTP 200 ("canary ok"+timestamp). Health: core pods Ready, KEDA runners 0→2→0, node pressure False, 3% CPU / 5% mem — no contention observed from the other agent. Cleaned up prior canary-313c. Going forward I will batch PASSes into periodic rollups and comment individually only on FAIL/incidents to keep this epic readable.
🧹 Incident: canary cleanup leak — RESOLVED (probe-cleanup gap, not a platform bug)
Cause: a released prod app’s source of truth is a gitops overlay (
plat/gitops clusters/local/apps/<owner>/<app>/prod/: Namespace, CNPG, OCIRepository, HelmRelease, Ingress, NetPols) reconciled by the FluxappsKustomization (prune:true, 1m). The probe deleted only the Forgejo repo + namespace, so Flux re-applied the still-present overlay → prod resurrected. Repo-deletion is by design not a decommission trigger (dispatcher has norepository.deletedhandler;delete_appkeeps the repo by default) — so no platform bug, no PR.Correct decommission:
mcp delete_app(owner,name,force_repo=true)— or manually:git rmthe overlay dir + drop the- <owner>/<app>/prodkustomization line → push →flux reconcile kustomization apps(prune) → delete ns. Gitops prune MUST precede ns deletion.Cleanup done: both leaked canary overlays removed in one scoped gitops commit (
plat/gitopsmain156fb46, app-canary/* only), reconciled, verified no resurrection over 2 min. Reclaimed 4 pods, 2 CNPG clusters, 4 PVCs (~2.2Gi), 2 namespaces; 31 tenant namespaces untouched.Hardening: the canary probe’s cleanup step now fully decommissions (gitops-prune → reconcile → ns delete, scoped to app-canary/*) so future canaries can’t leak. Resuming build-probes.
📊 Rollup — probes #3-4 PASS (canary-4dc1, canary-k9t3). App-building + preview generation reliable; all core pods Ready, KEDA runners scale 0→2→0.
✅ Hardened cleanup VALIDATED: probe #4 decommissioned the prior app (canary-4dc1) overlay-first — gitops commit
5878dec→flux reconcile kustomization apps(prune) → ns delete → repo delete (204) — and verified no resurrection over ~95s. Leak definitively closed; scope stayed strictly within app-canary/*.⚠️ Watch — build LATENCY rising under the other agent's load: probe #4 took ~16 min create→serve vs ~9-11 min for probes #1-3. Every stage still went green; this is latency contention (builder/runner throughput), NOT resource exhaustion — node held 3-6% CPU, ~5% mem, all pressures False. Builds are slower but not failing. Guarding against this crossing into timeouts: going latency-aware — I will not pile canary builds onto a busy builder, and will record create→serve time each probe to track the trend.
📊 Probe #5 — PASS (with a harness correction). canary-0b01 built, previewed, auto-merged, and serves HTTP 200 in prod (pod 1/1 Running). The probe agent ended its turn early (~11min) while the crew was still building, so its verdict came back ambiguous — but the build path was healthy the whole time (crew impl push, preview run, KEDA scaled 3 runners). Verified the true outcome directly.
Latency recalibration (correcting the earlier #4 note): the Claude crew takes ~10min to write+validate even a trivial app — that is BASELINE, not contention. So create→serve ~13-18min is normal; probe #4's ~16min was NOT a contention warning. Withdrawing the latency-trend concern; node has held 2-8% CPU throughout.
Hardened the probe: step (c) now ACTIVELY polls (kubectl every ~20-30s, sleeping) until a pod is Running 1/1 OR a terminal failure OR ~20min timeout — and polls BOTH the pr-1 preview AND the --prod env (PRs auto-merge to prod, tearing down the preview), so it no longer returns an ambiguous mid-build verdict. Cleaned the inert canary-7712 repo. 5/5 probes effectively PASS.
📊 Rollup — probes #6-7 PASS (7/7 total). App-building + preview generation reliable end-to-end; hardened harness (active polling of pr-1+prod, gitops-prune decommission) proven stable across runs. Baseline create→serve ~13-18min = ordinary crew authoring time (not contention); node held 3-6% CPU throughout, KEDA runners scale 0→2→0, all core pods Ready. Decommission cleanups verified no-resurrection each time (incl. rebasing over the other agent's concurrent gitops commits). Now in an EFFICIENT posture: cheap health/risk checks each ~30min cycle, full build-probe only on a build-critical version change / load ramp / prior-FAIL / every 3rd cycle. agents 0.7.6 + mcp 0.26.0 steady. No incidents.
Mission complete: 7/7 canary probes PASS across the 0.7.6 load window; both incidents resolved (ci-builder:0.2.0 version-skew via the v0.2.0 tag; canary cleanup leak closed with gitops-prune decommission, no-resurrection verified). The load event shipped 07-02 and the orchestrator loop ended. Continuous app-E2E coverage now lives in CI's nightly gate (mitosis
c1c29b4), superseding this epic.