release train: agents 0.7.6 (webhook idempotency + in-cluster preview probe) → gitops pin bump → genesis reseed #66
Labels
No labels
bug
discussion
duplicate
enhancement
goal
help wanted
horizon:backlog
horizon:governance
horizon:mvp
invalid
operator-decision
question
roadmap
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
open-platform/mitosis#66
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Ties together the dispatcher fixes in flight so they reach daughters as ONE coordinated release, following the proven wb-gate pattern (mcp 0.26.0 + agents 0.7.5 tags → plat/gitops#10 pins → plat/mitosis#61 seed).
v0.7.6→ CI builds image + chartfix/web-entrypoint-https-redirect(#64 rework, in flight) lands in the same window, fold it into the same pin/seed cycle — one reseed instead of twokubectl rollout restart deploy/dispatcher(probeGET /api/v1/admin/hooks/{id}by id — the LIST endpoint returns[]and cannot be trusted, see #63) and the reviewer engages with no ~10-min "preview liveness unconfirmed" fallback (#57)DELETE /api/v1/admin/hooks/<dup>(#63)Exit: daughters germinated from the refreshed seed run agents 0.7.6 with a single stable system webhook and a fast reviewer; #63 and #57 close on the validation evidence.
#64 rework is validated and PR-ready: plat/gitops#11 (file-provider redirectScheme middleware; live matrix incl. restart race-check + tunnel XFP semantics passed on the daughter). The fold-in option is now concrete — merging plat/gitops#11 in the same window as the agents 0.7.6 pin bump means ONE reseed carries webhook-idempotency + preview-probe + the :80 redirect.
One more fold-in candidate for the train window: plat/gitops#12 (#60 prerequisite (a) — hostNetwork registry-mirror forwarder + pod-network realm rewriter; statically validated, needs one disposable germination to prove the node-sourced hop + hostNet DNS). The validation germination this train already requires (fresh daughter from the new seed) is exactly that disposable platform — sequence: merge agents#10+#11 + gitops#11 (+#12 if accepted) → tag 0.7.6 → pin bump → reseed → ONE validation germination proves the whole set: webhook idempotency (restart → still one hook), no 10-min reviewer fallback, :80 redirect matrix, and mirror pulls through the hostNetwork forwarder.
RC validation IN FLIGHT on the vxrail (prod host, isolated cluster
val076, PORTS=none — platgold2 untouched): assembledplat/agents v0.7.6-rc.1(merge of agents#10+#11+#12; 77 unit tests pass on the MERGED tree) +plat/gitops rc/0.7.6(gitops#11+#12 + agents pin 0.7.6-rc.1), RC seed built withGITOPS_REF=rc/0.7.6, throwaway germination running the full assertion matrix: whiteboard, ONE-webhook-after-restart (#63/#67), no-fallback reviewer (#57/#67), :80 redirect incl. germinated-first-boot race check (#64), hostNetwork mirror + crictl pull (#60 prereq), pure-spec next-idea bodies (#11), prod-isolation snapshots before/after. Evidence lands here when the run completes.PROD ROLLOUT WINDOW (advance notice): when val076 run 3 goes green AND the prod data snapshot completes, this train merges to prod via gitops main. What rolls: traefik (adds the :80→https redirectScheme file-provider middleware; tunnel XFP semantics preserved), registry-node-config (hostPort→hostNetwork forwarder split + NET_BIND_SERVICE capfix; a few seconds of mirror darkness per node during DaemonSet rollover, pulls retry), agents 0.7.6 (webhook idempotency #63, DNS-first preview probe #57, pure-spec bodies #11; the dispatcher restart resumes in-flight runs). Expected transients: one dispatcher restart, brief image-pull retries, a traefik reload — anything persisting >10 min after the pin-bump commit is a rollout defect and gets reverted (single git revert on gitops main). I will comment here immediately before merging and after Flux reconciles.
val076 RC validation — 0.7.6-rc.1 train GREEN (full matrix, fresh germination)
Team-validate evidence. Throwaway platform
val076germinated on the VxRail host (k3d,PORTS=none, prod untouched) from a seed carrying gitopsrc/0.7.6(headb3b6723, incl. the caddy capfix) + agentsv0.7.6-rc.1, with germinate running from plat/mitosisfix/germinate-prerelease-tags(62b5ad9, PR #68). All three validated TOGETHER — PR #68 must merge as part of this train.Two release-blockers found and fixed by this validation
bin/germinateextracted chart tags with a numeric-only regex underset -euo pipefail; the RC pin0.7.6-rc.1matched nothing and aborted germination silently at the post-Forgejo secrets step. Fixed in PR #68 (verbatim sed capture, non-fatal). Validated: run 3 extracted0.7.6-rc.1verbatim and drove the RC build.caddy:2-alpineships/usr/bin/caddy cap_net_bind_service=ep;capabilities: drop: ["ALL"]gives an empty bounding set, so execve failsoperation not permittedand BOTH registry-node-config + realm-rewrite crashloop, killing node pulls ofgit.val076.local/*(mcp/agents ImagePullBackOff, germinate FATAL). Docker repro matrix:--cap-drop=ALLfails,--cap-drop=ALL --cap-add=NET_BIND_SERVICE --security-opt no-new-privilegesruns clean. Fixed onrc/0.7.6(b3b6723,add: ["NET_BIND_SERVICE"]in both manifests). Validated: forwarder Running 0 restarts on fresh admission.Assertion matrix (run 3, fresh germination, all quoted from live evidence)
==> building agents v0.7.6-rc.1 — the chart Flux pins for the dispatcher→plat/charts/agents:0.7.6-rc.1 ready(attempt 1/4, first try)agents@0.7.6-rc.1+e64f15c54e0fReady; OCIRepository digest0.7.6-rc.1@sha256:e64f15c…; deploy imagegit.val076.local/plat/agents:0.7.6-rc.1; zero unhealthy podshttp://agents.agents.svc.cluster.local:9909/hook, ids 2–8 404).rollout restart deploy/agents→ still exactly one, same id 1; dispatcher loggedsystem webhook verified id=1 … signed=true(verify path, no second create)run finished … exitCode=0 durationMs=1277220(04:19:58) → reviewerrun queued04:19:50 (gap ≈ 0s, vs 10-min ceiling) →verdict posted … exitCode=0→v1 auto-merged after validation(04:23:59) → prod URLhttps://valpulse-plat.val076.local/200 in-cluster. Log contains 0preview liveness unconfirmed(healthy path:preview live on a different sha; reviewing anyway). Dispatcher:builder left its own ship summary; no fallback. POL-2: bothplat--valpulse--pr-1andplat--valpulse--prodgotplat.sh/app=valpulselabels,plat-kyverno-dos-bound+tenant-quotaquotas, 6 netpols incldefault-deny301 https://git.val076.local/;X-Forwarded-Proto: https→200; https SNI →200. Re-asserted identical AFTERrollout restart deploy/traefikon a platform that GERMINATED with the middleware; cold-boot traefik pod logs 0does not existmiddleware errors (pre-restart log had 30 case-insensitive matches attributable to initial-reconcile ordering; post-restart steady state is clean and behavior was correct throughout)hostNetwork=true ready=1, pod Running 0 restarts; node netns127.0.0.1:31100 LISTEN;registry-realm-rewrite1/1;crictl pull git.val076.local/plat/ci-builder:0.2.0→ successbody=params are pure specs (badge / leaderboard / milestones) with noagent-worktext; label instruction is a plain comment line below the links{"version":"15.0.0+gitea-1.22.0"}HTTP 200;k3d-platgold2-*Up 2 days, zero restarts; host*:80/*:443listeners unchanged; every kubectl carried--context k3d-val076; PORTS=none throughoutTimings
Seed build ~1 min; germination (cluster create → YOUR PLATFORM card) ~24 min incl. serial CI builds (ci-builder 0.1.0+0.2.0, mcp 0.26.0, agents 0.7.6-rc.1 — all attempt 1/4); valpulse end-to-end create→prod-200 ~30 min (builder 21m17s, reviewer 4m9s, verdict→merge <1s).
Verdict: the 0.7.6-rc.1 train (agents
v0.7.6-rc.1+ gitopsrc/0.7.6@b3b6723+ mitosis PR #68) is validated end-to-end on a fresh germination and ready to merge. Throwawayval076deleted after this comment; forensics (up.logs of all 3 runs, seeds, fork keys) retained atvxrail:~/val076-mitosis.MERGING NOW (run 3 full green — see evidence comment above; prod data snapshot verified + off-host copy taken). Order: agents rc/0.7.6→main + tag v0.7.6 → mitosis #68 → gitops #11+#12 → gitops pin 0.7.6 (prod reconciles here) → final reseed → #65. Canary heads-up: the prod-roll transients described in the rollout-window comment apply from the pin-bump commit.
TRAIN MERGED. agents: release PR #13 (rc/0.7.6) → main, tag v0.7.6, chart built on origin CI. mitosis: PR #68 merged. gitops: #11 (redirect) + #12 (mirror+capfix) + pin PR #13 (agents 0.7.6) all merged — prod Flux reconciling now (expected transients per the rollout-window comment; canary loop owns prod-side response). Remaining train items, resuming next wake: final reseed → update+merge #65 (closes #62); evidence-close #57/#63/#64/#11; #60 prereq comment. Session-limit note: two worker teams (CI-1 #21, fork #70) were cut mid-mission — partial work will be salvaged, and clusters ci2/fork70 on the vxrail may be orphans pending sweep.
TRAIN COMPLETE. Shipped: agents v0.7.6 (release #13; #10/#11/#12), gitops #11+#12+pin #13 (prod reconciled — verified live: agents:0.7.6 image, single system webhook after the documented legacy-hook remediation), mitosis #68 + reseed #65 (genesis seed = released train, bundle-verified). Issues closed with evidence: #57, #62, #63, #64, #11. Two release-blockers found+fixed+validated by the RC process itself (germinate pre-release regex; caddy file-caps EPERM). Prod data: full verified backup + nightly timer + off-host copy BEFORE the roll; prod healthy throughout. Remaining follow-ups tracked on their own issues: #21 (PR #71 + runner proof), #70 (validation+PR), #60 (Cilium swap, blocker B), #67 (gate assertions — now partially encoded in PR #71 gate script).
Train COMPLETE — every item on this issue shipped and proved: agents v0.7.6 on prod + in the seed (release #13; #57/#63/#11 fixes validated on val076 AND by every CI germination since — A3/E-block assertions), gitops redirect+mirror+pin merged (prod reconciled, no incident), mitosis #68, genesis reseed merged (#65) and then AUTOMATED (#14 closed — reseeds now build/verify/PR/gate themselves; two ran today: r25 merged, r27 correctly redundant). The one germination-proves-everything plan became the standing coldstart gate. Closing the train.