release train: agents 0.7.6 (webhook idempotency + in-cluster preview probe) → gitops pin bump → genesis reseed #66

Closed
opened 2026-07-02 02:20:22 +00:00 by plat · 9 comments
Owner

Ties together the dispatcher fixes in flight so they reach daughters as ONE coordinated release, following the proven wb-gate pattern (mcp 0.26.0 + agents 0.7.5 tags → plat/gitops#10 pins → plat/mitosis#61 seed).

  • merge plat/agents#10 — dispatcher: idempotent system webhook by persisted id (fixes #63)
  • merge plat/agents#11 — dispatcher: probe previews via in-cluster DNS first (fixes #57)
  • tag plat/agents v0.7.6 → CI builds image + chart
  • plat/gitops: bump the agents pin 0.7.5 → 0.7.6 (same shape as plat/gitops#10)
  • if plat/gitops fix/web-entrypoint-https-redirect (#64 rework, in flight) lands in the same window, fold it into the same pin/seed cycle — one reseed instead of two
  • plat/mitosis: reseed genesis AFTER the pins land (follow-up to PR #65 / #62 — that reseed carries the lib.sh trim fix for depth-2; this one carries 0.7.6)
  • validate on a fresh germination: exactly ONE admin hook survives kubectl rollout restart deploy/dispatcher (probe GET /api/v1/admin/hooks/{id} by id — the LIST endpoint returns [] and cannot be trusted, see #63) and the reviewer engages with no ~10-min "preview liveness unconfirmed" fallback (#57)
  • interim remediation on any already-running platform: probe hook ids and DELETE /api/v1/admin/hooks/<dup> (#63)

Exit: daughters germinated from the refreshed seed run agents 0.7.6 with a single stable system webhook and a fast reviewer; #63 and #57 close on the validation evidence.

Ties together the dispatcher fixes in flight so they reach daughters as ONE coordinated release, following the proven wb-gate pattern (mcp 0.26.0 + agents 0.7.5 tags → plat/gitops#10 pins → plat/mitosis#61 seed). - [ ] merge plat/agents#10 — dispatcher: idempotent system webhook by persisted id (fixes #63) - [ ] merge plat/agents#11 — dispatcher: probe previews via in-cluster DNS first (fixes #57) - [ ] tag plat/agents `v0.7.6` → CI builds image + chart - [ ] plat/gitops: bump the agents pin 0.7.5 → 0.7.6 (same shape as plat/gitops#10) - [ ] if plat/gitops `fix/web-entrypoint-https-redirect` (#64 rework, in flight) lands in the same window, fold it into the same pin/seed cycle — one reseed instead of two - [ ] plat/mitosis: reseed genesis AFTER the pins land (follow-up to PR #65 / #62 — that reseed carries the lib.sh trim fix for depth-2; this one carries 0.7.6) - [ ] validate on a fresh germination: exactly ONE admin hook survives `kubectl rollout restart deploy/dispatcher` (probe `GET /api/v1/admin/hooks/{id}` by id — the LIST endpoint returns `[]` and cannot be trusted, see #63) and the reviewer engages with no ~10-min "preview liveness unconfirmed" fallback (#57) - [ ] interim remediation on any already-running platform: probe hook ids and `DELETE /api/v1/admin/hooks/<dup>` (#63) Exit: daughters germinated from the refreshed seed run agents 0.7.6 with a single stable system webhook and a fast reviewer; #63 and #57 close on the validation evidence.
Author
Owner

#64 rework is validated and PR-ready: plat/gitops#11 (file-provider redirectScheme middleware; live matrix incl. restart race-check + tunnel XFP semantics passed on the daughter). The fold-in option is now concrete — merging plat/gitops#11 in the same window as the agents 0.7.6 pin bump means ONE reseed carries webhook-idempotency + preview-probe + the :80 redirect.

#64 rework is validated and PR-ready: plat/gitops#11 (file-provider redirectScheme middleware; live matrix incl. restart race-check + tunnel XFP semantics passed on the daughter). The fold-in option is now concrete — merging plat/gitops#11 in the same window as the agents 0.7.6 pin bump means ONE reseed carries webhook-idempotency + preview-probe + the :80 redirect.
Author
Owner

One more fold-in candidate for the train window: plat/gitops#12 (#60 prerequisite (a) — hostNetwork registry-mirror forwarder + pod-network realm rewriter; statically validated, needs one disposable germination to prove the node-sourced hop + hostNet DNS). The validation germination this train already requires (fresh daughter from the new seed) is exactly that disposable platform — sequence: merge agents#10+#11 + gitops#11 (+#12 if accepted) → tag 0.7.6 → pin bump → reseed → ONE validation germination proves the whole set: webhook idempotency (restart → still one hook), no 10-min reviewer fallback, :80 redirect matrix, and mirror pulls through the hostNetwork forwarder.

One more fold-in candidate for the train window: plat/gitops#12 (#60 prerequisite (a) — hostNetwork registry-mirror forwarder + pod-network realm rewriter; statically validated, needs one disposable germination to prove the node-sourced hop + hostNet DNS). The validation germination this train already requires (fresh daughter from the new seed) is exactly that disposable platform — sequence: merge agents#10+#11 + gitops#11 (+#12 if accepted) → tag 0.7.6 → pin bump → reseed → ONE validation germination proves the whole set: webhook idempotency (restart → still one hook), no 10-min reviewer fallback, :80 redirect matrix, and mirror pulls through the hostNetwork forwarder.
Author
Owner

RC validation IN FLIGHT on the vxrail (prod host, isolated cluster val076, PORTS=none — platgold2 untouched): assembled plat/agents v0.7.6-rc.1 (merge of agents#10+#11+#12; 77 unit tests pass on the MERGED tree) + plat/gitops rc/0.7.6 (gitops#11+#12 + agents pin 0.7.6-rc.1), RC seed built with GITOPS_REF=rc/0.7.6, throwaway germination running the full assertion matrix: whiteboard, ONE-webhook-after-restart (#63/#67), no-fallback reviewer (#57/#67), :80 redirect incl. germinated-first-boot race check (#64), hostNetwork mirror + crictl pull (#60 prereq), pure-spec next-idea bodies (#11), prod-isolation snapshots before/after. Evidence lands here when the run completes.

RC validation IN FLIGHT on the vxrail (prod host, isolated cluster `val076`, PORTS=none — platgold2 untouched): assembled `plat/agents v0.7.6-rc.1` (merge of agents#10+#11+#12; 77 unit tests pass on the MERGED tree) + `plat/gitops rc/0.7.6` (gitops#11+#12 + agents pin 0.7.6-rc.1), RC seed built with `GITOPS_REF=rc/0.7.6`, throwaway germination running the full assertion matrix: whiteboard, ONE-webhook-after-restart (#63/#67), no-fallback reviewer (#57/#67), :80 redirect incl. germinated-first-boot race check (#64), hostNetwork mirror + crictl pull (#60 prereq), pure-spec next-idea bodies (#11), prod-isolation snapshots before/after. Evidence lands here when the run completes.
Author
Owner

PROD ROLLOUT WINDOW (advance notice): when val076 run 3 goes green AND the prod data snapshot completes, this train merges to prod via gitops main. What rolls: traefik (adds the :80→https redirectScheme file-provider middleware; tunnel XFP semantics preserved), registry-node-config (hostPort→hostNetwork forwarder split + NET_BIND_SERVICE capfix; a few seconds of mirror darkness per node during DaemonSet rollover, pulls retry), agents 0.7.6 (webhook idempotency #63, DNS-first preview probe #57, pure-spec bodies #11; the dispatcher restart resumes in-flight runs). Expected transients: one dispatcher restart, brief image-pull retries, a traefik reload — anything persisting >10 min after the pin-bump commit is a rollout defect and gets reverted (single git revert on gitops main). I will comment here immediately before merging and after Flux reconciles.

PROD ROLLOUT WINDOW (advance notice): when val076 run 3 goes green AND the prod data snapshot completes, this train merges to prod via gitops main. What rolls: traefik (adds the :80→https redirectScheme file-provider middleware; tunnel XFP semantics preserved), registry-node-config (hostPort→hostNetwork forwarder split + NET_BIND_SERVICE capfix; a few seconds of mirror darkness per node during DaemonSet rollover, pulls retry), agents 0.7.6 (webhook idempotency #63, DNS-first preview probe #57, pure-spec bodies #11; the dispatcher restart resumes in-flight runs). Expected transients: one dispatcher restart, brief image-pull retries, a traefik reload — anything persisting >10 min after the pin-bump commit is a rollout defect and gets reverted (single git revert on gitops main). I will comment here immediately before merging and after Flux reconciles.
Author
Owner

val076 RC validation — 0.7.6-rc.1 train GREEN (full matrix, fresh germination)

Team-validate evidence. Throwaway platform val076 germinated on the VxRail host (k3d, PORTS=none, prod untouched) from a seed carrying gitops rc/0.7.6 (head b3b6723, incl. the caddy capfix) + agents v0.7.6-rc.1, with germinate running from plat/mitosis fix/germinate-prerelease-tags (62b5ad9, PR #68). All three validated TOGETHER — PR #68 must merge as part of this train.

Two release-blockers found and fixed by this validation

  1. germinate pre-release tag regex — bin/germinate extracted chart tags with a numeric-only regex under set -euo pipefail; the RC pin 0.7.6-rc.1 matched nothing and aborted germination silently at the post-Forgejo secrets step. Fixed in PR #68 (verbatim sed capture, non-fatal). Validated: run 3 extracted 0.7.6-rc.1 verbatim and drove the RC build.
  2. caddy file-caps EPERM in gitops#12 — official caddy:2-alpine ships /usr/bin/caddy cap_net_bind_service=ep; capabilities: drop: ["ALL"] gives an empty bounding set, so execve fails operation not permitted and BOTH registry-node-config + realm-rewrite crashloop, killing node pulls of git.val076.local/* (mcp/agents ImagePullBackOff, germinate FATAL). Docker repro matrix: --cap-drop=ALL fails, --cap-drop=ALL --cap-add=NET_BIND_SERVICE --security-opt no-new-privileges runs clean. Fixed on rc/0.7.6 (b3b6723, add: ["NET_BIND_SERVICE"] in both manifests). Validated: forwarder Running 0 restarts on fresh admission.

Assertion matrix (run 3, fresh germination, all quoted from live evidence)

# Assertion Result Evidence
A0 RC pin drove the build PASS ==> building agents v0.7.6-rc.1 — the chart Flux pins for the dispatcher → plat/charts/agents:0.7.6-rc.1 ready (attempt 1/4, first try)
A Platform whole PASS 10/10 Flux Kustomizations Ready=True; 12 ClusterPolicies Ready; HelmRelease agents@0.7.6-rc.1+e64f15c54e0f Ready; OCIRepository digest 0.7.6-rc.1@sha256:e64f15c…; deploy image git.val076.local/plat/agents:0.7.6-rc.1; zero unhealthy pods
B #63 webhook idempotency PASS Pre-restart: exactly one system hook (id 1 → http://agents.agents.svc.cluster.local:9909/hook, ids 2–8 404). rollout restart deploy/agents → still exactly one, same id 1; dispatcher logged system webhook verified id=1 … signed=true (verify path, no second create)
C #57 no-fallback + full pipeline PASS valpulse: PR #1 opened 03:58:41 → builder run finished … exitCode=0 durationMs=1277220 (04:19:58) → reviewer run queued 04:19:50 (gap ≈ 0s, vs 10-min ceiling) → verdict posted … exitCode=0 → v1 auto-merged after validation (04:23:59) → prod URL https://valpulse-plat.val076.local/ 200 in-cluster. Log contains 0 preview liveness unconfirmed (healthy path: preview live on a different sha; reviewing anyway). Dispatcher: builder left its own ship summary; no fallback. POL-2: both plat--valpulse--pr-1 and plat--valpulse--prod got plat.sh/app=valpulse labels, plat-kyverno-dos-bound + tenant-quota quotas, 6 netpols incl default-deny
D #64 redirect first-boot PASS From in-cluster pod: plain http → 301 https://git.val076.local/; X-Forwarded-Proto: https → 200; https SNI → 200. Re-asserted identical AFTER rollout restart deploy/traefik on a platform that GERMINATED with the middleware; cold-boot traefik pod logs 0 does not exist middleware errors (pre-restart log had 30 case-insensitive matches attributable to initial-reconcile ordering; post-restart steady state is clean and behavior was correct throughout)
E gitops#12 mirror PASS DaemonSet hostNetwork=true ready=1, pod Running 0 restarts; node netns 127.0.0.1:31100 LISTEN; registry-realm-rewrite 1/1; crictl pull git.val076.local/plat/ci-builder:0.2.0 → success
F agents#12 pure-spec bodies PASS Ship summary carries 3 next-idea links; URL-decoded body= params are pure specs (badge / leaderboard / milestones) with no agent-work text; label instruction is a plain comment line below the links
G Prod isolation PASS Before AND after: {"version":"15.0.0+gitea-1.22.0"} HTTP 200; k3d-platgold2-* Up 2 days, zero restarts; host *:80/*:443 listeners unchanged; every kubectl carried --context k3d-val076; PORTS=none throughout

Timings

Seed build ~1 min; germination (cluster create → YOUR PLATFORM card) ~24 min incl. serial CI builds (ci-builder 0.1.0+0.2.0, mcp 0.26.0, agents 0.7.6-rc.1 — all attempt 1/4); valpulse end-to-end create→prod-200 ~30 min (builder 21m17s, reviewer 4m9s, verdict→merge <1s).

Verdict: the 0.7.6-rc.1 train (agents v0.7.6-rc.1 + gitops rc/0.7.6 @ b3b6723 + mitosis PR #68) is validated end-to-end on a fresh germination and ready to merge. Throwaway val076 deleted after this comment; forensics (up.logs of all 3 runs, seeds, fork keys) retained at vxrail:~/val076-mitosis.

## val076 RC validation — 0.7.6-rc.1 train GREEN (full matrix, fresh germination) **Team-validate evidence.** Throwaway platform `val076` germinated on the VxRail host (k3d, `PORTS=none`, prod untouched) from a seed carrying **gitops `rc/0.7.6`** (head `b3b6723`, incl. the caddy capfix) + **agents `v0.7.6-rc.1`**, with germinate running from **plat/mitosis `fix/germinate-prerelease-tags` (`62b5ad9`, PR #68)**. All three validated TOGETHER — **PR #68 must merge as part of this train.** ### Two release-blockers found and fixed by this validation 1. **germinate pre-release tag regex** — `bin/germinate` extracted chart tags with a numeric-only regex under `set -euo pipefail`; the RC pin `0.7.6-rc.1` matched nothing and aborted germination silently at the post-Forgejo secrets step. Fixed in PR #68 (verbatim sed capture, non-fatal). Validated: run 3 extracted `0.7.6-rc.1` verbatim and drove the RC build. 2. **caddy file-caps EPERM in gitops#12** — official `caddy:2-alpine` ships `/usr/bin/caddy cap_net_bind_service=ep`; `capabilities: drop: ["ALL"]` gives an empty bounding set, so execve fails `operation not permitted` and BOTH registry-node-config + realm-rewrite crashloop, killing node pulls of `git.val076.local/*` (mcp/agents ImagePullBackOff, germinate FATAL). Docker repro matrix: `--cap-drop=ALL` fails, `--cap-drop=ALL --cap-add=NET_BIND_SERVICE --security-opt no-new-privileges` runs clean. Fixed on `rc/0.7.6` (`b3b6723`, `add: ["NET_BIND_SERVICE"]` in both manifests). Validated: forwarder Running 0 restarts on fresh admission. ### Assertion matrix (run 3, fresh germination, all quoted from live evidence) | # | Assertion | Result | Evidence | |---|-----------|--------|----------| | A0 | RC pin drove the build | PASS | `==> building agents v0.7.6-rc.1 — the chart Flux pins for the dispatcher` → `plat/charts/agents:0.7.6-rc.1 ready` (attempt 1/4, first try) | | A | Platform whole | PASS | 10/10 Flux Kustomizations Ready=True; 12 ClusterPolicies Ready; HelmRelease `agents@0.7.6-rc.1+e64f15c54e0f` Ready; OCIRepository digest `0.7.6-rc.1@sha256:e64f15c…`; deploy image `git.val076.local/plat/agents:0.7.6-rc.1`; zero unhealthy pods | | B | #63 webhook idempotency | PASS | Pre-restart: exactly one system hook (id 1 → `http://agents.agents.svc.cluster.local:9909/hook`, ids 2–8 404). `rollout restart deploy/agents` → still exactly one, same id 1; dispatcher logged `system webhook verified id=1 … signed=true` (verify path, no second create) | | C | #57 no-fallback + full pipeline | PASS | valpulse: PR #1 opened 03:58:41 → builder `run finished … exitCode=0 durationMs=1277220` (04:19:58) → reviewer `run queued` 04:19:50 (**gap ≈ 0s**, vs 10-min ceiling) → `verdict posted … exitCode=0` → `v1 auto-merged after validation` (04:23:59) → prod URL `https://valpulse-plat.val076.local/` **200** in-cluster. Log contains **0** `preview liveness unconfirmed` (healthy path: `preview live on a different sha; reviewing anyway`). Dispatcher: `builder left its own ship summary; no fallback`. POL-2: both `plat--valpulse--pr-1` and `plat--valpulse--prod` got `plat.sh/app=valpulse` labels, `plat-kyverno-dos-bound` + `tenant-quota` quotas, 6 netpols incl `default-deny` | | D | #64 redirect first-boot | PASS | From in-cluster pod: plain http → `301 https://git.val076.local/`; `X-Forwarded-Proto: https` → `200`; https SNI → `200`. Re-asserted identical AFTER `rollout restart deploy/traefik` on a platform that GERMINATED with the middleware; cold-boot traefik pod logs **0** `does not exist` middleware errors (pre-restart log had 30 case-insensitive matches attributable to initial-reconcile ordering; post-restart steady state is clean and behavior was correct throughout) | | E | gitops#12 mirror | PASS | DaemonSet `hostNetwork=true ready=1`, pod Running 0 restarts; node netns `127.0.0.1:31100 LISTEN`; `registry-realm-rewrite` 1/1; `crictl pull git.val076.local/plat/ci-builder:0.2.0` → success | | F | agents#12 pure-spec bodies | PASS | Ship summary carries 3 next-idea links; URL-decoded `body=` params are pure specs (badge / leaderboard / milestones) with **no `agent-work` text**; label instruction is a plain comment line below the links | | G | Prod isolation | PASS | Before AND after: `{"version":"15.0.0+gitea-1.22.0"}` HTTP 200; `k3d-platgold2-*` Up 2 days, zero restarts; host `*:80`/`*:443` listeners unchanged; every kubectl carried `--context k3d-val076`; PORTS=none throughout | ### Timings Seed build ~1 min; germination (cluster create → YOUR PLATFORM card) ~24 min incl. serial CI builds (ci-builder 0.1.0+0.2.0, mcp 0.26.0, agents 0.7.6-rc.1 — all attempt 1/4); valpulse end-to-end create→prod-200 ~30 min (builder 21m17s, reviewer 4m9s, verdict→merge <1s). **Verdict: the 0.7.6-rc.1 train (agents `v0.7.6-rc.1` + gitops `rc/0.7.6` @ `b3b6723` + mitosis PR #68) is validated end-to-end on a fresh germination and ready to merge.** Throwaway `val076` deleted after this comment; forensics (up.logs of all 3 runs, seeds, fork keys) retained at `vxrail:~/val076-mitosis`.
Author
Owner

MERGING NOW (run 3 full green — see evidence comment above; prod data snapshot verified + off-host copy taken). Order: agents rc/0.7.6→main + tag v0.7.6 → mitosis #68 → gitops #11+#12 → gitops pin 0.7.6 (prod reconciles here) → final reseed → #65. Canary heads-up: the prod-roll transients described in the rollout-window comment apply from the pin-bump commit.

MERGING NOW (run 3 full green — see evidence comment above; prod data snapshot verified + off-host copy taken). Order: agents rc/0.7.6→main + tag v0.7.6 → mitosis #68 → gitops #11+#12 → gitops pin 0.7.6 (prod reconciles here) → final reseed → #65. Canary heads-up: the prod-roll transients described in the rollout-window comment apply from the pin-bump commit.
Author
Owner

TRAIN MERGED. agents: release PR #13 (rc/0.7.6) → main, tag v0.7.6, chart built on origin CI. mitosis: PR #68 merged. gitops: #11 (redirect) + #12 (mirror+capfix) + pin PR #13 (agents 0.7.6) all merged — prod Flux reconciling now (expected transients per the rollout-window comment; canary loop owns prod-side response). Remaining train items, resuming next wake: final reseed → update+merge #65 (closes #62); evidence-close #57/#63/#64/#11; #60 prereq comment. Session-limit note: two worker teams (CI-1 #21, fork #70) were cut mid-mission — partial work will be salvaged, and clusters ci2/fork70 on the vxrail may be orphans pending sweep.

TRAIN MERGED. agents: release PR #13 (rc/0.7.6) → main, tag v0.7.6, chart built on origin CI. mitosis: PR #68 merged. gitops: #11 (redirect) + #12 (mirror+capfix) + pin PR #13 (agents 0.7.6) all merged — prod Flux reconciling now (expected transients per the rollout-window comment; canary loop owns prod-side response). Remaining train items, resuming next wake: final reseed → update+merge #65 (closes #62); evidence-close #57/#63/#64/#11; #60 prereq comment. Session-limit note: two worker teams (CI-1 #21, fork #70) were cut mid-mission — partial work will be salvaged, and clusters ci2/fork70 on the vxrail may be orphans pending sweep.
Author
Owner

TRAIN COMPLETE. Shipped: agents v0.7.6 (release #13; #10/#11/#12), gitops #11+#12+pin #13 (prod reconciled — verified live: agents:0.7.6 image, single system webhook after the documented legacy-hook remediation), mitosis #68 + reseed #65 (genesis seed = released train, bundle-verified). Issues closed with evidence: #57, #62, #63, #64, #11. Two release-blockers found+fixed+validated by the RC process itself (germinate pre-release regex; caddy file-caps EPERM). Prod data: full verified backup + nightly timer + off-host copy BEFORE the roll; prod healthy throughout. Remaining follow-ups tracked on their own issues: #21 (PR #71 + runner proof), #70 (validation+PR), #60 (Cilium swap, blocker B), #67 (gate assertions — now partially encoded in PR #71 gate script).

TRAIN COMPLETE. Shipped: agents v0.7.6 (release #13; #10/#11/#12), gitops #11+#12+pin #13 (prod reconciled — verified live: agents:0.7.6 image, single system webhook after the documented legacy-hook remediation), mitosis #68 + reseed #65 (genesis seed = released train, bundle-verified). Issues closed with evidence: #57, #62, #63, #64, #11. Two release-blockers found+fixed+validated by the RC process itself (germinate pre-release regex; caddy file-caps EPERM). Prod data: full verified backup + nightly timer + off-host copy BEFORE the roll; prod healthy throughout. Remaining follow-ups tracked on their own issues: #21 (PR #71 + runner proof), #70 (validation+PR), #60 (Cilium swap, blocker B), #67 (gate assertions — now partially encoded in PR #71 gate script).
Author
Owner

Train COMPLETE — every item on this issue shipped and proved: agents v0.7.6 on prod + in the seed (release #13; #57/#63/#11 fixes validated on val076 AND by every CI germination since — A3/E-block assertions), gitops redirect+mirror+pin merged (prod reconciled, no incident), mitosis #68, genesis reseed merged (#65) and then AUTOMATED (#14 closed — reseeds now build/verify/PR/gate themselves; two ran today: r25 merged, r27 correctly redundant). The one germination-proves-everything plan became the standing coldstart gate. Closing the train.

Train COMPLETE — every item on this issue shipped and proved: agents v0.7.6 on prod + in the seed (release #13; #57/#63/#11 fixes validated on val076 AND by every CI germination since — A3/E-block assertions), gitops redirect+mirror+pin merged (prod reconciled, no incident), mitosis #68, genesis reseed merged (#65) and then AUTOMATED (#14 closed — reseeds now build/verify/PR/gate themselves; two ran today: r25 merged, r27 correctly redundant). The one germination-proves-everything plan became the standing coldstart gate. Closing the train.
plat closed this issue 2026-07-02 19:04:51 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
open-platform/mitosis#66
No description provided.