🧫 gauntlet: rehome failing #258

New issue

Closed

opened 2026-09-02 03:32:19 +00:00 by plat · 1 comment

plat commented

2026-09-02 03:32:19 +00:00

Owner

The gauntlet scenario rehome failed in https://git.open-platform.sh/open-platform/mitosis/actions/runs/21902 (2026-09-02T06:24:57Z).

Log tail:

    in-cluster DNS set: *.re-gt880-rehome.local -> Traefik (git/mcp/apps)
    cleartext identity + in-cluster DNS rewritten
    registry pull-secret re-keyed to git.re-gt880-rehome.local
    runner pull-secret re-keyed to git.re-gt880-rehome.local
    plat-mcp OAuth re-pointed (client kept)
    grafana OAuth re-pointed (client kept)
    fleet-edge OAuth re-pointed (client kept)
    repo websites re-pointed: 1
    gitops pushed
    node mirror recipe installed for git.re-gt880-rehome.local — restarting node(s) to apply (k3s reads it at start)
      k3d-gt880-rehome-server-0   Ready   control-plane,master   17m   v1.31.5+k3s1
    rolled mcp/mcp
    rolled agents/agents
    rolled mcp/plat-auth
    rolled monitoring/kube-prometheus-stack-grafana
    https://git.re-gt880-rehome.local/ → 200 ✓
    https://mcp.re-gt880-rehome.local/health → ok ✓

╔════════════════════════════════════════════════════════════════════════════╗
║                           PLATFORM REHOMED                                  ║
╚════════════════════════════════════════════════════════════════════════════╝
  gt880-rehome.local  →  re-gt880-rehome.local

  Outside the cluster, finish with:
    - DNS (or your tunnel): point git.re-gt880-rehome.local, mcp.re-gt880-rehome.local,
      grafana.re-gt880-rehome.local, www.re-gt880-rehome.local and app hosts (or *.re-gt880-rehome.local)
      at this cluster. The old hosts stop being served — routes moved.
    - local /etc/hosts users: replace the gt880-rehome.local lines with
      127.0.0.1  git.re-gt880-rehome.local  mcp.re-gt880-rehome.local  grafana.re-gt880-rehome.local
    - the CA did NOT change: existing plat-ca.crt trust keeps working.

  Brand: the forge title/site links were reset for re-gt880-rehome.local — if you had
  customized clusters/local/platform/forgejo/brand/brand.json, re-apply your
  edits there (python3 generate.py) and commit.
==> [4/4] full gate at re-gt880-rehome.local
== coldstart gate: context=k3d-gt880-rehome domain=re-gt880-rehome.local ==
== agents tag: 0.43.0 | mcp tag: 0.52.1 ==
PASS [A1-kustomizations] (   0s) 14 Kustomizations all Ready=True, incl. kyverno + kyverno-policies: apps cert-manager cnpg-operator flux-system keda kyverno kyverno-policies monitoring monitoring-scrapes pki platform priorityclasses tenants traefik 
PASS [A1-clusterpolicies] (   1s) 15 Kyverno ClusterPolicies present (want >=12)
PASS [A2-one-webhook] (   7s) exactly one system webhook: id=2(url=http://agents.agents.svc.cluster.local:9909/hook) 
PASS [A3-webhook-idempotent] (  62s) after rollout restart deploy/agents: still exactly one hook: id=2(url=http://agents.agents.svc.cluster.local:9909/hook) 
PASS [A3-webhook-verified-log] (  62s) dispatcher log contains 'system webhook verified' (agents 0.43.0)
PASS [A4-redirect-matrix] (  68s) no-XFP=301 XFP:https=200 https=200 (Host git.re-gt880-rehome.local via traefik.traefik.svc)
PASS [A4-traefik-log-clean] (  68s) zero 'middleware ... does not exist' errors in traefik logs
PASS [A5-forwarder-running] (  68s) forgejo/registry-node-config-4jsg5 Running
PASS [A5-node-listener] (  68s) node netns listener on :31100: tcp        0      0 127.0.0.1:31100         0.0.0.0:*               LISTEN      
PASS [A5-mirror-pull] (  68s) crictl pull git.re-gt880-rehome.local/open-platform/mcp:0.52.1 succeeded (anonymous)
PASS [A6-deploys-available] (  69s) deploy/agents Available=True (chart 0.43.0), deploy/mcp Available=True (chart 0.52.1)
PASS [A7-fork-genealogy] (  70s) mitosis migrated from https://git.open-platform.sh/open-platform/mitosis.git with parent history (>=5 commits)
PASS [A8-wiki] (  70s) native wiki live with 22 pages (source ships 20 content pages)
PASS [A9-grafana] (  71s) deploy/kube-prometheus-stack-grafana Available=True with 30 sidecar dashboards (fleet controls live)
PASS [A10-backup] (  71s) backup ns: 4 CronJobs (data protection born-on; off-host mirror=parent-only)
PASS [A11-ondemand] (  71s) sablier Available + on-demand-apps policy present (scale-to-zero engine born-on)
PASS [A12-front-door-answers] (  77s) https://www.re-gt880-rehome.local/ -> 200 from an in-cluster pod (deploy/www Available=True in open-platform--www--prod)
FAIL [A12-front-door-apex] (  77s) https://re-gt880-rehome.local/ -> '200' (want 30x -> https://www.re-gt880-rehome.local/)
PASS [A12-front-door-no-crew] (  78s) open-platform/www: releases=[0.1.1,0.1.0] pulls=0 issues=0 (bare template — the crew never built it)
SKIP [P1-upstream-pr] (  78s) UPSTREAM_SIGNING_KEY unset (humans without the mother key skip)

COLDSTART GATE FAIL (  78s): 1 assertion(s) failed:
 - [A12-front-door-apex] https://re-gt880-rehome.local/ -> '200' (want 30x -> https://www.re-gt880-rehome.local/)

Re-run just this scenario: dispatch the gauntlet workflow with scenarios=rehome. The scenario is code: scripts/gauntlet/scenarios/rehome.sh. This issue is edited in place on every failure and closes itself on the next pass.

The gauntlet scenario **rehome** failed in https://git.open-platform.sh/open-platform/mitosis/actions/runs/21902 (2026-09-02T06:24:57Z). Log tail: ``` in-cluster DNS set: *.re-gt880-rehome.local -> Traefik (git/mcp/apps) cleartext identity + in-cluster DNS rewritten registry pull-secret re-keyed to git.re-gt880-rehome.local runner pull-secret re-keyed to git.re-gt880-rehome.local plat-mcp OAuth re-pointed (client kept) grafana OAuth re-pointed (client kept) fleet-edge OAuth re-pointed (client kept) repo websites re-pointed: 1 gitops pushed node mirror recipe installed for git.re-gt880-rehome.local — restarting node(s) to apply (k3s reads it at start) k3d-gt880-rehome-server-0 Ready control-plane,master 17m v1.31.5+k3s1 rolled mcp/mcp rolled agents/agents rolled mcp/plat-auth rolled monitoring/kube-prometheus-stack-grafana https://git.re-gt880-rehome.local/ → 200 ✓ https://mcp.re-gt880-rehome.local/health → ok ✓ ╔════════════════════════════════════════════════════════════════════════════╗ ║ PLATFORM REHOMED ║ ╚════════════════════════════════════════════════════════════════════════════╝ gt880-rehome.local → re-gt880-rehome.local Outside the cluster, finish with: - DNS (or your tunnel): point git.re-gt880-rehome.local, mcp.re-gt880-rehome.local, grafana.re-gt880-rehome.local, www.re-gt880-rehome.local and app hosts (or *.re-gt880-rehome.local) at this cluster. The old hosts stop being served — routes moved. - local /etc/hosts users: replace the gt880-rehome.local lines with 127.0.0.1 git.re-gt880-rehome.local mcp.re-gt880-rehome.local grafana.re-gt880-rehome.local - the CA did NOT change: existing plat-ca.crt trust keeps working. Brand: the forge title/site links were reset for re-gt880-rehome.local — if you had customized clusters/local/platform/forgejo/brand/brand.json, re-apply your edits there (python3 generate.py) and commit. ==> [4/4] full gate at re-gt880-rehome.local == coldstart gate: context=k3d-gt880-rehome domain=re-gt880-rehome.local == == agents tag: 0.43.0 | mcp tag: 0.52.1 == PASS [A1-kustomizations] ( 0s) 14 Kustomizations all Ready=True, incl. kyverno + kyverno-policies: apps cert-manager cnpg-operator flux-system keda kyverno kyverno-policies monitoring monitoring-scrapes pki platform priorityclasses tenants traefik PASS [A1-clusterpolicies] ( 1s) 15 Kyverno ClusterPolicies present (want >=12) PASS [A2-one-webhook] ( 7s) exactly one system webhook: id=2(url=http://agents.agents.svc.cluster.local:9909/hook) PASS [A3-webhook-idempotent] ( 62s) after rollout restart deploy/agents: still exactly one hook: id=2(url=http://agents.agents.svc.cluster.local:9909/hook) PASS [A3-webhook-verified-log] ( 62s) dispatcher log contains 'system webhook verified' (agents 0.43.0) PASS [A4-redirect-matrix] ( 68s) no-XFP=301 XFP:https=200 https=200 (Host git.re-gt880-rehome.local via traefik.traefik.svc) PASS [A4-traefik-log-clean] ( 68s) zero 'middleware ... does not exist' errors in traefik logs PASS [A5-forwarder-running] ( 68s) forgejo/registry-node-config-4jsg5 Running PASS [A5-node-listener] ( 68s) node netns listener on :31100: tcp 0 0 127.0.0.1:31100 0.0.0.0:* LISTEN PASS [A5-mirror-pull] ( 68s) crictl pull git.re-gt880-rehome.local/open-platform/mcp:0.52.1 succeeded (anonymous) PASS [A6-deploys-available] ( 69s) deploy/agents Available=True (chart 0.43.0), deploy/mcp Available=True (chart 0.52.1) PASS [A7-fork-genealogy] ( 70s) mitosis migrated from https://git.open-platform.sh/open-platform/mitosis.git with parent history (>=5 commits) PASS [A8-wiki] ( 70s) native wiki live with 22 pages (source ships 20 content pages) PASS [A9-grafana] ( 71s) deploy/kube-prometheus-stack-grafana Available=True with 30 sidecar dashboards (fleet controls live) PASS [A10-backup] ( 71s) backup ns: 4 CronJobs (data protection born-on; off-host mirror=parent-only) PASS [A11-ondemand] ( 71s) sablier Available + on-demand-apps policy present (scale-to-zero engine born-on) PASS [A12-front-door-answers] ( 77s) https://www.re-gt880-rehome.local/ -> 200 from an in-cluster pod (deploy/www Available=True in open-platform--www--prod) FAIL [A12-front-door-apex] ( 77s) https://re-gt880-rehome.local/ -> '200' (want 30x -> https://www.re-gt880-rehome.local/) PASS [A12-front-door-no-crew] ( 78s) open-platform/www: releases=[0.1.1,0.1.0] pulls=0 issues=0 (bare template — the crew never built it) SKIP [P1-upstream-pr] ( 78s) UPSTREAM_SIGNING_KEY unset (humans without the mother key skip) COLDSTART GATE FAIL ( 78s): 1 assertion(s) failed: - [A12-front-door-apex] https://re-gt880-rehome.local/ -> '200' (want 30x -> https://www.re-gt880-rehome.local/) ``` Re-run just this scenario: dispatch the `gauntlet` workflow with `scenarios=rehome`. The scenario is code: `scripts/gauntlet/scenarios/rehome.sh`. This issue is edited in place on every failure and closes itself on the next pass.

plat referenced this issue

2026-09-02 03:36:05 +00:00

rehome: node-restart aftercare (uncordon, exact Ready, wait for the forge) #260

plat referenced this issue

2026-09-02 06:33:01 +00:00

sweep the escaped domain spelling; seed-release in-cluster API (#258, #263) #264

plat referenced this issue from a commit

2026-09-02 08:56:20 +00:00

sweep the escaped domain spelling; seed-release in-cluster API (#258, #263) (#264)

plat commented

2026-09-02 09:19:58 +00:00

Author