open-platform/mitosis
3
0
Fork 0
Code Issues 27 Pull requests 9 Releases 4 Packages 1 Wiki Activity Actions

Horizontal scaling: migrate product-garden off the DO droplet, then make it the platform's 2nd node #101

New issue
Open
opened 2026-07-17 04:18:12 +00:00 by plat · 4 comments
plat commented 2026-07-17 04:18:12 +00:00
Owner
Copy link

Handoff: migrate product-garden off the DO droplet, then make it the platform's second node

You are taking on a horizontal-scaling initiative for the mitosis platform (Forgejo-as-control-plane,
open-platform.sh). Use subagent teams / ultracode workflows — this is multi-phase and
parallelizable. Discover before you act; validate every phase; never break the live platform.

The goal (two moves, in order)

  1. Test the migration feature at real scale by migrating everything off the DigitalOcean
    droplet pg-grand (which currently runs the product-garden depth-2 QA platform) ONTO the
    main platform (open-platform.sh). Then decommission product-garden and free the droplet.
  2. Repurpose the freed droplet as a second cluster node, so the platform stops being capped
    at one node's ~254 pods. Prove pods schedule across ≥2 nodes. This is step one of a real
    multi-machine cluster — design it so adding node 3, 4, … is a repeatable runbook, not a one-off.

Known facts / custody (verify all of this first — don't trust it blindly)

  • Main platform = platgold2, a k3s v1.31.5 cluster wrapped in k3d (Docker) on host
    vxrail (Tailscale 100.84.55.62, ssh vxrail). Reach the API: ssh vxrail 'docker exec k3d-platgold2-server-0 kubectl …'. It is a single node (k3d-platgold2-server-0,
    control-plane+master), pod CIDR 10.42.0.0/24 → ~254 pod IP ceiling (this is the wall we're
    removing). Edge = Cloudflare tunnel (mitosis/tunnel-token) + wildcard DNS + Let's Encrypt.
    GitOps = plat/gitops (Flux); cluster/node config lives there + in the k3d/k3s launch on vxrail.
  • DO droplet pg-grand: 167.172.29.178, nyc3, s-4vcpu-8gb (4 vCPU / 8 GB), Ubuntu 24.04,
    active. Runs product-garden (a sovereign daughter platform; enforce + forwardAuth + cosign ON).
    SSH: try ~/.ssh/mitosis-do as root (my keys got publickey denied — find the right key; the
    DO deploy key or a key in mitosis/). DO API token: /Users/trevato/projects/mitosis/dg-token
    (Bearer). doctl/curl the DO API for droplet/network control.
  • Migration feature: the platform migrates repos/apps via Forgejo "New Migration" →
    dispatcher classifyMigration → bootstrapMigration → a migrator agent run
    (agents-entity src/dispatcher.ts:~649, ~1803; prompts/migrator.md; mcp create_app({migrate:true})).
    Also relevant: germinate/daughter creation (mitosis/bin/germinate), and app decommission =
    remove the gitops overlay (Flux prunes; direct kubectl delete ns alone gets resurrected).
  • Operator identity: trevato (operator, owns product orgs) vs plat (admin). Local git
    credential for git.open-platform.sh authenticates as plat (site admin). See memory
    openplatform-prod-custody, product-garden-qa-platform, mitosis-entity-workflows,
    yardwaze-client-platform (a prior sovereign-deploy with the exact k3s + Cloudflare-tunnel topo).

The hard architectural question (resolve in discovery, don't assume)

platgold2's k3s API runs INSIDE a k3d Docker container on vxrail; k3d is designed for local
multi-node via Docker, not remote nodes. So "join the DO droplet as an agent" has two candidate
shapes — evaluate both and pick with evidence:
(A) Add a remote k3s agent to the existing k3d-wrapped server over Tailscale (put the droplet
on the tailnet; expose/point the agent at the k3d server's API + node token; solve cross-WAN
CNI — flannel backend, MTU, the k3d docker-bridge-vs-real-node networking). Least disruptive
if it works; unusual and possibly fragile.
(B) Evolve off k3d to a real multi-node k3s (server on vxrail bare/VM, agents on DO + future
machines, flat network over Tailscale/WireGuard). Cleaner long-term, bigger lift, must migrate
the live platform's state — high risk, needs a careful cutover plan.
Recommend one with a reversibility/risk analysis before touching prod. Networking is the crux:
the nodes need a flat, stable network (Tailscale mesh is the strong default; vxrail is already on it).

Phasing (gate each phase on validation; keep the live platform healthy throughout)

  • Phase 0 — Discovery (parallel team): inventory product-garden on the droplet (orgs, repos,
    apps, DATA, users, edge/DNS); map platgold2's k3d/k3s launch + networking + gitops node config;
    spike the migration feature on ONE small app end-to-end; spike the node-join networking (A vs B)
    on a throwaway. Produce a topology decision + a migration inventory + a risk register.
  • Phase 1 — Migrate product-garden → open-platform.sh: drive the real migration per app/repo
    (Forgejo New Migration → migrator), preserving data (this is a QA platform but treat its data
    as real — snapshot/backup first). Validate each migrated app runs on the main platform. This is
    the migration-feature test the operator wants — capture what worked / what the feature is missing.
  • Phase 2 — Decommission product-garden + free the droplet: once migration is validated, tear
    down product-garden (gitops overlays + DNS + the droplet's k8s), snapshot the droplet, wipe to a
    clean Ubuntu base. Confirm nothing on open-platform.sh depended on it.
  • Phase 3 — Join the droplet as a node + prove scale: implement the chosen topology; put the
    droplet on the tailnet; join it; confirm kubectl get nodes shows it Ready and schedulable;
    cordon-drain-test; schedule real pods on it; confirm the ~254-pod wall is gone (aggregate capacity
    across nodes). Write the repeatable add-a-node runbook so machines 3+ are trivial.
  • Phase 4 — Harden + document: node labels/taints (keep control-plane workloads off the remote
    agent if flaky-WAN; put stateless app pods there), netpol/CNI across nodes, monitoring covers both
    nodes (the Grafana stack is live — extend node-exporter/scrape), gitops as source of truth for the
    node set. Update mitosis docs + memory.

Guardrails (non-negotiable)

  • The live platform (19 orgs actively building) must not go down. No experiments on platgold2's
    networking without a rollback; spike on throwaway droplets/clusters first.
  • Preserve data through the migration — snapshot before you migrate or delete anything.
  • Reversible steps only on prod; every prod change gated on a validated spike.
  • Security: don't widen exposure — the tailnet is private; don't put the k3s API on the public
    internet. Respect the operator/admin split. Cross-node secrets stay sealed (sops/Flux).
  • Report progress on an issue; surface blockers as severity → impact → path.

Acceptance

  • product-garden fully migrated to open-platform.sh (apps live + validated) OR a documented,
    operator-approved decision on what to migrate vs archive; droplet freed.
  • Migration-feature test written up: what worked, gaps, fixes filed on plat/agents / plat/mcp.
  • The droplet is a Ready, schedulable node; real pods run on it; aggregate pod capacity > one /24.
  • A repeatable "add a machine" runbook committed to mitosis docs.
  • No downtime on the live platform; monitoring green across both nodes.

🤖 Filed by Claude Code as an agent handoff.

# Handoff: migrate product-garden off the DO droplet, then make it the platform's second node You are taking on a horizontal-scaling initiative for the mitosis platform (Forgejo-as-control-plane, open-platform.sh). **Use subagent teams / ultracode workflows** — this is multi-phase and parallelizable. Discover before you act; validate every phase; never break the live platform. ## The goal (two moves, in order) 1. **Test the migration feature at real scale** by migrating everything off the DigitalOcean droplet `pg-grand` (which currently runs the `product-garden` depth-2 QA platform) ONTO the main platform (open-platform.sh). Then decommission product-garden and free the droplet. 2. **Repurpose the freed droplet as a second cluster node**, so the platform stops being capped at one node's ~254 pods. Prove pods schedule across ≥2 nodes. This is step one of a real multi-machine cluster — design it so adding node 3, 4, … is a repeatable runbook, not a one-off. ## Known facts / custody (verify all of this first — don't trust it blindly) - **Main platform = `platgold2`**, a **k3s v1.31.5** cluster wrapped in **k3d** (Docker) on host **vxrail** (Tailscale `100.84.55.62`, `ssh vxrail`). Reach the API: `ssh vxrail 'docker exec k3d-platgold2-server-0 kubectl …'`. It is a **single node** (`k3d-platgold2-server-0`, control-plane+master), pod CIDR `10.42.0.0/24` → **~254 pod IP ceiling** (this is the wall we're removing). Edge = Cloudflare tunnel (`mitosis/tunnel-token`) + wildcard DNS + Let's Encrypt. GitOps = `plat/gitops` (Flux); cluster/node config lives there + in the k3d/k3s launch on vxrail. - **DO droplet `pg-grand`**: `167.172.29.178`, nyc3, s-4vcpu-8gb (4 vCPU / 8 GB), Ubuntu 24.04, active. Runs `product-garden` (a sovereign daughter platform; enforce + forwardAuth + cosign ON). SSH: try `~/.ssh/mitosis-do` as root (my keys got `publickey` denied — find the right key; the DO deploy key or a key in `mitosis/`). DO API token: `/Users/trevato/projects/mitosis/dg-token` (Bearer). `doctl`/curl the DO API for droplet/network control. - **Migration feature**: the platform migrates repos/apps via Forgejo **"New Migration"** → dispatcher `classifyMigration` → `bootstrapMigration` → a **migrator** agent run (agents-entity `src/dispatcher.ts:~649, ~1803`; `prompts/migrator.md`; mcp `create_app({migrate:true})`). Also relevant: `germinate`/daughter creation (`mitosis/bin/germinate`), and app decommission = remove the gitops overlay (Flux prunes; direct `kubectl delete ns` alone gets resurrected). - **Operator identity**: `trevato` (operator, owns product orgs) vs `plat` (admin). Local git credential for git.open-platform.sh authenticates as `plat` (site admin). See memory `openplatform-prod-custody`, `product-garden-qa-platform`, `mitosis-entity-workflows`, `yardwaze-client-platform` (a prior sovereign-deploy with the exact k3s + Cloudflare-tunnel topo). ## The hard architectural question (resolve in discovery, don't assume) platgold2's k3s API runs INSIDE a k3d Docker container on vxrail; k3d is designed for _local_ multi-node via Docker, not remote nodes. So "join the DO droplet as an agent" has two candidate shapes — evaluate both and pick with evidence: (A) **Add a remote k3s agent to the existing k3d-wrapped server** over Tailscale (put the droplet on the tailnet; expose/point the agent at the k3d server's API + node token; solve cross-WAN CNI — flannel backend, MTU, the k3d docker-bridge-vs-real-node networking). Least disruptive if it works; unusual and possibly fragile. (B) **Evolve off k3d to a real multi-node k3s** (server on vxrail bare/VM, agents on DO + future machines, flat network over Tailscale/WireGuard). Cleaner long-term, bigger lift, must migrate the live platform's state — high risk, needs a careful cutover plan. Recommend one with a reversibility/risk analysis before touching prod. Networking is the crux: the nodes need a flat, stable network (Tailscale mesh is the strong default; vxrail is already on it). ## Phasing (gate each phase on validation; keep the live platform healthy throughout) - **Phase 0 — Discovery (parallel team):** inventory product-garden on the droplet (orgs, repos, apps, DATA, users, edge/DNS); map platgold2's k3d/k3s launch + networking + gitops node config; spike the migration feature on ONE small app end-to-end; spike the node-join networking (A vs B) on a throwaway. Produce a topology decision + a migration inventory + a risk register. - **Phase 1 — Migrate product-garden → open-platform.sh:** drive the real migration per app/repo (Forgejo New Migration → migrator), **preserving data** (this is a QA platform but treat its data as real — snapshot/backup first). Validate each migrated app runs on the main platform. This is the migration-feature test the operator wants — capture what worked / what the feature is missing. - **Phase 2 — Decommission product-garden + free the droplet:** once migration is validated, tear down product-garden (gitops overlays + DNS + the droplet's k8s), snapshot the droplet, wipe to a clean Ubuntu base. Confirm nothing on open-platform.sh depended on it. - **Phase 3 — Join the droplet as a node + prove scale:** implement the chosen topology; put the droplet on the tailnet; join it; confirm `kubectl get nodes` shows it Ready and schedulable; cordon-drain-test; schedule real pods on it; confirm the ~254-pod wall is gone (aggregate capacity across nodes). Write the **repeatable add-a-node runbook** so machines 3+ are trivial. - **Phase 4 — Harden + document:** node labels/taints (keep control-plane workloads off the remote agent if flaky-WAN; put stateless app pods there), netpol/CNI across nodes, monitoring covers both nodes (the Grafana stack is live — extend node-exporter/scrape), gitops as source of truth for the node set. Update `mitosis` docs + memory. ## Guardrails (non-negotiable) - **The live platform (19 orgs actively building) must not go down.** No experiments on platgold2's networking without a rollback; spike on throwaway droplets/clusters first. - **Preserve data** through the migration — snapshot before you migrate or delete anything. - **Reversible steps only** on prod; every prod change gated on a validated spike. - **Security**: don't widen exposure — the tailnet is private; don't put the k3s API on the public internet. Respect the operator/admin split. Cross-node secrets stay sealed (sops/Flux). - Report progress on an issue; surface blockers as severity → impact → path. ## Acceptance - product-garden fully migrated to open-platform.sh (apps live + validated) OR a documented, operator-approved decision on what to migrate vs archive; droplet freed. - Migration-feature test written up: what worked, gaps, fixes filed on plat/agents / plat/mcp. - The droplet is a Ready, schedulable node; real pods run on it; aggregate pod capacity > one /24. - A repeatable "add a machine" runbook committed to `mitosis` docs. - No downtime on the live platform; monitoring green across both nodes. 🤖 Filed by Claude Code as an agent handoff.
plat commented 2026-07-17 05:44:40 +00:00
Author
Owner
Copy link

Progress report — Phases 0–1 (2026-07-17)

Phase 0 Discovery ✅

  • The pod cap is binding today: 232–240 of 250 max-pods running; CI runner pods FailedScheduling hourly ("Too many pods") while CPU/mem sit at 15%/8% of requests. Node 2 is the structural fix.
  • Topology decision: Option A (remote k3s agent joins the k3d-wrapped server over Tailscale). Evidence: API already reachable at the vxrail tailscale IP:36703; /etc/rancher/k3s/config.yaml inside the container is live-honored (max-pods precedent) so server changes need only a rehearsed container restart, no recreate; every networking change is host-level and reversible. Option B (re-platform to bare k3s) = 15–60min downtime + unsupported sqlite relocation → documented fallback only. Spike on throwaways (droplet spike-node-1 + k3d spikejoin) in flight; prod untouched until the runbook is validated.
  • Safety nets: DO snapshot pg-grand-pre-migration-20260716 + 27M granular archive (Forgejo DB, 8 app DBs pg_restore-verified, agents ledger, creds) in dual custody (Mac + vxrail).

Phase 1 Migration — in flight, recipe validated ✅

  • Sequencing (operator-approved): all repos migrate NOW (zero pod cost); app activation deferred until node 2 is Ready — create_app{migrate:true} provisions dormant overlays (not listed in apps kustomization), then release_app activates post-join. Forgejo-native end to end.
  • Spike (verdant/almanac): full fidelity (issues/PRs+merged flags/releases/branches), silent adopt-released, dormant overlay, pod count unchanged.
  • Batch (10 repos): verdant/{beds,tend} + trevato/to-dev with dormant create_app; permitflow twin (diverged from main's permitflow — same repo names) + qa-smoke + flask-hello + pg-gitops → archive org product-garden (mirror-parked where a live migrate would spawn a builder). beds/to-dev adopted + cockpit parked already; zero agent runs spawned.

Migration-feature bugs found (live)

  1. Domain-claim clobber: migrated repos carry the source website URL → runDomainClaim treats it as a BYOD claim, and findDomainClaimIssue filters by a label that doesn't exist on fresh repos — Forgejo returns ALL issues for a nonexistent label, so the dispatcher rewrote migrated issue #11 in place. Repaired byte-identical; batch recipe patches website immediately post-migrate. Fix needed: verify label exists / use the body marker.
  2. ensureAgentLabels TOCTOU → duplicate labels under concurrent sweeps.
  3. Non-platform-shaped migrated repos trigger the full autonomous migrator (create_app + agent spawn) — correct for foreign apps, dangerous for infra archives; mirror-park is the escape hatch.

Related throughput incident (filed + fix in review)

plat/agents#32 / plat/mcp#16: retry_build workflow-dispatches the pull_request-only preview workflow → guaranteed 7s failures → funnel + validator-gate poisoning → rework loops (55 poisoned runs / 4 orgs / 48h, ~$351 burned in 7.7h overnight, 36 PRs handed back to humans after credit exhaustion). Fix PRs (gate event-filter, funnel guard, dispatch guard + inference fix) are being implemented + adversarially verified now. Also filed plat/agents#33 (homestead crash-loop on missing ref).

Next

Batch completes → migration writeup; spike runbook → derelict bare-k3s decommission on vxrail (operator-approved) → Phase 2 teardown + wipe → Phase 3 join → Phase 4 harden + raise MAX_CONCURRENT 6→10 (ceiling stays $25/h).

🤖 Operator session (Claude Code), issue #101.

## Progress report — Phases 0–1 (2026-07-17) ### Phase 0 Discovery ✅ - **The pod cap is binding today**: 232–240 of 250 max-pods running; CI runner pods FailedScheduling hourly ("Too many pods") while CPU/mem sit at 15%/8% of requests. Node 2 is the structural fix. - **Topology decision: Option A** (remote k3s agent joins the k3d-wrapped server over Tailscale). Evidence: API already reachable at the vxrail tailscale IP:36703; `/etc/rancher/k3s/config.yaml` inside the container is live-honored (max-pods precedent) so server changes need only a rehearsed container restart, no recreate; every networking change is host-level and reversible. Option B (re-platform to bare k3s) = 15–60min downtime + unsupported sqlite relocation → documented fallback only. Spike on throwaways (droplet `spike-node-1` + k3d `spikejoin`) in flight; prod untouched until the runbook is validated. - **Safety nets**: DO snapshot `pg-grand-pre-migration-20260716` + 27M granular archive (Forgejo DB, 8 app DBs pg_restore-verified, agents ledger, creds) in dual custody (Mac + vxrail). ### Phase 1 Migration — in flight, recipe validated ✅ - **Sequencing (operator-approved)**: all repos migrate NOW (zero pod cost); app **activation deferred until node 2 is Ready** — `create_app{migrate:true}` provisions dormant overlays (not listed in apps kustomization), then `release_app` activates post-join. Forgejo-native end to end. - **Spike (verdant/almanac)**: full fidelity (issues/PRs+merged flags/releases/branches), silent `adopt-released`, dormant overlay, pod count unchanged. - **Batch (10 repos)**: verdant/{beds,tend} + trevato/to-dev with dormant create_app; permitflow twin (diverged from main's permitflow — same repo names) + qa-smoke + flask-hello + pg-gitops → archive org `product-garden` (mirror-parked where a live migrate would spawn a builder). beds/to-dev adopted + cockpit parked already; zero agent runs spawned. ### Migration-feature bugs found (live) 1. **Domain-claim clobber**: migrated repos carry the source `website` URL → `runDomainClaim` treats it as a BYOD claim, and `findDomainClaimIssue` filters by a label that doesn't exist on fresh repos — Forgejo returns ALL issues for a nonexistent label, so the dispatcher **rewrote migrated issue #11 in place**. Repaired byte-identical; batch recipe patches `website` immediately post-migrate. Fix needed: verify label exists / use the body marker. 2. **`ensureAgentLabels` TOCTOU** → duplicate labels under concurrent sweeps. 3. Non-platform-shaped migrated repos trigger the full autonomous migrator (create_app + agent spawn) — correct for foreign apps, dangerous for infra archives; mirror-park is the escape hatch. ### Related throughput incident (filed + fix in review) plat/agents#32 / plat/mcp#16: `retry_build` workflow-dispatches the `pull_request`-only preview workflow → guaranteed 7s failures → funnel + validator-gate poisoning → rework loops (55 poisoned runs / 4 orgs / 48h, ~$351 burned in 7.7h overnight, 36 PRs handed back to humans after credit exhaustion). Fix PRs (gate event-filter, funnel guard, dispatch guard + inference fix) are being implemented + adversarially verified now. Also filed plat/agents#33 (homestead crash-loop on missing ref). ### Next Batch completes → migration writeup; spike runbook → derelict bare-k3s decommission on vxrail (operator-approved) → Phase 2 teardown + wipe → Phase 3 join → Phase 4 harden + raise MAX_CONCURRENT 6→10 (ceiling stays $25/h). 🤖 Operator session (Claude Code), issue #101.
plat commented 2026-07-17 06:23:06 +00:00
Author
Owner
Copy link

Progress — Phase 1 COMPLETE, node-join spike VALIDATED, fleet fix SHIPPED (2026-07-17)

Phase 1 ✅

All 11 repos migrated, zero agent runs spawned, pod count untouched (236). Activation list (verdant/{almanac,beds,tend}, trevato/to-dev): full fidelity (issues/PRs/merged-flags/releases byte-checked) + dormant create_app{migrate:true} overlays — they activate after node 2 joins. Archive org product-garden: qa-smoke + flask-hello (full migrations, adopt-released; the 112MB repo took 6.8s), permitflow×5 + gitops as read-only mirrors (Forgejo silently drops issues/PRs/releases on mirror:true — git-only; issue history preserved in the Forgejo DB dump; ⚠️ mirrors must never be converted to regular — they'd classify fresh and spawn builders). Test report committed: docs/migration-feature-test.md.

Node-join spike ✅ — Option A validated end-to-end on throwaways

Runbook committed: docs/add-a-node.md. Highlights: agent Ready in ~35s over Tailscale; one rehearsed server restart (~3.3s API outage, all pods replaced in ~15s — k3d drains on stop, so the restart happens BEFORE the first agent joins); advertise-address/port + flannel public-ip-overwrite (NOT node-external-ip, which routes the cluster into the derelict bare k3s on host 6443); vxlan checksum-offload fix required after each server restart; exactly 2 host iptables rules, with rollback. Cross-node pod/service/DNS/logs/exec/reboot all validated; spike fully cleaned up.

Fleet fix shipped (from the codewear/catalog#9 investigation)

agents 0.20.1 + mcp 0.32.2 are LIVE (plat/agents#34, plat/mcp#17 — closes #32/#16): the preview verdict gate ignores dispatch-event runs, the funnel no longer re-arms on self-inflicted dispatch failures, and retry_build refuses non-dispatchable workflows with a useful hint. codewear/catalog#9 has been nudged back into the pipeline. Still open: #33 (homestead crash-loop), #35 (sweep noise on parked mirrors).

In flight

Phase-2 readiness checks (mirror sync disabling, dependency sweep, GO/NO-GO for the droplet wipe) and the operator-approved derelict-bare-k3s decommission on vxrail are running now. Then: wipe → tailnet → join per runbook → activate the migrated apps on node 2 → raise MAX_CONCURRENT 6→10.

🤖 Operator session (Claude Code), issue #101.

## Progress — Phase 1 COMPLETE, node-join spike VALIDATED, fleet fix SHIPPED (2026-07-17) ### Phase 1 ✅ All 11 repos migrated, **zero agent runs spawned, pod count untouched (236)**. Activation list (verdant/{almanac,beds,tend}, trevato/to-dev): full fidelity (issues/PRs/merged-flags/releases byte-checked) + dormant `create_app{migrate:true}` overlays — they activate after node 2 joins. Archive org `product-garden`: qa-smoke + flask-hello (full migrations, adopt-released; the 112MB repo took 6.8s), permitflow×5 + gitops as read-only mirrors (Forgejo silently drops issues/PRs/releases on `mirror:true` — git-only; issue history preserved in the Forgejo DB dump; ⚠️ mirrors must never be converted to regular — they'd classify `fresh` and spawn builders). Test report committed: `docs/migration-feature-test.md`. ### Node-join spike ✅ — Option A validated end-to-end on throwaways Runbook committed: `docs/add-a-node.md`. Highlights: agent Ready in ~35s over Tailscale; one rehearsed server restart (**~3.3s API outage, all pods replaced in ~15s** — k3d drains on stop, so the restart happens BEFORE the first agent joins); `advertise-address/port` + flannel `public-ip-overwrite` (NOT `node-external-ip`, which routes the cluster into the derelict bare k3s on host 6443); vxlan checksum-offload fix required after each server restart; exactly 2 host iptables rules, with rollback. Cross-node pod/service/DNS/logs/exec/reboot all validated; spike fully cleaned up. ### Fleet fix shipped (from the codewear/catalog#9 investigation) `agents 0.20.1` + `mcp 0.32.2` are LIVE (plat/agents#34, plat/mcp#17 — closes #32/#16): the preview verdict gate ignores dispatch-event runs, the funnel no longer re-arms on self-inflicted dispatch failures, and `retry_build` refuses non-dispatchable workflows with a useful hint. codewear/catalog#9 has been nudged back into the pipeline. Still open: #33 (homestead crash-loop), #35 (sweep noise on parked mirrors). ### In flight Phase-2 readiness checks (mirror sync disabling, dependency sweep, GO/NO-GO for the droplet wipe) and the operator-approved derelict-bare-k3s decommission on vxrail are running now. Then: wipe → tailnet → join per runbook → activate the migrated apps on node 2 → raise MAX_CONCURRENT 6→10. 🤖 Operator session (Claude Code), issue #101.
plat commented 2026-07-17 06:50:04 +00:00
Author
Owner
Copy link

Phases 2–4: droplet freed → node 2 LIVE → hardened (2026-07-17)

Phase 2 ✅ — product-garden decommissioned, droplet wiped

Readiness pass GO: mirror sync disabled on all 6 archive mirrors, dependency sweep clean (nothing on main references product-garden.com), archives re-verified in dual custody. legacy-source/legacy-pg (the one bare pod that a server restart would have deleted) converted to a Deployment + data restored first. Droplet rebuilt to clean Ubuntu 24.04 (snapshot pg-grand-pre-migration-20260716 retained). Derelict bare k3s v1.34.3 on vxrail decommissioned (port 6443 + 10.42 host route freed).

Phase 3 ✅ — pg-grand is node 2, the pod wall is GONE

Joined per the validated runbook (Option A, k3s agent over Tailscale). kubectl get nodes: pg-grand Ready, podCIDR 10.42.1.0/24, own max-pods 110 → aggregate 360 pods across two /24s (was a hard 250). One server restart, measured ~3.3s API outage, all pods recovered. Validated live: 4 pods scheduled onto pg-grand; cross-node pod↔pod both directions; 1MB payload transfers clean (no MTU blackhole); ClusterIP + CoreDNS from node 2; UdpInCsumErrors flat (checksum-offload fix holding). Real platform pods now run on node 2.

Phase 4 ✅ — hardened + durable

  • Node labels: pg-grand = node-role=app-worker; new app pods spread there by least-allocated, relieving the server.
  • Monitoring: node-exporter DaemonSet auto-scheduled onto node 2; Prometheus scrapes it — zero extra wiring.
  • Durability: the host rules + flannel checksum fix are runtime-only, and ⚠️ vxrail's ~/open-platform flake is stale/divergent (would resurrect the derelict k3s) — so node config is imperative-by-design. A self-healing idempotent keeper (vxrail:~/platgold2-node-keeper.sh) reasserts both after reboot/server-restart. Documented in docs/add-a-node.md (repeatable — machines 3+ need zero server-side changes).
  • Throughput lever: agents maxConcurrent 6→10 (gitops), $25/h per-org ceiling unchanged as the governor.

Acceptance status

✅ product-garden migrated + droplet freed · ✅ migration-feature test written (docs/migration-feature-test.md, bugs filed, 2 fixed) · ✅ droplet is a Ready schedulable node, real pods on it, capacity > one /24 · ✅ repeatable add-a-node runbook committed · ✅ no downtime, monitoring green across both nodes.

App activation (verdant/{almanac,beds,tend} + trevato/to-dev with data restore) is finishing now — to-dev already activated + promoted to prod.

🤖 Operator session (Claude Code), issue #101.

## Phases 2–4: droplet freed → node 2 LIVE → hardened (2026-07-17) ### Phase 2 ✅ — product-garden decommissioned, droplet wiped Readiness pass GO: mirror sync disabled on all 6 archive mirrors, dependency sweep clean (nothing on main references product-garden.com), archives re-verified in dual custody. `legacy-source/legacy-pg` (the one bare pod that a server restart would have deleted) converted to a Deployment + data restored first. Droplet **rebuilt to clean Ubuntu 24.04** (snapshot `pg-grand-pre-migration-20260716` retained). Derelict bare k3s v1.34.3 on vxrail decommissioned (port 6443 + 10.42 host route freed). ### Phase 3 ✅ — pg-grand is node 2, the pod wall is GONE Joined per the validated runbook (Option A, k3s agent over Tailscale). `kubectl get nodes`: **pg-grand Ready**, podCIDR `10.42.1.0/24`, own max-pods 110 → **aggregate 360 pods across two /24s** (was a hard 250). One server restart, measured ~3.3s API outage, all pods recovered. Validated live: 4 pods scheduled onto pg-grand; cross-node pod↔pod both directions; 1MB payload transfers clean (no MTU blackhole); ClusterIP + CoreDNS from node 2; UdpInCsumErrors flat (checksum-offload fix holding). Real platform pods now run on node 2. ### Phase 4 ✅ — hardened + durable - **Node labels**: pg-grand = `node-role=app-worker`; new app pods spread there by least-allocated, relieving the server. - **Monitoring**: node-exporter DaemonSet auto-scheduled onto node 2; Prometheus scrapes it — zero extra wiring. - **Durability**: the host rules + flannel checksum fix are runtime-only, and ⚠️ vxrail's `~/open-platform` flake is stale/divergent (would resurrect the derelict k3s) — so node config is imperative-by-design. A self-healing idempotent keeper (`vxrail:~/platgold2-node-keeper.sh`) reasserts both after reboot/server-restart. Documented in `docs/add-a-node.md` (repeatable — machines 3+ need zero server-side changes). - **Throughput lever**: `agents maxConcurrent 6→10` (gitops), $25/h per-org ceiling unchanged as the governor. ### Acceptance status ✅ product-garden migrated + droplet freed · ✅ migration-feature test written (`docs/migration-feature-test.md`, bugs filed, 2 fixed) · ✅ droplet is a Ready schedulable node, real pods on it, capacity > one /24 · ✅ repeatable add-a-node runbook committed · ✅ no downtime, monitoring green across both nodes. App activation (verdant/{almanac,beds,tend} + trevato/to-dev with data restore) is finishing now — to-dev already activated + promoted to prod. 🤖 Operator session (Claude Code), issue #101.
plat commented 2026-07-17 07:04:31 +00:00
Author
Owner
Copy link

✅ Issue complete — horizontal scaling delivered, all acceptance met

Acceptance checklist

  • ✅ product-garden migrated + droplet freed. 11 repos → open-platform.sh (4 activation apps + archive org product-garden); droplet rebuilt to clean Ubuntu (snapshot retained).
  • ✅ Apps live + data preserved. verdant/{almanac,beds,tend} + trevato/to-dev activated (release_app, registered in gitops kustomization) and DB-restored: to-dev 7 todos/1 user · almanac 25 crops · beds 22 plantings/2 beds · tend 1 user — every count matches the source inventory; all HTTP 200, clean logs.
  • ✅ Migration-feature test written. docs/migration-feature-test.md; bugs filed (domain-claim clobber, mirror content-flag drop, TOCTOU labels, sweep noise #35); the preview-poisoning loop fixed + shipped (agents 0.20.1 / mcp 0.32.2).
  • ✅ Droplet is a Ready, schedulable node; capacity > one /24. pg-grand = node 2, podCIDR 10.42.1.0/24, aggregate 360 pods across two nodes (was a hard 250). Real app pods run on it (5 of the 8 node-2 pods are migrated workloads). Cross-node pod/svc/DNS validated both directions; 1MB payloads clean; UdpInCsumErrors flat.
  • ✅ Repeatable add-a-node runbook. docs/add-a-node.md — Option A (k3s agent over Tailscale), machines 3+ need zero server-side changes.
  • ✅ No downtime; monitoring green both nodes. node-exporter DaemonSet auto-covers node 2; Prometheus scrapes it.

Throughput

agents maxConcurrent 6→10 (node 2 absorbs the extra runs); $25/h per-org ceiling unchanged as the governor.

Follow-ups filed / flagged

  • plat/gitops#21 — Kyverno apiserver-egress not synced to 8 pre-existing namespaces (hotfixed live; needs generateExisting/synchronize).
  • plat/agents#33 (homestead missing-ref crash-loop), #35 (sweep noise on parked mirrors).
  • Placement optimization (below) — a nodeAffinity to steer new app/CNPG pods toward node-role=app-worker would make server relief deliberate rather than incidental.
  • Operator-only: (1) Cloudflare — delete tunnel cbfa9ed4-… + product-garden.com DNS. (2) ⚠️ Remove services.k3s from the vxrail NixOS flake — the decommissioned derelict k3s is blocked at runtime but a reboot/nixos-rebuild resurrects it; the same reboot fragility is why node 2's ~/platgold2-node-keeper.sh needs one manual run post-reboot.

🤖 Operator session (Claude Code). Closing as delivered.

## ✅ Issue complete — horizontal scaling delivered, all acceptance met ### Acceptance checklist - ✅ **product-garden migrated + droplet freed.** 11 repos → open-platform.sh (4 activation apps + archive org `product-garden`); droplet rebuilt to clean Ubuntu (snapshot retained). - ✅ **Apps live + data preserved.** verdant/{almanac,beds,tend} + trevato/to-dev activated (release_app, registered in gitops kustomization) and DB-restored: **to-dev 7 todos/1 user · almanac 25 crops · beds 22 plantings/2 beds · tend 1 user** — every count matches the source inventory; all HTTP 200, clean logs. - ✅ **Migration-feature test written.** `docs/migration-feature-test.md`; bugs filed (domain-claim clobber, mirror content-flag drop, TOCTOU labels, sweep noise #35); the preview-poisoning loop fixed + shipped (agents 0.20.1 / mcp 0.32.2). - ✅ **Droplet is a Ready, schedulable node; capacity > one /24.** pg-grand = node 2, podCIDR 10.42.1.0/24, aggregate **360 pods across two nodes** (was a hard 250). Real app pods run on it (5 of the 8 node-2 pods are migrated workloads). Cross-node pod/svc/DNS validated both directions; 1MB payloads clean; UdpInCsumErrors flat. - ✅ **Repeatable add-a-node runbook.** `docs/add-a-node.md` — Option A (k3s agent over Tailscale), machines 3+ need zero server-side changes. - ✅ **No downtime; monitoring green both nodes.** node-exporter DaemonSet auto-covers node 2; Prometheus scrapes it. ### Throughput `agents maxConcurrent 6→10` (node 2 absorbs the extra runs); $25/h per-org ceiling unchanged as the governor. ### Follow-ups filed / flagged - plat/gitops#21 — Kyverno apiserver-egress not synced to 8 pre-existing namespaces (hotfixed live; needs generateExisting/synchronize). - plat/agents#33 (homestead missing-ref crash-loop), #35 (sweep noise on parked mirrors). - Placement optimization (below) — a nodeAffinity to steer new app/CNPG pods toward `node-role=app-worker` would make server relief deliberate rather than incidental. - **Operator-only:** (1) Cloudflare — delete tunnel `cbfa9ed4-…` + product-garden.com DNS. (2) ⚠️ Remove `services.k3s` from the vxrail NixOS flake — the decommissioned derelict k3s is blocked at runtime but a reboot/nixos-rebuild resurrects it; the same reboot fragility is why node 2's `~/platgold2-node-keeper.sh` needs one manual run post-reboot. 🤖 Operator session (Claude Code). Closing as delivered.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
reseed/r1038
proposal-the-wardrobe
seed-embedded-etcd
docs-app-roles
docs/aws-eks-build-history
audit/aws-2026-08-25
azure-docs-truth-pass
aks-coredns-hairpin-fix
fix/aks-ground-trust-cidrs
fix/aks-cert-zone-doubling
fix/azure-gate-nixos-bicep
feat/azure-gate
azure-aks-mode
feat/deploy-platform
feat/seed-release
seed/packages-integration-spec
feat/seed-workflow
resilience/standby-site
resilience/backup-born-on
fix/germinate-timeout-contention
feat/reseed-noop-guard
fix/reseed-setg
fix/reseed-script
fix/reseed-yaml
feat/reseed-workflow
fix/seed-createdfrom-durable
feat/gate-parallel-nightly
fix/gate-a4-detection
feat/fork-on-germinate
feat/ci-1-coldstart-gate
chore/reseed-genesis-post-61
fix/germinate-prerelease-tags
feat/wb-gate-seed-refresh
feat/dom-2-cilium-fqdn-egress
feat/rbac-5-webhook-hmac-seal
feat/robust-seed-refresh
feat/rbac-7-postgres-rls-design
feat/sec-3-secrets-threat-model
feat/dom-1a-fork-reseal-cloudflare-token
feat/sec-1-age-key-rotation
fix/germinate-python3-runtimedep
docs/pol-0-policy-governance
docs/hyb-3-opentofu-substrate
chore/refresh-genesis-seed
docs/enterprise-platform
docs/design-proposals
feat/observability-grafana
fix/cold-start-seed-branch-and-mtu
seed-r897
seed-r886
seed-r873
seed-r829
Labels
Clear labels   
bug
Something is not working

  
discussion
An idea that is still being discussed

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
goal

   
help wanted
Need some help

  
horizon:backlog

   
horizon:governance

   
horizon:mvp

   
invalid
Something is wrong

  
operator-decision
Blocked on the owner's call — no agent work until decided

  
question
More information is needed

  
roadmap

   
wontfix
This won't be fixed

No labels
bug
discussion
duplicate
enhancement
goal
help wanted
horizon:backlog
horizon:governance
horizon:mvp
invalid
operator-decision
question
roadmap
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
open-platform/mitosis#101
No description provided.