open-platform/mitosis
3
0
Fork 0
Code Issues 27 Pull requests 9 Releases 4 Packages 1 Wiki Activity Actions

AWS deployment audit — EC2 leader + EKS backlog (34 items) #212

Closed
benjivers wants to merge 2 commits from audit/aws-2026-08-25 into main
pull from: audit/aws-2026-08-25
merge into: open-platform:main
Conversation 3 Commits 2 Files changed 1 +340
benjivers commented 2026-08-25 19:07:42 +00:00
Collaborator
Copy link

What this is

A consolidated audit of everything AWS in the mitosis platform, requested 2026-08-25. Adds one doc — docs/ops/aws-audit-2026-08-25.md — a prioritized, checkbox-able backlog. No code changes; this is the list of issues + concrete changes to work through.

Scope swept

  • deploy/aws/ — single-node EC2 leader stack → git.benjivers.com (k3s, Free-Plan account)
  • deploy/eks/ — managed-EKS stack → git.aws.benjivers.com (billed control plane, us-east-1)
  • EKS-mode germinate wiring — bin/germinate + bin/lib.sh on feat/2vcpu-daughter-fixes (incl. uncommitted working-tree changes)
  • Live cluster state — pulled from the running git.aws.benjivers.com cluster

Headline findings

4 critical

  • C1 — the entire deploy/eks/ stack (behind a live, billing cluster) is untracked in git; exists only on one laptop. (Verified safe to commit — deploy/.gitignore already excludes state/tfvars.)
  • C2 — EKS Terraform state is local-only (no S3 backend) and holds plaintext secrets; laptop loss = unrecoverable orphaned billing.
  • C3 + C4 — the EKS germinate TLS path is half-landed: PLATFORM_EDGE_ISSUER support is an unapplied patch, the shipped seed predates it, and certs point at a letsencrypt-dns01 issuer that is never created. An eks germination today comes up with untrusted certs on every browser-facing edge.

Live cluster (running now)

  • L1 — open-platform/www prod is stuck (HelmRelease Progressing, 0 pods) on the AWS cluster; fleet/www is healthy.
  • L2 — PLAT_FORWARD_AUTH unset → every app Ingress trips the plat-access coverage policy.
  • L3 — the live git.benjivers.com EC2 leader couldn't be inspected (op-benjivers MCP needs OAuth) — action for you to authorize so it can be swept.

12 high — open EKS API endpoint (0.0.0.0/0, taken by the live cluster); credit-guard that fails open and only brakes compute (control plane keeps billing); down.sh that can leave a billable NLB + Route53 zone orphaned; EC2 SSH default 0.0.0.0/0; no cost guardrails; unencrypted root EBS with no backups holding the sovereign key; single leader with no prevent_destroy + destructive user-data replacement; plus germinate ECR/kyverno-mirror/external-dns gaps.

12 medium + 6 low — full detail, file:line, and the concrete change for each in the doc, plus a "do not fix — already correct" section and a suggested landing order.

Note

The doc is deliberately a checklist so items can be ticked off (or split into issues) as they land. Nothing here is auto-fixed — flag which items you want turned into PRs and I'll take them in the suggested order (C1/C2 first).

🤖 Generated with Claude Code

## What this is A consolidated audit of **everything AWS** in the mitosis platform, requested 2026-08-25. Adds one doc — `docs/ops/aws-audit-2026-08-25.md` — a prioritized, checkbox-able backlog. No code changes; this is the list of issues + concrete changes to work through. ## Scope swept - **`deploy/aws/`** — single-node EC2 leader stack → `git.benjivers.com` (k3s, Free-Plan account) - **`deploy/eks/`** — managed-EKS stack → `git.aws.benjivers.com` (billed control plane, us-east-1) - **EKS-mode germinate wiring** — `bin/germinate` + `bin/lib.sh` on `feat/2vcpu-daughter-fixes` (incl. uncommitted working-tree changes) - **Live cluster state** — pulled from the running `git.aws.benjivers.com` cluster ## Headline findings **4 critical** - **C1** — the entire `deploy/eks/` stack (behind a *live, billing* cluster) is **untracked in git**; exists only on one laptop. (Verified safe to commit — `deploy/.gitignore` already excludes state/tfvars.) - **C2** — EKS Terraform state is **local-only** (no S3 backend) and holds plaintext secrets; laptop loss = unrecoverable orphaned billing. - **C3 + C4** — the EKS germinate **TLS path is half-landed**: `PLATFORM_EDGE_ISSUER` support is an unapplied patch, the shipped seed predates it, and certs point at a `letsencrypt-dns01` issuer that is **never created**. An eks germination today comes up with untrusted certs on every browser-facing edge. **Live cluster (running now)** - **L1** — `open-platform/www` **prod is stuck** (HelmRelease `Progressing`, 0 pods) on the AWS cluster; `fleet/www` is healthy. - **L2** — `PLAT_FORWARD_AUTH` unset → every app Ingress trips the `plat-access` coverage policy. - **L3** — the live `git.benjivers.com` EC2 leader couldn't be inspected (op-benjivers MCP needs OAuth) — **action for you** to authorize so it can be swept. **12 high** — open EKS API endpoint (`0.0.0.0/0`, taken by the live cluster); credit-guard that **fails open** and only brakes compute (control plane keeps billing); `down.sh` that can leave a billable NLB + Route53 zone orphaned; EC2 SSH default `0.0.0.0/0`; no cost guardrails; unencrypted root EBS with no backups holding the sovereign key; single leader with no `prevent_destroy` + destructive user-data replacement; plus germinate ECR/kyverno-mirror/external-dns gaps. **12 medium + 6 low** — full detail, file:line, and the concrete change for each in the doc, plus a "do not fix — already correct" section and a suggested landing order. ## Note The doc is deliberately a checklist so items can be ticked off (or split into issues) as they land. Nothing here is auto-fixed — flag which items you want turned into PRs and I'll take them in the suggested order (C1/C2 first). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
docs(ops): AWS deployment audit — EC2 leader + EKS backlog
All checks were successful
guard-ppt / no-ppt (push) Successful in 42s
Details
guard-ppt / no-ppt (pull_request) Successful in 39s
Details
1067fc8484 
Consolidated audit of everything AWS: the single-node EC2 leader stack
(deploy/aws/, git.benjivers.com), the managed-EKS stack (deploy/eks/,
git.aws.benjivers.com), the EKS-mode germinate wiring on
feat/2vcpu-daughter-fixes, and live cluster state. 4 crit, 12 high, 12 med,
6 low + 3 live-cluster items, with concrete changes and a landing order.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docs(ops): fill in live EC2-leader sweep (op-benjivers authorized)
All checks were successful
guard-ppt / no-ppt (push) Successful in 1m22s
Details
guard-ppt / no-ppt (pull_request) Successful in 1m17s
Details
acf7e7b1a8 
git.benjivers.com swept 2026-08-25: 9 apps, fleet/{www,relay,review} +
invoicing/fleet-invoice healthy; fleet/insights prod wedged UpgradeFailed on
a stale ecr-cred ECR pull secret. Both clusters confirmed PLAT_FORWARD_AUTH-off.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
benjivers commented 2026-08-25 19:22:54 +00:00
Author
Collaborator
Copy link

L3 resolved — live EC2 leader (git.benjivers.com) swept

op-benjivers is now authorized, so the one gap in the original audit is filled. Pushed to the branch (acf7e7b).

9 apps on the leader — fleet/{www,relay,review,insights} + invoicing/fleet-invoice, all pulling from ECR (<acct>.dkr.ecr.us-east-1.amazonaws.com).

App Prod status
fleet/www ✅ healthy (v3)
fleet/relay ✅ healthy (v2)
fleet/review ✅ healthy (v4)
invoicing/fleet-invoice ✅ healthy (v23, app + worker)
fleet/insights 🔴 HelmRelease UpgradeFailed — stalled Deployment + recurring FailedToRetrieveImagePullSecret (ecr-cred)

New live finding (now L3 in the doc): fleet/insights prod is wedged because the ecr-cred image-pull secret can't be retrieved. The leader pulls everything from ECR and ECR auth tokens expire ~12h, so a static platform/registry-pull-secret.yaml goes stale without a refresher. The pod happens to still be Running (0 restarts) but the release can't complete and a reschedule would fail to pull. Fix: repair/add the ECR-token refresh and roll the release — bumped to step 0 (operational hotfix) in the landing order since it's live right now. Related to germinate H9 (ECR wiring).

PLAT_FORWARD_AUTH-off (L2) is confirmed on both clusters. Everything else in the audit stands.

## L3 resolved — live EC2 leader (`git.benjivers.com`) swept `op-benjivers` is now authorized, so the one gap in the original audit is filled. Pushed to the branch (`acf7e7b`). **9 apps on the leader** — `fleet/{www,relay,review,insights}` + `invoicing/fleet-invoice`, all pulling from ECR (`<acct>.dkr.ecr.us-east-1.amazonaws.com`). | App | Prod status | |---|---| | fleet/www | ✅ healthy (v3) | | fleet/relay | ✅ healthy (v2) | | fleet/review | ✅ healthy (v4) | | invoicing/fleet-invoice | ✅ healthy (v23, app + worker) | | **fleet/insights** | 🔴 HelmRelease `UpgradeFailed` — stalled Deployment + recurring `FailedToRetrieveImagePullSecret (ecr-cred)` | **New live finding (now L3 in the doc):** `fleet/insights` prod is wedged because the `ecr-cred` image-pull secret can't be retrieved. The leader pulls everything from ECR and ECR auth tokens expire ~12h, so a static `platform/registry-pull-secret.yaml` goes stale without a refresher. The pod happens to still be Running (0 restarts) but the release can't complete and a reschedule would fail to pull. **Fix:** repair/add the ECR-token refresh and roll the release — bumped to step 0 (operational hotfix) in the landing order since it's live right now. Related to germinate **H9** (ECR wiring). `PLAT_FORWARD_AUTH`-off (L2) is confirmed on **both** clusters. Everything else in the audit stands.
benjivers commented 2026-08-25 19:40:20 +00:00
Author
Collaborator
Copy link

Filed the two live findings as tracking issues (both root-cause to the daughter-cluster gitops, not this repo's source — the ecr-cred mechanism isn't in mitosis source and the leader pulls from ECR off the default k3s path):

  • L3 → #215 — fleet/insights wedged on stale ecr-cred (git.benjivers.com leader); fix = ECR-token refresher CronJob in that cluster's plat/gitops.
  • L1 → #216 — open-platform/www stuck HelmRelease (git.aws.benjivers.com EKS); fix = pull flux controller logs + reconcile/rollback on that cluster.

Each issue also notes the follow-up plat/mitosis-source question (support AWS/ECR k3s as a first-class profile w/ refresher; HelmRelease install-retry hardening).

Filed the two **live** findings as tracking issues (both root-cause to the daughter-cluster gitops, not this repo's source — the `ecr-cred` mechanism isn't in mitosis source and the leader pulls from ECR off the default k3s path): - **L3** → #215 — `fleet/insights` wedged on stale `ecr-cred` (git.benjivers.com leader); fix = ECR-token refresher CronJob in that cluster's `plat/gitops`. - **L1** → #216 — `open-platform/www` stuck HelmRelease (git.aws.benjivers.com EKS); fix = pull flux controller logs + reconcile/rollback on that cluster. Each issue also notes the follow-up plat/mitosis-source question (support AWS/ECR k3s as a first-class profile w/ refresher; HelmRelease install-retry hardening).
benjivers commented 2026-08-25 20:37:20 +00:00
Author
Collaborator
Copy link

Closing per Ben — the current-state audit isn't the artifact wanted. The build-process history (how AWS/AKS/EKS actually got built) supersedes this in #221. The two live app-status items surfaced here remain tracked as #215 (ecr-cred stale) and #216 (open-platform/www stuck) if they're ever needed.

Closing per Ben — the current-state audit isn't the artifact wanted. The build-process history (how AWS/AKS/EKS actually got built) supersedes this in **#221**. The two live app-status items surfaced here remain tracked as **#215** (ecr-cred stale) and **#216** (open-platform/www stuck) if they're ever needed.
benjivers closed this pull request 2026-08-25 20:37:21 +00:00
All checks were successful
guard-ppt / no-ppt (push) Successful in 1m22s
Required
Details
guard-ppt / no-ppt (pull_request) Successful in 1m17s
Required
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
Something is not working

  
discussion
An idea that is still being discussed

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
goal

   
help wanted
Need some help

  
horizon:backlog

   
horizon:governance

   
horizon:mvp

   
invalid
Something is wrong

  
operator-decision
Blocked on the owner's call — no agent work until decided

  
question
More information is needed

  
roadmap

   
wontfix
This won't be fixed

No labels
bug
discussion
duplicate
enhancement
goal
help wanted
horizon:backlog
horizon:governance
horizon:mvp
invalid
operator-decision
question
roadmap
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
open-platform/mitosis!212
No description provided.