Live: open-platform/www prod stuck (HelmRelease Progressing, 0 pods) on git.aws.benjivers.com EKS #216

New issue

Closed

opened 2026-08-25 19:40:07 +00:00 by benjivers · 0 comments

benjivers commented

2026-08-25 19:40:07 +00:00

Collaborator

Found during the AWS audit (PR #212, finding L1). This is a live issue on the git.aws.benjivers.com EKS cluster.

Symptom

open-platform/www prod:

helmrelease.ready = false   reason = Progressing   ("reconciliation in progress")
deployment.found  = false
pods              = []      (zero)
chart www@0.1.1+77386226b940

The HelmRelease is stuck reconciling and never rendered a Deployment — no pods at all. fleet/www prod on the same cluster is healthy (InstallSucceeded, 1/1 running), so this is app/release-specific, not a cluster-wide outage.

Notes toward root cause

"Deployment not found" + long-running Progressing means the install is blocked before the Deployment is created — typical causes: a Helm dependency/wait, a values/template error in the www chart, or an admission (kyverno) block during render. Needs the flux helm-controller / source-controller logs on the EKS cluster to pin down (app has 0 pods, so tail_logs returns nothing).
Distinct from L3: on EKS, image pulls use the node ECR role, not an ecr-cred secret, so this is not the same failure mode.
The EKS TLS path is also currently half-landed (see PR #212 C3/C4) — if www's chart waits on a cert/Ingress readiness, that could stall the release; worth checking whether the two are related.

Fix (lands on the EKS cluster, tracked here)

Pull helm-controller/source-controller logs for open-platform--www--prod on git.aws.benjivers.com and identify the blocking resource.
Reconcile or roll back the HelmRelease; if it's a chart/values defect, fix in the open-platform/www app repo on that host.
Feed anything generalizable (e.g. install-retry / remediation on HelmRelease) back as a real plat/mitosis source hardening if warranted.

Ref: audit PR #212 → docs/ops/aws-audit-2026-08-25.md (L1).

Found during the AWS audit (PR #212, finding **L1**). This is a **live** issue on the `git.aws.benjivers.com` EKS cluster. ## Symptom `open-platform/www` **prod**: ``` helmrelease.ready = false reason = Progressing ("reconciliation in progress") deployment.found = false pods = [] (zero) chart www@0.1.1+77386226b940 ``` The HelmRelease is stuck reconciling and **never rendered a Deployment** — no pods at all. `fleet/www` prod on the *same* cluster is healthy (`InstallSucceeded`, 1/1 running), so this is app/release-specific, not a cluster-wide outage. ## Notes toward root cause - "Deployment not found" + long-running `Progressing` means the install is blocked **before** the Deployment is created — typical causes: a Helm dependency/wait, a values/template error in the `www` chart, or an admission (kyverno) block during render. Needs the flux `helm-controller` / `source-controller` logs on the EKS cluster to pin down (app has 0 pods, so `tail_logs` returns nothing). - Distinct from L3: on EKS, image pulls use the **node ECR role**, not an `ecr-cred` secret, so this is not the same failure mode. - The EKS TLS path is also currently half-landed (see PR #212 C3/C4) — if `www`'s chart waits on a cert/Ingress readiness, that could stall the release; worth checking whether the two are related. ## Fix (lands on the EKS cluster, tracked here) 1. Pull `helm-controller`/`source-controller` logs for `open-platform--www--prod` on `git.aws.benjivers.com` and identify the blocking resource. 2. Reconcile or roll back the HelmRelease; if it's a chart/values defect, fix in the `open-platform/www` app repo on that host. 3. Feed anything generalizable (e.g. install-retry / `remediation` on HelmRelease) back as a real plat/mitosis source hardening if warranted. Ref: audit PR #212 → `docs/ops/aws-audit-2026-08-25.md` (L1).

benjivers added the

bug

label

2026-08-25 19:40:07 +00:00

benjivers referenced this issue

2026-08-25 19:40:20 +00:00

AWS deployment audit — EC2 leader + EKS backlog (34 items) #212

benjivers referenced this issue

2026-08-25 20:37:20 +00:00

AWS deployment audit — EC2 leader + EKS backlog (34 items) #212

benjivers closed this issue

2026-08-25 20:40:46 +00:00