Live: fleet/insights prod wedged on stale `ecr-cred` pull secret (git.benjivers.com leader) #215

New issue

Closed

opened 2026-08-25 19:39:57 +00:00 by benjivers · 0 comments

benjivers commented

2026-08-25 19:39:57 +00:00

Collaborator

Found during the AWS audit (PR #212, finding L3). This is a live issue on the git.benjivers.com EC2 k3s leader.

Symptom

fleet/insights prod HelmRelease is UpgradeFailed:

failed early due to stalled resources:
[Deployment/fleet--insights--prod/insights status: 'Failed']

with a recurring pod Warning:

FailedToRetrieveImagePullSecret: Unable to retrieve some image pull secrets (ecr-cred);
attempting to pull the image may not succeed.

The pod happens to still be Running (0 restarts) on the already-pulled image, but the release can't complete and any reschedule would fail to pull. fleet/{www,relay,review} and invoicing/fleet-invoice are healthy — so it's this release/secret, not cluster-wide.

Root cause

Every app on the leader pulls from ECR (<acct>.dkr.ecr.us-east-1.amazonaws.com/...) using an imagePullSecrets: [ecr-cred].
ECR auth tokens expire ~12h. There is no live refresher keeping ecr-cred current, so the secret goes stale / missing and the pull-secret lookup fails.

Why this is NOT a plat/mitosis source change

The string ecr-cred does not exist anywhere in the mitosis source (grep -rn ecr-cred bin/ deploy/ → nothing).
Mitosis' AWS design is node-IAM-ECR-role pulls, no pull secret (bin/germinate:71-72, :813 — "kubelet pulls via the node's ECR role"). Default k3s (what this leader runs) is meant to pull from the in-cluster Forgejo registry via the 127.0.0.1:31100 mirror (registry-node-config), not ECR.
So the leader was wired to ECR + ecr-cred out-of-band; that mechanism lives in the leader's own gitops (plat/gitops on git.benjivers.com), not here.

Fix (lands in the daughter gitops, tracked here)

On git.benjivers.com plat/gitops, add (or repair) an ECR-token refresher CronJob — periodically aws ecr get-login-password → recreate the ecr-cred docker-registry secret in each app namespace, on a schedule well under the ~12h expiry (e.g. every 6–8h). (Pattern already used ad hoc in deploy/eks/mirror-images-eks.sh:84-88, which notes the 12h expiry.)
Then flux reconcile / roll fleet--insights--prod to clear the wedged UpgradeFailed.

Follow-up worth deciding (real plat/mitosis question)

Should mitosis support an AWS/ECR-backed k3s leader as a first-class profile (shipping the ecr-cred refresher in the germinate gitops), or should the leader be moved back to the default in-cluster-registry path so ECR + ecr-cred isn't needed at all? Right now it's an unsupported, hand-wired config that silently rots every 12h.

Ref: audit PR #212 → docs/ops/aws-audit-2026-08-25.md (L3, H9).

Found during the AWS audit (PR #212, finding **L3**). This is a **live** issue on the `git.benjivers.com` EC2 k3s leader. ## Symptom `fleet/insights` **prod** HelmRelease is `UpgradeFailed`: ``` failed early due to stalled resources: [Deployment/fleet--insights--prod/insights status: 'Failed'] ``` with a recurring pod Warning: ``` FailedToRetrieveImagePullSecret: Unable to retrieve some image pull secrets (ecr-cred); attempting to pull the image may not succeed. ``` The pod happens to still be `Running` (0 restarts) on the already-pulled image, but the release can't complete and **any reschedule would fail to pull**. `fleet/{www,relay,review}` and `invoicing/fleet-invoice` are healthy — so it's this release/secret, not cluster-wide. ## Root cause - Every app on the leader pulls from ECR (`<acct>.dkr.ecr.us-east-1.amazonaws.com/...`) using an `imagePullSecrets: [ecr-cred]`. - **ECR auth tokens expire ~12h.** There is no live refresher keeping `ecr-cred` current, so the secret goes stale / missing and the pull-secret lookup fails. ## Why this is NOT a plat/mitosis source change - The string `ecr-cred` **does not exist anywhere in the mitosis source** (`grep -rn ecr-cred bin/ deploy/` → nothing). - Mitosis' AWS design is **node-IAM-ECR-role pulls, no pull secret** (`bin/germinate:71-72`, `:813` — "kubelet pulls via the node's ECR role"). Default k3s (what this leader runs) is meant to pull from the **in-cluster Forgejo registry** via the `127.0.0.1:31100` mirror (`registry-node-config`), not ECR. - So the leader was wired to ECR + `ecr-cred` **out-of-band**; that mechanism lives in the **leader's own gitops** (`plat/gitops` on `git.benjivers.com`), not here. ## Fix (lands in the daughter gitops, tracked here) 1. On `git.benjivers.com` `plat/gitops`, add (or repair) an **ECR-token refresher CronJob** — periodically `aws ecr get-login-password` → recreate the `ecr-cred` docker-registry secret in each app namespace, on a schedule well under the ~12h expiry (e.g. every 6–8h). (Pattern already used ad hoc in `deploy/eks/mirror-images-eks.sh:84-88`, which notes the 12h expiry.) 2. Then `flux reconcile` / roll `fleet--insights--prod` to clear the wedged `UpgradeFailed`. ## Follow-up worth deciding (real plat/mitosis question) Should mitosis **support** an AWS/ECR-backed k3s leader as a first-class profile (shipping the ecr-cred refresher in the germinate gitops), or should the leader be moved back to the default in-cluster-registry path so ECR + `ecr-cred` isn't needed at all? Right now it's an unsupported, hand-wired config that silently rots every 12h. Ref: audit PR #212 → `docs/ops/aws-audit-2026-08-25.md` (L3, H9).

benjivers added the

bug

label

2026-08-25 19:39:57 +00:00

benjivers referenced this issue

2026-08-25 19:40:20 +00:00

AWS deployment audit — EC2 leader + EKS backlog (34 items) #212

benjivers referenced this issue

2026-08-25 20:37:20 +00:00

AWS deployment audit — EC2 leader + EKS backlog (34 items) #212

benjivers closed this issue

2026-08-25 20:40:44 +00:00