monitoring: grafana-folder-acl CronJob fails every 2m post-germinate — cannot reach grafana service (ECONNREFUSED) #233

New issue

Closed

opened 2026-08-26 20:09:27 +00:00 by benjivers · 2 comments

benjivers commented

2026-08-26 20:09:27 +00:00

Collaborator

Symptom

On a fresh single-node germination, the grafana-folder-acl CronJob (*/2 * * * *, namespace monitoring) fails every run and leaves a steady stream of Error pods:

NAMESPACE    NAME                                STATUS
monitoring   grafana-folder-acl-29796244-...     Error
monitoring   grafana-folder-acl-29796246-...     Error
monitoring   grafana-folder-acl-29796248-...     Error

Pod log (Bun script):

TypeError: Unable to connect. Is the computer able to access the url?
  path: "http://kube-prometheus-stack-grafana.monitoring/api/folders?limit=100",
  errno: 0,
  code: "ConnectionRefused"
Bun v1.4.0 (Linux x64)

No grafana-folder-acl job has ever succeeded on this cluster.

Not a startup race

Grafana itself is healthy — kube-prometheus-stack-grafana pod is 3/3 Running, service 10.43.89.145:80/TCP, endpoints 10.42.0.80:3000. Yet the failures persist for many minutes and across every job cycle after Grafana became Ready.

To rule out the CronJob image, I ran an independent probe pod in the same namespace:

kubectl -n monitoring run netcheck --rm -i --restart=Never \
  --image=curlimages/curl -- curl http://kube-prometheus-stack-grafana.monitoring/api/health
# -> HTTP 000, pod terminated (Error)

So a plain curl pod in monitoring also cannot reach the Grafana service — the folder-acl job is just the visible victim.

Likely area: NetworkPolicies

monitoring has a default-deny-ingress plus allow-lists:

NAME                            POD-SELECTOR
allow-ingress-grafana           app.kubernetes.io/name=grafana
allow-ingress-intra-namespace   <none>
allow-ingress-prometheus-flow   app.kubernetes.io/name=prometheus
allow-ingress-prometheus-www    app.kubernetes.io/name=prometheus
default-deny-ingress            <none>

allow-ingress-intra-namespace (empty pod-selector) is presumably meant to let in-namespace pods (the folder-acl job, the sidecars) reach Grafana, but ingress to Grafana:3000 is being refused/blocked in practice. Worth checking whether allow-ingress-grafana actually admits the folder-acl Job's pod labels / the intra-namespace policy covers port 3000, or whether the folder-acl Job needs a label the allow-list selects on.

Environment

plat/mitosis origin/main @ b0ea45d, deploy/do, single droplet (ha=false), s-4vcpu-8gb, nyc3, k3s v1.31.5+k3s1.
Grafana 13.1.1; folder-acl image is a Bun script.
Observed ~25 min after germination completed; still failing.

Impact

Low urgency (Grafana serves fine; only the folder-ACL provisioning CronJob is broken), but it produces a continuous stream of Error pods every 2 minutes forever, which clutters the namespace and will trip any "failed pods" alerting.

## Symptom On a fresh single-node germination, the `grafana-folder-acl` CronJob (`*/2 * * * *`, namespace `monitoring`) **fails every run** and leaves a steady stream of `Error` pods: ``` NAMESPACE NAME STATUS monitoring grafana-folder-acl-29796244-... Error monitoring grafana-folder-acl-29796246-... Error monitoring grafana-folder-acl-29796248-... Error ``` Pod log (Bun script): ``` TypeError: Unable to connect. Is the computer able to access the url? path: "http://kube-prometheus-stack-grafana.monitoring/api/folders?limit=100", errno: 0, code: "ConnectionRefused" Bun v1.4.0 (Linux x64) ``` **No `grafana-folder-acl` job has ever succeeded** on this cluster. ## Not a startup race Grafana itself is healthy — `kube-prometheus-stack-grafana` pod is `3/3 Running`, service `10.43.89.145:80/TCP`, endpoints `10.42.0.80:3000`. Yet the failures persist for many minutes and across every job cycle after Grafana became Ready. To rule out the CronJob image, I ran an **independent** probe pod in the same namespace: ``` kubectl -n monitoring run netcheck --rm -i --restart=Never \ --image=curlimages/curl -- curl http://kube-prometheus-stack-grafana.monitoring/api/health # -> HTTP 000, pod terminated (Error) ``` So a plain curl pod in `monitoring` also cannot reach the Grafana service — the folder-acl job is just the visible victim. ## Likely area: NetworkPolicies `monitoring` has a `default-deny-ingress` plus allow-lists: ``` NAME POD-SELECTOR allow-ingress-grafana app.kubernetes.io/name=grafana allow-ingress-intra-namespace <none> allow-ingress-prometheus-flow app.kubernetes.io/name=prometheus allow-ingress-prometheus-www app.kubernetes.io/name=prometheus default-deny-ingress <none> ``` `allow-ingress-intra-namespace` (empty pod-selector) is presumably meant to let in-namespace pods (the folder-acl job, the sidecars) reach Grafana, but ingress to Grafana:3000 is being refused/blocked in practice. Worth checking whether `allow-ingress-grafana` actually admits the folder-acl Job's pod labels / the intra-namespace policy covers port 3000, or whether the folder-acl Job needs a label the allow-list selects on. ## Environment - `plat/mitosis` `origin/main` @ `b0ea45d`, `deploy/do`, single droplet (`ha=false`), `s-4vcpu-8gb`, nyc3, k3s `v1.31.5+k3s1`. - Grafana `13.1.1`; folder-acl image is a Bun script. - Observed ~25 min after germination completed; still failing. ## Impact Low urgency (Grafana serves fine; only the folder-ACL provisioning CronJob is broken), but it produces a continuous stream of `Error` pods every 2 minutes forever, which clutters the namespace and will trip any "failed pods" alerting.

benjivers added the

bug

label

2026-08-26 20:09:27 +00:00

benjivers referenced this issue

2026-08-27 19:17:48 +00:00

germination: seeded _app-template is private + owned by the plat user — operators expect a discoverable/public app template #236

trevato commented

2026-08-31 18:56:55 +00:00

Owner

Fix on gitops main (already live on the parent; daughters inherit at next reseed): the allow-ingress-intra-namespace policy now also matches by namespace membership — your evidence showed the namespaceSelector path (traefik → grafana) working while the podSelector-only intra-namespace allow refused every short-lived job pod, which is a known-fragile match on k3s's embedded netpol controller. If your platform still shows Error pods after pulling the netpol change, re-open with a fresh kubectl -n monitoring run netcheck … probe.

Fix on gitops main (already live on the parent; daughters inherit at next reseed): the `allow-ingress-intra-namespace` policy now also matches by namespace membership — your evidence showed the namespaceSelector path (traefik → grafana) working while the podSelector-only intra-namespace allow refused every short-lived job pod, which is a known-fragile match on k3s's embedded netpol controller. If your platform still shows Error pods after pulling the netpol change, re-open with a fresh `kubectl -n monitoring run netcheck …` probe.

trevato commented

2026-08-31 21:01:32 +00:00

Owner

Closing platform-side: the namespace-membership match is live on the parent and ships in seed r779, so every new deploy is clean. For the existing platform from this report, pull the one-hunk netpol change into your gitops (clusters/local/platform/monitoring/netpol.yaml, gitops 617b0554) — or redeploy from the current seed — and the folder-acl job goes green. Re-open with a fresh netcheck probe if it doesn't.

Closing platform-side: the namespace-membership match is live on the parent and ships in seed r779, so every new deploy is clean. For the existing platform from this report, pull the one-hunk netpol change into your gitops (`clusters/local/platform/monitoring/netpol.yaml`, gitops 617b0554) — or redeploy from the current seed — and the folder-acl job goes green. Re-open with a fresh `netcheck` probe if it doesn't.

trevato closed this issue

2026-08-31 21:01:33 +00:00