flux notification-controller: 'cannot assign requested address' to forgejo-http under load (per-org alert providers) #203

New issue

Open

opened 2026-08-24 21:53:34 +00:00 by plat · 0 comments

plat commented

2026-08-24 21:53:34 +00:00

Owner

Seen 2026-08-24 ~19:45Z on prod (vxrail load avg 42–63): every Kustomization alert (alert/plat-gitops, alert/org-*) logs

NotificationDispatchFailed … failed creating Gitea client: Get http://forgejo-http.forgejo.svc.cluster.local:3000/api/v1/version: dial tcp 10.43.36.153:3000: connect: cannot assign requested address

Host-level sockets/conntrack were fine (conntrack 41k/2.6M, TIME_WAIT 41), so this is source-port pressure inside the notification-controller pod: one Gitea client per provider × ~30 org providers × every reconcile, each doing a fresh /api/v1/version handshake. Same window: isolated runners logged connection refused re-registering, and an mcp release build (task 21813) died mid-flight after 13 min; the retry built in 3 min.

Worth: consolidating org alerts onto one provider (or a longer interval), and checking notification-controller's tcp_tw_reuse/keep-alive behaviour. Not caused by the template rename — surfaced while shipping it.

🤖 Generated with Claude Code

Seen 2026-08-24 ~19:45Z on prod (vxrail load avg 42–63): every Kustomization alert (`alert/plat-gitops`, `alert/org-*`) logs NotificationDispatchFailed … failed creating Gitea client: Get http://forgejo-http.forgejo.svc.cluster.local:3000/api/v1/version: dial tcp 10.43.36.153:3000: connect: cannot assign requested address Host-level sockets/conntrack were fine (conntrack 41k/2.6M, TIME_WAIT 41), so this is source-port pressure inside the notification-controller pod: one Gitea client per provider × ~30 org providers × every reconcile, each doing a fresh `/api/v1/version` handshake. Same window: isolated runners logged `connection refused` re-registering, and an mcp release build (task 21813) died mid-flight after 13 min; the retry built in 3 min. Worth: consolidating org alerts onto one provider (or a longer `interval`), and checking notification-controller's `tcp_tw_reuse`/keep-alive behaviour. Not caused by the template rename — surfaced while shipping it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

plat referenced this issue

2026-08-24 21:54:54 +00:00

program-log: the template deploys itself (2026-08-24) #204