forwardAuth breaks all app ingresses: traefik cannot resolve per-app kubernetescrd Middleware (middleware does not exist) #93

New issue

Open

opened 2026-07-06 19:23:01 +00:00 by plat · 0 comments

plat commented

2026-07-06 19:23:01 +00:00

Owner

Summary

On the product-garden.com granddaughter (bare-k3s VPS), enabling forwardAuth (RBAC-2, PLAT_FORWARD_AUTH=1 + platAuth.enabled=true) makes every app ingress 404 publicly. The plat-auth service itself is healthy (healthz 200, and its /forward endpoint returns a correct 302-to-Forgejo-OAuth decision), but traefik cannot resolve the per-app forwardAuth Middleware:

ERR error="middleware \"plat--qa-smoke--prod-plat-access@kubernetescrd\" does not exist"
    entryPointName=web routerName=web-plat-qa-smoke-prod-qa-smoke-...

Traefik invalidates the whole router when a referenced middleware can't be loaded → 404 on every app hostname.

What was ruled out

The plat-access / plat-strip Middleware CRs do exist in the app namespace (traefik.io/v1alpha1, correct names).
traefik providers.kubernetescrd is enabled with allowCrossNamespace=true (no namespace restriction), and kubectl auth can-i list middlewares.traefik.io = yes for the traefik SA.
A separate traefik RBAC gap (cannot list configmaps at cluster scope) was found and granted — the middleware error persisted after, so configmaps was not the cause.
traefik was restarted twice; the error persists. These are the ONLY @kubernetescrd middlewares on the platform (the platform's own redirect-to-https is a @file-provider middleware), so the kubernetescrd middleware provider may not be loading any CRD middlewares at all on this stack/version.

Impact + current mitigation

forwardAuth cannot be enabled on an app-serving platform until this is fixed — it takes down all app HTTPS. On the granddaughter I disabled it (platAuth.enabled=false, PLAT_FORWARD_AUTH=0) so apps serve; the crew-built qa-smoke app then served 200 over the public tunnel. The forwardAuth wiring (OAuth app, sealed creds, plat-auth deployment, auth ingress) is otherwise correct and healthy.

Suspected direction

traefik's kubernetescrd Middleware informer isn't populating its registry (CR present + RBAC ok + provider watching, yet "does not exist"). Candidates: (a) traefik chart/version's kubernetescrd provider not actually watching Middleware kind despite the flag; (b) an informer-cache sync stall (the platform's traefik ClusterRole is missing configmaps, which can block WaitForCacheSync for the whole shared informer factory — granting it did not clear it here, but the traefik ClusterRole should still be audited); (c) CRD group/version mismatch between the installed traefik and the traefik.io/v1alpha1 CRs the renderer emits.

Repro

Germinate a platform, enable forwardAuth, build any app → its public URL 404s with the middleware-does-not-exist error in traefik logs. Note: this needs a real app served through traefik — the k3d coldstart gate never builds+serves an app, so it never exercises forwardAuth end-to-end (same blind spot as #89/#91).

## Summary On the `product-garden.com` granddaughter (bare-k3s VPS), enabling forwardAuth (RBAC-2, `PLAT_FORWARD_AUTH=1` + `platAuth.enabled=true`) makes **every app ingress 404** publicly. The `plat-auth` service itself is healthy (healthz 200, and its `/forward` endpoint returns a correct 302-to-Forgejo-OAuth decision), but traefik cannot resolve the per-app forwardAuth Middleware: ``` ERR error="middleware \"plat--qa-smoke--prod-plat-access@kubernetescrd\" does not exist" entryPointName=web routerName=web-plat-qa-smoke-prod-qa-smoke-... ``` Traefik invalidates the whole router when a referenced middleware can't be loaded → 404 on every app hostname. ## What was ruled out - The `plat-access` / `plat-strip` Middleware CRs **do exist** in the app namespace (`traefik.io/v1alpha1`, correct names). - traefik `providers.kubernetescrd` is enabled with `allowCrossNamespace=true` (no namespace restriction), and `kubectl auth can-i list middlewares.traefik.io` = **yes** for the traefik SA. - A separate traefik RBAC gap (`cannot list configmaps at cluster scope`) was found and granted — the middleware error **persisted** after, so configmaps was not the cause. - traefik was restarted twice; the error persists. These are the ONLY `@kubernetescrd` middlewares on the platform (the platform's own `redirect-to-https` is a `@file`-provider middleware), so the kubernetescrd middleware provider may not be loading any CRD middlewares at all on this stack/version. ## Impact + current mitigation forwardAuth **cannot be enabled** on an app-serving platform until this is fixed — it takes down all app HTTPS. On the granddaughter I **disabled it** (`platAuth.enabled=false`, `PLAT_FORWARD_AUTH=0`) so apps serve; the crew-built `qa-smoke` app then served 200 over the public tunnel. The forwardAuth wiring (OAuth app, sealed creds, `plat-auth` deployment, auth ingress) is otherwise correct and healthy. ## Suspected direction traefik's kubernetescrd Middleware informer isn't populating its registry (CR present + RBAC ok + provider watching, yet "does not exist"). Candidates: (a) traefik chart/version's kubernetescrd provider not actually watching Middleware kind despite the flag; (b) an informer-cache sync stall (the platform's traefik ClusterRole is missing `configmaps`, which can block `WaitForCacheSync` for the whole shared informer factory — granting it did not clear it here, but the traefik ClusterRole should still be audited); (c) CRD group/version mismatch between the installed traefik and the `traefik.io/v1alpha1` CRs the renderer emits. ## Repro Germinate a platform, enable forwardAuth, build any app → its public URL 404s with the middleware-does-not-exist error in traefik logs. Note: this needs a real app served through traefik — the k3d coldstart gate never builds+serves an app, so it never exercises forwardAuth end-to-end (same blind spot as #89/#91).

plat referenced this issue

2026-07-06 19:24:10 +00:00

/goal /ultracode-workflows [GATE] validated grandchild acceptance gate — secrets+domains+rbac+policy on a VPS depth-2/3 platform #46