DOM-2 [NO-GO / DO NOT MERGE]: germinate on Cilium CNI — evidence it breaks first-try germination #59

Closed

plat wants to merge 1 commit from feat/dom-2-cilium-fqdn-egress into main

plat commented

2026-07-01 21:19:25 +00:00

Owner

DOM-2 (plat/mitosis#37) asked to swap the germination CNI to Cilium. Per the brief's STRICT-honesty clause, this is a NO-GO: on the k3d/k3s germination stack, neither viable Cilium configuration germinates a full platform first-try. This branch (bin/up: --flannel-backend=none + --disable-network-policy + --disable-kube-proxy + Cilium 1.17.17 kube-proxy-replacement) is preserved as evidence — do not merge.

Two independent, reproducible blockers (both tested live on vxrail, clusters torn down)

1. kube-proxy PRESENT (Cilium CNI + netpol only): platform reaches the CI-build phase (all 25+ pods Running), then mcp/agents — and every in-cluster-built app image — fail ImagePull. Root cause: the platform's registry mirror is reached by containerd over a hostPort (127.0.0.1:31100, the registry-node-config DaemonSet), and Cilium only programs hostPort when it replaces kube-proxy. Setting enable-host-port=true alone (kpr=false) does not bind the port (verified: node wget 127.0.0.1:31100 → connection refused).

2. kube-proxy REPLACED (kpr=true, auto-enables hostPort): germination hard-fails at 'install Flux + point at gitops'. Flux's 4 controllers + metrics-server + local-path-provisioner all CrashLoopBackOff with x509: certificate signed by unknown authority when dialing the apiserver via the kubernetes ClusterIP. Cilium routing is correct (service maps 10.43.0.1:443 → 172.18.0.2:6443, a pod reaches it and gets 401), but the cert served under kpr routing is not trusted by the pods' ServiceAccount CA bundle (curl --cacert ca.crt → exit 60). bpf.hostLegacyRouting=true did not fix it. This is a known k3s/k3d + Cilium-kpr apiserver-trust incompatibility.

What DOES work on Cilium (proven, shipping separately)

On the kube-proxy-present run, before the registry blocker, I proved the value of Cilium end-to-end:

cilium status OK (1.17.17), Controllers healthy.
Standard networking.k8s.io/v1 NetworkPolicy still enforces natively: default-deny blocked (HTTP 000/timeout), an allow rule restored it (HTTP 200), endpoint ingress ENFORCEMENT=Enabled.
The mcp renderer's CiliumNetworkPolicy toFQDNs egress works: allowlisted api.github.com→200, non-allowlisted api.gitlab.com/pypi.org→BLOCKED, DNS preserved, egress ENFORCEMENT=Enabled.

Recommendation

Keep kube-router (the current default CNI) for germination — do not merge this bin/up change.
Ship the FQDN-egress renderer (plat/mcp feat/dom-2-fqdn-egress + plat/_app-template feat/dom-2-allowed-hosts): it is additive, gated default-off behind PLAT_FQDN_EGRESS, tested, and proven to work the moment the platform runs on Cilium.
Before any future Cilium adoption, the germination seam needs: (a) the registry mirror to stop depending on hostPort (e.g. a NodePort/hostNetwork socat, so kube-proxy can stay), OR (b) the k3s+Cilium-kpr apiserver-cert-trust issue resolved (dedicated apiserver LB SAN / cert handling).

DOM-2 (plat/mitosis#37) asked to swap the germination CNI to Cilium. Per the brief's STRICT-honesty clause, this is a **NO-GO**: on the k3d/k3s germination stack, **neither** viable Cilium configuration germinates a full platform first-try. This branch (bin/up: `--flannel-backend=none` + `--disable-network-policy` + `--disable-kube-proxy` + Cilium 1.17.17 kube-proxy-replacement) is preserved as evidence — **do not merge**. ## Two independent, reproducible blockers (both tested live on vxrail, clusters torn down) **1. kube-proxy PRESENT (Cilium CNI + netpol only):** platform reaches the CI-build phase (all 25+ pods Running), then `mcp`/`agents` — and every in-cluster-built app image — fail `ImagePull`. Root cause: the platform's registry mirror is reached by containerd over a **hostPort** (127.0.0.1:31100, the `registry-node-config` DaemonSet), and Cilium only programs hostPort when it replaces kube-proxy. Setting `enable-host-port=true` alone (kpr=false) does **not** bind the port (verified: node `wget 127.0.0.1:31100` → connection refused). **2. kube-proxy REPLACED (kpr=true, auto-enables hostPort):** germination **hard-fails at 'install Flux + point at gitops'**. Flux's 4 controllers + metrics-server + local-path-provisioner all CrashLoopBackOff with `x509: certificate signed by unknown authority` when dialing the apiserver via the kubernetes ClusterIP. Cilium routing is correct (service maps `10.43.0.1:443 → 172.18.0.2:6443`, a pod reaches it and gets 401), but the cert served under kpr routing is not trusted by the pods' ServiceAccount CA bundle (curl `--cacert ca.crt` → exit 60). `bpf.hostLegacyRouting=true` did **not** fix it. This is a known k3s/k3d + Cilium-kpr apiserver-trust incompatibility. ## What DOES work on Cilium (proven, shipping separately) On the kube-proxy-present run, before the registry blocker, I proved the *value* of Cilium end-to-end: - `cilium status` OK (1.17.17), Controllers healthy. - **Standard `networking.k8s.io/v1` NetworkPolicy still enforces natively**: default-deny blocked (HTTP 000/timeout), an allow rule restored it (HTTP 200), endpoint ingress ENFORCEMENT=Enabled. - **The mcp renderer's CiliumNetworkPolicy toFQDNs egress works**: allowlisted `api.github.com`→200, non-allowlisted `api.gitlab.com`/`pypi.org`→BLOCKED, DNS preserved, egress ENFORCEMENT=Enabled. ## Recommendation - **Keep kube-router** (the current default CNI) for germination — do not merge this bin/up change. - **Ship the FQDN-egress renderer** (plat/mcp `feat/dom-2-fqdn-egress` + plat/_app-template `feat/dom-2-allowed-hosts`): it is additive, gated default-off behind `PLAT_FQDN_EGRESS`, tested, and proven to work the moment the platform runs on Cilium. - **Before any future Cilium adoption**, the germination seam needs: (a) the registry mirror to stop depending on hostPort (e.g. a NodePort/hostNetwork socat, so kube-proxy can stay), OR (b) the k3s+Cilium-kpr apiserver-cert-trust issue resolved (dedicated apiserver LB SAN / cert handling).

plat added 1 commit

2026-07-01 21:19:25 +00:00