The engine room: how one git repo becomes a running platform. This page walks open-platform/gitops top to bottom — the Flux graph, secrets, identities, TLS, the registry, and tenant governance. Read the repo and you can verify every claim here. For the guided overview, start at Architecture.

The gitops repo

Every part of the platform — from Forgejo itself to every tenant app — is a file in open-platform/gitops. A change to the platform is a git commit. Flux converges the cluster to the repo on a 1-minute interval.

The repo root holds .sops.yaml and clusters/. Under clusters/local/:

• platform/ — the platform components (one directory each)
• apps/ — per-org, per-app overlays (see below)
• domains.yaml — the bring-your-own-domain registry: one line <domain>: <org>, and apps in that org resolve to it
• flux-kustomizations.yaml — the Flux entrypoint: 12 Kustomizations with an explicit dependsOn DAG

The Flux graph

Compactly, with ← meaning "depends on":

• cert-manager — the root; everything TLS descends from it
• pki ← cert-manager
• monitoring ← cert-manager; monitoring-scrapes ← monitoring
• kyverno ← cert-manager; kyverno-policies ← kyverno
• flow ← cert-manager, traefik
• platform ← cert-manager, traefik, keda
• apps ← platform, pki
• traefik, keda, cnpg-operator — independent roots

Two design decisions are written into the file as comments. monitoring is deliberately not a dependency of apps ("an observability hiccup must never stall application reconciliation"). ServiceMonitors live in the separate monitoring-scrapes Kustomization, because a dry-run failure on an unknown kind blocks an entire Kustomization. SOPS decryption is declared on exactly four Kustomizations: monitoring, flow, platform, apps.

The green check

Flux reports back to Forgejo. Each apply posts a commit status onto the gitops repo it came from. A green check on a commit means the cluster runs that commit. A red mark means the apply failed. Statuses appear in the commit list and on pull requests.

The feedback also flows the other way. A push to a gitops repo triggers a webhook, and Flux reconciles in seconds. Without the webhook, Flux polls on a one-minute interval. Each org's own gitops repo gets both parts at birth: the status posts and the push webhook. Daughters inherit the same wiring.

What each component is:

Component	One line
forgejo	The control plane: the Forgejo Helm chart + CNPG Postgres, the git.open-platform.sh Ingress, the isolated CI runner pool
traefik	Edge ingress; system-cluster-critical priority with guaranteed CPU, so a saturated node cannot kill the edge
cnpg-operator	CloudNativePG; runs every Postgres on the platform (Forgejo's and each app's)
keda	Powers the scale-from-zero CI pool via a ScaledJob postgres trigger
mcp	The platform-bot / control-plane API at mcp.open-platform.sh
agents	The AI build-crew dispatcher; all token-flow tuning is env
flow	The Flow Deck console (a read-only surface)
monitoring	kube-prometheus-stack: Prometheus (60d retention), Grafana with Forgejo OAuth
monitoring-scrapes	ServiceMonitors for the agents and mcp /metrics endpoints
kyverno / kyverno-policies	The admission engine + the 12-policy governance baseline (see below)
letsencrypt	The public ACME (DNS-01) ClusterIssuer + the public wildcard certificate
cosign-signing	The platform's image-signing keypair — the source of provenance trust
registry-node-config	The node-local registry endpoint on 127.0.0.1:31100 (see Registry)
node-fabric	A per-node DaemonSet; keeps tx-checksum offload disabled on flannel's vxlan device (multi-node packet integrity)
plat-storage	In-cluster MinIO + a cnpg-backups bucket with a least-privilege backup user for Postgres WAL backups
plat-system	The reserved-subdomain registry (git, mcp, grafana, console, ...)
pki	The private platform CA (see TLS)
coredns	Split-horizon DNS: platform hostnames resolve to traefik from inside the cluster
cert-manager	Issues every certificate; the root of the DAG

On a freshly-germinated lean daughter, monitoring, monitoring-scrapes, and flow are stripped at germination — observability is parent-only on a lean daughter today. Everything else above is present from birth.

How an app lands — and leaves

Every app environment is an overlay at clusters/local/apps/<org>/<app>/<env>/, where <env> is prod, pr-<N> (a PR preview), or occasionally dev. An overlay is live only when it is listed in apps/kustomization.yaml — the list, not the directory tree, is the deploy roster.

Each env dir is a kustomization.yaml, a manifests.yaml, and four sealed secrets. manifests.yaml renders the whole app:

1. Namespace <org>--<app>--<env> with plat.sh/* ownership labels and pod-security.kubernetes.io/enforce: restricted
2. CNPG Postgres <app>-pg (only when the app declares a database)
3. Certificate wildcard-plat-tls for <app>-<org>.open-platform.sh
4. OCIRepository pinning a chart tag from oci://forgejo-http.forgejo.svc.cluster.local:3000/<org>/charts/<app> (previews pin a PR tag like 0.0.0-pr.13.0757a36)
5. HelmRelease wiring the DB secret and the self-serve env into the pod
6. Ingress on <app>-<org>.open-platform.sh (previews: pr-<N>-<app>-<org>.open-platform.sh)

No human writes this YAML by hand. create_app and preview provisioning render the overlay, seal the secrets, append the roster entry, and commit. Flux applies the commit within a minute.

• Rollout: a release bumps the chart tag in the overlay → the OCIRepository sees the new tag → the HelmRelease upgrades.
• Config change: a self-serve secret change bumps a userEnvRev hash → the pod annotation changes → the Deployment re-rolls. (Chart env is read only at pod start.)
• Decommission: remove the roster entry and the overlay dir. Flux prune: true deletes everything, namespace included. Deleting only the namespace does not work — Flux resurrects it. The overlay is the source of truth.

Secrets: SOPS + age

Every secret in the repo is sealed with SOPS to a single age recipient — the platform's sovereign key. Two rules live in .sops.yaml:

1. clusters/.+\.yaml$ with encrypted_regex: ^(data|stringData)$ — only data fields are sealed; metadata and comments stay readable in git
2. clusters/.+/values\.env$ — the whole file

The age private key lives cluster-side in the sops-age secret in flux-system. Only the four Kustomizations that declare decryption can use it. When a platform reproduces, the daughter mints a fresh key, and every secret is regenerated and resealed. The parent's sealed values in the seed are inert ciphertext (see Grow Your Own Platform).

Every app overlay carries four sealed secrets: the image pull secret, the app's Forgejo OAuth credentials, a mirror of the platform CA cert, and the app's self-serve env. The self-serve env is sealed from the repo's Actions secrets by the sync-env flow — never edited in gitops directly (see Secrets and Config).

Identities: plat, ci-bot, registry bots

• plat — the site-admin machine user, the platform's root. It owns the platform org open-platform, home of the system repos; it owns no repo itself. Platform services authenticate with its token; humans never work as plat.
• ci-bot — a restricted user whose token is write:repository, write:issue only; it opens automated PRs (e.g. seed refreshes).
• Registry credentials — per-org, write:package-only. The REGISTRY_TOKEN an app's CI pushes images and charts with is one of these — never an admin token.

Power is scoped by construction, not by policy documents. The human-facing side of this split is in Users Teams and Access; the security analysis is in Security Model.

TLS: two realms

• The private platform CA (pki/): a self-signed bootstrap issuer creates a 10-year plat-local-ca CA (secret plat-local-ca-tls). The plat-local-ca-issuer signs every internal *.open-platform.sh certificate — including each app's wildcard-plat-tls.
• Public (letsencrypt/): the letsencrypt-dns01 ClusterIssuer (ACME DNS-01 via Cloudflare) issues wildcard-public-tls for browser-trusted hosts.

On a local platform, trust the CA once and every app shows a green padlock:

1. Export the plat-local-ca-tls secret.
2. Add the certificate to your OS trust store: security add-trusted-cert on macOS, update-ca-certificates on Linux.

The YOUR PLATFORM card prints the exact command for your OS at germination.

Registry and the node mirror

Every app's chart and image live in Forgejo's own package registry, under the owning org: charts at oci://forgejo-http.forgejo.svc.cluster.local:3000/<org>/charts/<app>, images at git.open-platform.sh/<org>/<app>. Platform components use open-platform/charts/* the same way — the platform ships itself through the same registry it gives tenants.

Nodes pull platform images through a local mirror. containerd maps git.open-platform.sh → http://127.0.0.1:31100 via a registries.yaml frozen into each node at cluster creation. At runtime, a hostNetwork Caddy DaemonSet binds loopback-only 127.0.0.1:31100 on every node and forwards to a small rewriter Deployment. The rewriter rewrites the registry's Www-Authenticate realm to the local endpoint, so containerd's token fetch resolves locally — no public DNS required, on any domain. This is why platform HelmReleases carry imagePullSecrets: []: registry auth is node-level.

Two more pieces:

• A docker.io pull-through cache (registry v3 proxying mirror.gcr.io, 10Gi cache) keeps CI base-image pulls off Docker Hub's rate limits.
• Provenance: CI cosign-signs every app image, and the verify-image-provenance ClusterPolicy admits only images that validate against the platform's public key. The keypair lives in cosign-signing/, deliberately under the platform Kustomization — the one with SOPS decryption.

Kyverno: tenant governance

Twelve ClusterPolicies form the governance baseline. Kyverno enforces them at admission with an HA controller. The two that shape every tenant namespace:

• plat-ns-resource-governance — generates a ResourceQuota and a LimitRange in every app namespace
• plat-ns-network-governance — generates default-deny plus allow NetworkPolicies in every app namespace

Both key off the plat.sh/app namespace label with synchronize: true self-heal: delete a generated quota and it comes back. The rest of the roster: disallow-privileged-hostpath, require-resource-limits, restrict-image-registries, plat-ns-dos-bound, disallow-unmanaged-secrets, require-registered-ingress-host, verify-image-provenance, no-admin-clusterrole-binding, agent-runs-use-scoped-role, and require-forwardauth-on-app-ingress (audit-only; its Enforce component is deliberately not shipped).

The admission webhook is scoped to namespaces that carry the plat.sh/app label — tenant namespaces only. Background policies report on every namespace they match.

When something looks wrong

• Run flux get kustomizations — it shows which of the 12 is unhappy, and why. Reconciliation is 1-minute; most "why hasn't it shipped" questions answer themselves here.
• Run kubectl -n agents logs deploy/agents — the build dispatcher; per-run crew logs live in the pod under /data/logs.
• Open the repo's Actions tab — every CI build's logs.
• An app namespace you deleted came back? That is Flux doing its job — decommission through the overlay.
• An Ingress that never matches? Every public Ingress must declare the entrypoints web,websecure — tunnel-fronted traffic arrives on web, so a websecure-only router silently matches nothing.