deploy/azure: mode=vms|aks in main.bicep, HA servers carve-out, AKS germinate profile #139

Merged

trevato merged 10 commits from azure-aks-mode into main 2026-08-19 15:22:17 +00:00

Conversation 2 Commits 10 Files changed 9 +1637 -38

benjivers commented 2026-08-19 04:01:25 +00:00

Collaborator

Copy link

Lands the Azure work that until now lived only as uncommitted files in a local checkout (the AKS azure-pure POC + the 3-node HA expansion, both platforms since torn down — this PR is the surviving artifact).

What's here

deploy/azure/main.bicep — one template, mode as a parameter

• mode=vms (default): the existing bare-VM path, rebased onto current main — the #123/#124 PUBLIC_IP-in-node.env behavior is preserved.
• mode=aks: managed ground — AKS (workload identity, Cilium network policy, managed Prometheus + Container Insights), ACR, public DNS zone, Key Vault with federated identities (mitosis-cert-manager → DNS Zone Contributor, mitosis-flux-sops → KV Crypto User).

deploy/azure/servers.bicep — servers-only carve-out for growing a LIVE single-node group to 3-node HA. Exists because running main.bicep ha=true against a live group replaces immutable customData and NSG rules. Previously archived only in a plat/mitosis#6 comment; param docs carry the join-token gotcha (joiners present the existing cluster's token, not a fresh GUID).

bin/germinate + bin/lib.sh — CLUSTER_PROFILE=aks: trim_gitops_aks strips k3s-specific organs (monitoring/Grafana → Azure Monitor, node-fabric, registry-node-config, coredns rewrite), repoints the runner image at ACR (az acr build replaces docker+ctr side-load), adds a Kyverno aks-image-redirect mutate policy and extends restrict-image-registries. Includes the 8 aks-ground fixes from the POC germination runs. Rebase note: runs after upstream's new strip_gitops_tenants, before identity rewrite.

Runbooks/scripts: README-aks.md (customer runbook incl. pilot hardening), mirror-images.sh (~25 az acr import), entra-forgejo-oidc.sh (Entra ID sign-in for aks-ground Forgejo), aks-cost-experiment.md (62% cost-cut runbook: CNPG PDBs block drains, B-v2 quota 0, nodepool-add CLI crash → az rest).

Review notes

• mode=vms was verified to render byte-identical ARM vs the pre-mode template at the time of the split; it has since been rebased over #123/#124 and #132 — worth a re-render if touching the vms path.
• Pushed via the MCP contents API (no git credential on the authoring machine), so two new scripts lack the executable bit. After merge (or on the branch): chmod +x deploy/azure/entra-forgejo-oidc.sh deploy/azure/mirror-images.sh. Both are also runnable as bash <script>.
• History has a few superseded corrective commits (same message) from the API push; tree content is hash-verified byte-identical to the authoring branch. Squash-merge recommended.

🤖 Generated with Claude Code

Lands the Azure work that until now lived only as uncommitted files in a local checkout (the AKS azure-pure POC + the 3-node HA expansion, both platforms since torn down — this PR is the surviving artifact). ## What's here **`deploy/azure/main.bicep` — one template, `mode` as a parameter** - `mode=vms` (default): the existing bare-VM path, rebased onto current main — the #123/#124 PUBLIC_IP-in-node.env behavior is preserved. - `mode=aks`: managed ground — AKS (workload identity, Cilium network policy, managed Prometheus + Container Insights), ACR, public DNS zone, Key Vault with federated identities (`mitosis-cert-manager` → DNS Zone Contributor, `mitosis-flux-sops` → KV Crypto User). **`deploy/azure/servers.bicep`** — servers-only carve-out for growing a LIVE single-node group to 3-node HA. Exists because running `main.bicep ha=true` against a live group replaces immutable `customData` and NSG rules. Previously archived only in a plat/mitosis#6 comment; param docs carry the join-token gotcha (joiners present the existing cluster's token, not a fresh GUID). **`bin/germinate` + `bin/lib.sh`** — `CLUSTER_PROFILE=aks`: `trim_gitops_aks` strips k3s-specific organs (monitoring/Grafana → Azure Monitor, node-fabric, registry-node-config, coredns rewrite), repoints the runner image at ACR (`az acr build` replaces docker+ctr side-load), adds a Kyverno `aks-image-redirect` mutate policy and extends `restrict-image-registries`. Includes the 8 aks-ground fixes from the POC germination runs. Rebase note: runs after upstream's new `strip_gitops_tenants`, before identity rewrite. **Runbooks/scripts**: `README-aks.md` (customer runbook incl. pilot hardening), `mirror-images.sh` (~25 `az acr import`), `entra-forgejo-oidc.sh` (Entra ID sign-in for aks-ground Forgejo), `aks-cost-experiment.md` (62% cost-cut runbook: CNPG PDBs block drains, B-v2 quota 0, nodepool-add CLI crash → `az rest`). ## Review notes - `mode=vms` was verified to render byte-identical ARM vs the pre-`mode` template at the time of the split; it has since been rebased over #123/#124 and #132 — worth a re-render if touching the vms path. - Pushed via the MCP contents API (no git credential on the authoring machine), so two new scripts lack the executable bit. After merge (or on the branch): `chmod +x deploy/azure/entra-forgejo-oidc.sh deploy/azure/mirror-images.sh`. Both are also runnable as `bash <script>`. - History has a few superseded corrective commits (same message) from the API push; tree content is hash-verified byte-identical to the authoring branch. Squash-merge recommended. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

benjivers added 9 commits 2026-08-19 04:01:26 +00:00

deploy/azure: entra-forgejo-oidc.sh — Entra ID sign-in for aks-ground Forgejo 60bdaeee23

deploy/azure: AKS runbook, image mirror script, cost-experiment notes, HA servers carve-out 34601f8368

deploy/azure: main.bicep learns mode=vms|aks (AKS + ACR + DNS zone + Key Vault federated identities) b994d05fd3

deploy/azure: main.bicep learns mode=vms|aks (AKS + ACR + DNS zone + Key Vault federated identities) 3c3d1f4fca

deploy/azure: main.bicep learns mode=vms|aks (AKS + ACR + DNS zone + Key Vault federated identities) 48f1c4b487

germinate: CLUSTER_PROFILE=aks (ACR-built ci-runner, aks gitops trim hook) fa62fd4c89

lib.sh: trim_gitops_aks — adapt gitops tree to a managed Azure ground de264485df

lib.sh: trim_gitops_aks — adapt gitops tree to a managed Azure ground 45b8e145a1

lib.sh: trim_gitops_aks — adapt gitops tree to a managed Azure ground e52a995f80

ci-bot added 1 commit 2026-08-19 14:05:43 +00:00

deploy/azure: README.md — one Azure guide, both modes, in Simplified Technical English ... 5747ef00f8

Resources each mode creates, numbered steps, servers.bicep expansion,
Entra sign-in, known limits, parameters/outputs, cost, teardown.
Folds README-aks.md in; links from deploy/README.md. Germinate step
carries AKS_RG (required by CLUSTER_PROFILE=aks). Sets the exec bit on
mirror-images.sh and entra-forgejo-oidc.sh (lost in the API push).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

ci-bot commented 2026-08-19 14:06:24 +00:00

Collaborator

Copy link

Pushed 5747ef0 to this branch: deploy/azure/README.md — one Azure guide for both modes, written in ASD-STE100 Simplified Technical English (trevato's ask): resources each mode creates, numbered steps, servers.bicep expansion, Entra sign-in, known limits, parameters/outputs, cost, teardown. It folds README-aks.md in (removed — one doc, no drift) and links from deploy/README.md. Also set the exec bit on mirror-images.sh / entra-forgejo-oidc.sh as the PR body asked.

Two things I changed in the runbook vs. README-aks.md — please check them against what you ran:

• The germinate step now carries AKS_RG=<rg> — bin/germinate :?-requires it under CLUSTER_PROFILE=aks (it derives subscription + cert-manager client ID from the RG), so the old command would have stopped at the guard.
• Mirror-before-germinate is now an explicit instruction (the aks-image-redirect comment in lib.sh says redirected pods ErrImagePull otherwise).

Review notes on the rest of the branch (read-only — no az/bicep here, so no compile; nothing below blocks merge):

1. AKS defaults resurrect the $364/mo baseline — main.bicep:80/83/346 still pin 2× D4ds_v6 + Container Insights + managed Prometheus; your own client-readiness item 7. The experiment landed on 1× D4as_v7, monitoring off (with the 12-disk attach ceiling). Worth deciding the default before merge — the README documents what the template does today.
2. mirror-images.sh:64-66 — the "RETRY" branch runs the identical command (only the >/dev/null is gone), so the comment over-promises; either drop it or make the retry differ. Also forgejo:15.0.5-rootless — prod runs 15.0.6; worth re-pinning against the seed vintage you cut from.
3. entra-forgejo-oidc.sh:37 reads the Forgejo admin password into ADMIN_PW and never uses it (:54 reserved). Least privilege: drop until the REST fallback exists.
4. servers.bicep:97/122 — for (idx, j) in serverIndexes names the item idx and the index j; it renders correctly (I traced pip/nic/vm/zones/cloudInit), but the inverted names will trip the next reader.
5. aks-cost-experiment.md carries a subscription ID + the customer hostname, and records a torn-down platform — fine on the private forge; flagging in case you'd rather scrub the ID or move it under docs/.
6. Commit history: four same-subject commits (b994d05/3c3d1f4/48f1c4b, de26448/45b8e14/e52a995) — squash-merge as you suggest.

If any README wording doesn't match what the aks profile actually does, say so here and I'll fix it.

— Claude (for trevato), via ci-bot

Pushed `5747ef0` to this branch: **`deploy/azure/README.md`** — one Azure guide for both modes, written in ASD-STE100 Simplified Technical English (trevato's ask): resources each mode creates, numbered steps, `servers.bicep` expansion, Entra sign-in, known limits, parameters/outputs, cost, teardown. It folds `README-aks.md` in (removed — one doc, no drift) and links from `deploy/README.md`. Also set the exec bit on `mirror-images.sh` / `entra-forgejo-oidc.sh` as the PR body asked. Two things I changed in the runbook vs. `README-aks.md` — please check them against what you ran: - The germinate step now carries **`AKS_RG=<rg>`** — `bin/germinate` `:?`-requires it under `CLUSTER_PROFILE=aks` (it derives subscription + cert-manager client ID from the RG), so the old command would have stopped at the guard. - Mirror-before-germinate is now an explicit instruction (the `aks-image-redirect` comment in `lib.sh` says redirected pods ErrImagePull otherwise). Review notes on the rest of the branch (read-only — no `az`/`bicep` here, so no compile; nothing below blocks merge): 1. **AKS defaults resurrect the $364/mo baseline** — `main.bicep:80/83/346` still pin 2× `D4ds_v6` + Container Insights + managed Prometheus; your own client-readiness item 7. The experiment landed on 1× `D4as_v7`, monitoring off (with the 12-disk attach ceiling). Worth deciding the default before merge — the README documents what the template does today. 2. `mirror-images.sh:64-66` — the "RETRY" branch runs the identical command (only the `>/dev/null` is gone), so the comment over-promises; either drop it or make the retry differ. Also `forgejo:15.0.5-rootless` — prod runs 15.0.6; worth re-pinning against the seed vintage you cut from. 3. `entra-forgejo-oidc.sh:37` reads the Forgejo admin password into `ADMIN_PW` and never uses it (`:54` reserved). Least privilege: drop until the REST fallback exists. 4. `servers.bicep:97/122` — `for (idx, j) in serverIndexes` names the *item* `idx` and the *index* `j`; it renders correctly (I traced pip/nic/vm/zones/cloudInit), but the inverted names will trip the next reader. 5. `aks-cost-experiment.md` carries a subscription ID + the customer hostname, and records a torn-down platform — fine on the private forge; flagging in case you'd rather scrub the ID or move it under `docs/`. 6. Commit history: four same-subject commits (`b994d05`/`3c3d1f4`/`48f1c4b`, `de26448`/`45b8e14`/`e52a995`) — squash-merge as you suggest. If any README wording doesn't match what the aks profile actually does, say so here and I'll fix it. — Claude (for trevato), via ci-bot

trevato commented 2026-08-19 15:19:29 +00:00

Owner

Copy link

Re-render check you asked for in the PR body — done with bicep 0.39.0 (via nix), head 5747ef0:

• main.bicep and servers.bicep both compile, 0 errors. main.bicep emits the expected BCP318 "may be null" warnings on the if (mode == …) resources — benign, every access is behind the same condition or a mode ternary.
• mode=vms ARM is byte-identical to main for all five resources (NSG, VNet, public IP, NIC, VM bodies — diffed the compiled JSON per type, ignoring the added condition). Only variable change is nodeCount gaining the mode guard; outputs are the same expressions under a mode ternary.
• One default changed: adminSshPublicKey was required, now defaults to '' (so aks can omit it). A vms deploy that forgets the key now fails at VM creation instead of at parameter validation — acceptable, just noting it.
• bin/lib.sh delta is one self-contained function (trim_gitops_aks, +346/−0); bin/germinate changes are gated on CLUSTER_PROFILE=aks — default k3s path untouched, and the coldstart gate is green on this head.

From my side this is good to squash-merge; the items in my earlier comment are follow-ups, not blockers.

— Claude (for trevato)

Re-render check you asked for in the PR body — done with `bicep` 0.39.0 (via nix), head `5747ef0`: - `main.bicep` and `servers.bicep` both compile, **0 errors**. `main.bicep` emits the expected BCP318 "may be null" warnings on the `if (mode == …)` resources — benign, every access is behind the same condition or a mode ternary. - **`mode=vms` ARM is byte-identical to `main`** for all five resources (NSG, VNet, public IP, NIC, VM bodies — diffed the compiled JSON per type, ignoring the added `condition`). Only variable change is `nodeCount` gaining the `mode` guard; outputs are the same expressions under a `mode` ternary. - One default changed: `adminSshPublicKey` was required, now defaults to `''` (so `aks` can omit it). A `vms` deploy that forgets the key now fails at VM creation instead of at parameter validation — acceptable, just noting it. - `bin/lib.sh` delta is one self-contained function (`trim_gitops_aks`, +346/−0); `bin/germinate` changes are gated on `CLUSTER_PROFILE=aks` — default k3s path untouched, and the coldstart gate is green on this head. From my side this is good to squash-merge; the items in my earlier comment are follow-ups, not blockers. — Claude (for trevato)

trevato merged commit fd01a7e8cb into main 2026-08-19 15:22:17 +00:00

trevato referenced this pull request from a commit 2026-08-19 15:22:18 +00:00

deploy/azure: mode=vms|aks in main.bicep, HA servers carve-out, AKS germinate profile (#139)

trevato referenced this pull request 2026-08-19 16:07:11 +00:00

deploy/azure: README truth pass (follow-up to #139) #141

trevato referenced this pull request from a commit 2026-08-19 16:08:12 +00:00

deploy/azure: README truth pass (follow-up to #139) (#141)

benjivers referenced this pull request 2026-08-25 20:37:11 +00:00

AWS / Azure-AKS / EKS cloud-deploy build history #221

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something is not working

  

discussion

An idea that is still being discussed

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

goal

  

help wanted

Need some help

  

horizon:backlog

  

horizon:governance

  

horizon:mvp

  

invalid

Something is wrong

  

operator-decision

Blocked on the owner's call — no agent work until decided

  

question

More information is needed

  

roadmap

  

wontfix

This won't be fixed

No labels

bug

discussion

duplicate

enhancement

goal

help wanted

horizon:backlog

horizon:governance

horizon:mvp

invalid

operator-decision

question

roadmap

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-platform/mitosis!139

No description provided.