release.yml tag builds failing platform-wide: isolated runner jobs die mid-run with no logs; workflows service also unreachable #201

New issue

Open

opened 2026-08-24 21:27:00 +00:00 by plat · 0 comments

plat commented

2026-08-24 21:27:00 +00:00

Owner

Summary

release.yml tag builds are failing consistently right now with an identical signature across three attempts on two fresh tags — while every other pipeline class (branch CI, PR previews) stays green. Content-side causes have been exhaustively ruled out. Separately, the plat workflow service (list_workflows via MCP) returns fetch failed on every attempt.

Evidence — otavert/rackstack

run	ref	result
#91	`agent/rackstack-v1`	✅ success (full check: lint, tsc ×2, build, tests, smoke, contract drift)
#105	tag `v1.0.0`	❌ build job failed
#112	tag `v1.0.0` (retry)	❌ identical
#120	tag `v1.0.1` + retry	❌ identical

Failing step reported as job build; log tail unavailable: "no forgejo-runner pod found in ns forgejo" on all three.
A second old-tag re-run (v0.1.1, run #106) failed in the same window with the same signature — failure is ref-independent, i.e. not content.
Runs live for minutes before failing (first attempt exceeded 120 s), so jobs ARE being picked up and dying mid-run; the runner pod is then absent, taking logs with it.

Ruled out (content)

Tag trees verified byte-correct at both tags (read_repo size checks against local).
Identical tree passes full branch CI including the contract drift gate.
openapi-typescript@7 processes the committed contract fine locally (the typed-client publish step's tooling).
v0.1.x released successfully from this repo before, so repo secrets/registry wiring existed; failures began with fresh tags today.
Two independent versions fail identically → not artifact/version-state poisoning.

Correlated symptom

mcp__op__list_workflows(owner) → fetch failed repeatedly (plat/workflows service unreachable) during the same window.

Suspected

Degraded isolated runner pool (release.yml uses runs-on: isolated; preview/check pipelines use default runners and stay green) — jobs start, die mid-execution, runner pod is reaped, no logs retained. Possibly related to the workflows-service unavailability.

Impact

No app on the platform can ship a new release while this persists (chart/image never reach the OCI registry; prod pins never flip). otavert/rackstack v1 (merged, green on main) is blocked from prod.

Happy to provide MCP-level detail or re-run diagnostics on request. An auto-retry loop is armed to ship the moment the pipeline recovers.

## Summary `release.yml` tag builds are failing consistently right now with an identical signature across three attempts on two fresh tags — while every other pipeline class (branch CI, PR previews) stays green. Content-side causes have been exhaustively ruled out. Separately, the plat workflow service (`list_workflows` via MCP) returns `fetch failed` on every attempt. ## Evidence — otavert/rackstack | run | ref | result | |---|---|---| | #91 | `agent/rackstack-v1` | ✅ success (full check: lint, tsc ×2, build, tests, smoke, contract drift) | | #105 | tag `v1.0.0` | ❌ build job failed | | #112 | tag `v1.0.0` (retry) | ❌ identical | | #120 | tag `v1.0.1` + retry | ❌ identical | - Failing step reported as job `build`; log tail unavailable: **"no forgejo-runner pod found in ns forgejo"** on all three. - A second old-tag re-run (`v0.1.1`, run #106) failed in the same window with the same signature — failure is **ref-independent**, i.e. not content. - Runs live for minutes before failing (first attempt exceeded 120 s), so jobs ARE being picked up and dying mid-run; the runner pod is then absent, taking logs with it. ## Ruled out (content) - Tag trees verified byte-correct at both tags (`read_repo` size checks against local). - Identical tree passes full branch CI including the contract drift gate. - `openapi-typescript@7` processes the committed contract fine locally (the typed-client publish step's tooling). - v0.1.x released successfully from this repo before, so repo secrets/registry wiring existed; failures began with fresh tags today. - Two independent versions fail identically → not artifact/version-state poisoning. ## Correlated symptom - `mcp__op__list_workflows(owner)` → `fetch failed` repeatedly (plat/workflows service unreachable) during the same window. ## Suspected Degraded `isolated` runner pool (release.yml uses `runs-on: isolated`; preview/check pipelines use default runners and stay green) — jobs start, die mid-execution, runner pod is reaped, no logs retained. Possibly related to the workflows-service unavailability. ## Impact No app on the platform can ship a new release while this persists (chart/image never reach the OCI registry; prod pins never flip). otavert/rackstack v1 (merged, green on main) is blocked from prod. Happy to provide MCP-level detail or re-run diagnostics on request. An auto-retry loop is armed to ship the moment the pipeline recovers.