A preview builds on the droplet runner, as the release does: vxrail's push of the ~450 MB layer dies at the registry ingress (499) #58

Merged

grokbot merged 1 commit from dev/preview-runs-on-droplet into main

2026-10-18 01:59:43 +00:00

grokbot commented

2026-10-18 01:50:46 +00:00

Owner

The preview job ran on runs-on: isolated. Two runners serve that label, and the job failed every time vxrail picked it up.

Evidence

Every failed image push ran on plat-standby-forgejo-runner-vxrail: runs 900, 906, 919, 925, 938, 949, 951, 953, 955, 968, 990, 1024 and 1026.
Each one fails the same way: one monolithic PUT https://git.open-platform.sh/v2/trevato-apps/forge/blobs/uploads/<id>?digest=sha256:c7604faa… returns 499 Client Closed Request. Run 968 also got a 500 at 63.94 s before a retry.
Each attempt is cut off at about 63 s. Buildkit then retries, so "pushing layers" comes to 62.4 s, 126 s, 189.6 s or 256.9 s.
c7604faa looks like vxrail's cached apt/chromium layer (about 450 MB). The whole image is 718 MiB.
The green previews (947 #46, 980 #49) ran on plat-standby-forgejo-runner-isolated and pushed in 0.5 to 1.3 s.
The release build hit the same 499 on 2026-10-15 and moved to runs-on: droplet in 8e2125c4. This PR does the same for the preview.

Lasting infra alternative (platform side): take the isolated label off the vxrail runner, or raise the registry ingress timeout (about 60 s now) or its body handling for blob PUTs.

The `preview` job ran on `runs-on: isolated`. Two runners serve that label, and the job failed every time vxrail picked it up. **Evidence** - Every failed image push ran on `plat-standby-forgejo-runner-vxrail`: runs 900, 906, 919, 925, 938, 949, 951, 953, 955, 968, 990, 1024 and 1026. - Each one fails the same way: one monolithic `PUT https://git.open-platform.sh/v2/trevato-apps/forge/blobs/uploads/<id>?digest=sha256:c7604faa…` returns `499 Client Closed Request`. Run 968 also got a `500` at 63.94 s before a retry. - Each attempt is cut off at about 63 s. Buildkit then retries, so "pushing layers" comes to 62.4 s, 126 s, 189.6 s or 256.9 s. - c7604faa looks like vxrail's cached apt/chromium layer (about 450 MB). The whole image is 718 MiB. - The green previews (947 #46, 980 #49) ran on `plat-standby-forgejo-runner-isolated` and pushed in 0.5 to 1.3 s. - The release build hit the same 499 on 2026-10-15 and moved to `runs-on: droplet` in 8e2125c4. This PR does the same for the preview. **Lasting infra alternative (platform side):** take the `isolated` label off the vxrail runner, or raise the registry ingress timeout (about 60 s now) or its body handling for blob PUTs.

grokbot added 1 commit

2026-10-18 01:50:47 +00:00

A preview builds on the droplet runner, as the release does: vxrail's push of the ~450 MB layer dies at the registry ingress (499)

check / gate (2) (pull_request) Successful in 18s

Details

check / gate (1) (pull_request) Successful in 23s

Details

check / gate (3) (pull_request) Successful in 23s

Details

check / check (pull_request) Successful in 1s

Details

preview / preview (pull_request) Successful in 17s