Pack transfers at once follow the host's memory, so a 512 MiB cell does not run sixteen pushes and their hooks past its limit #61

Merged

grokbot merged 1 commit from dev/transfer-gate-memory into main

2026-10-18 05:10:47 +00:00

grokbot commented

2026-10-18 04:30:35 +00:00

Owner

Failing: test/agents.test.ts › fifty agents at once › fifty pushes to distinct branches all complete over the hook socket. It timed out after 180000 ms, and the pushes were rejected with remote: forge: hook relay failed (TimeoutError: The operation timed out.) / ! [remote rejected] agent-N (pre-receive hook declined) / curl 18 transfer closed.

Runner: plat-standby-forgejo-runner-isolated-598cbbb49c-jvj8r (memory.max 512 MiB, swap 66 MiB, cpu.max 50000/100000, nproc 4). mem_peak reached the limit in every run there.

Not caused by #49. gate (1) results for this test, by runner:

vxrail (16 GiB): 3–6 s in every run.
isolated: passes in 40–56 s on main 6ec6c1c9, 18917b93, 006d3977 and bde2a088, and on #46 ee9e9234, whose tree is identical to main cfb4cf16 (run 1065, 41.4 s). It fails at the 120 s relay timeout on #42 (1042, 1046) and on #49 (1072, 1074). The outcome is close to a coin flip on that runner, and the code under test does not change it: #49's diff touches nothing on the push path. Locally, bde2a088, cfb4cf16 and 0f4fc22b all run it in about 2.3 s.

Root cause: the receive-pack transfer gate is fixed at 16, whatever the memory. Each push holds receive-pack plus a pre-receive hook, which is a whole Bun runtime. Measured locally at 16: 13 hooks and 14 receive-packs held at once, 320 MB RSS. The fifty clients come to about 1 GB RSS but only about 120 MB PSS, because most of that is shared pages. So on a 512 MiB cell, the server side is what pushes memory past the limit.

Fix: the default is one transfer per 64 MiB of the cgroup/host memory, clamped to 2–16 (memoryLimitBytes). That gives 8 on 512 MiB and leaves every host with 1 GiB or more at 16. FORGE_GIT_TRANSFER_CONCURRENCY still overrides it. A queued push waits for its permit before its hook starts, so the wait does not count against the relay's deadline. With 8, the measured peak is 7 hooks and 8 receive-packs, 155 MB, and the time is unchanged (2.1 s). The fifty-push test is unchanged, and a unit test covers transfersForMemory. tsc --noEmit is clean, and agents.test.ts is 29/29.

**Failing:** `test/agents.test.ts` › *fifty agents at once › fifty pushes to distinct branches all complete over the hook socket*. It timed out after 180000 ms, and the pushes were rejected with `remote: forge: hook relay failed (TimeoutError: The operation timed out.)` / `! [remote rejected] agent-N (pre-receive hook declined)` / `curl 18 transfer closed`. **Runner:** `plat-standby-forgejo-runner-isolated-598cbbb49c-jvj8r` (memory.max 512 MiB, swap 66 MiB, cpu.max 50000/100000, nproc 4). `mem_peak` reached the limit in every run there. **Not caused by #49.** gate (1) results for this test, by runner: - vxrail (16 GiB): 3–6 s in every run. - isolated: passes in 40–56 s on main 6ec6c1c9, 18917b93, 006d3977 and bde2a088, and on #46 ee9e9234, whose tree is identical to main cfb4cf16 (run 1065, 41.4 s). It fails at the 120 s relay timeout on #42 (1042, 1046) and on #49 (1072, 1074). The outcome is close to a coin flip on that runner, and the code under test does not change it: #49's diff touches nothing on the push path. Locally, bde2a088, cfb4cf16 and 0f4fc22b all run it in about 2.3 s. **Root cause:** the `receive-pack` transfer gate is fixed at 16, whatever the memory. Each push holds `receive-pack` plus a pre-receive hook, which is a whole Bun runtime. Measured locally at 16: 13 hooks and 14 receive-packs held at once, 320 MB RSS. The fifty clients come to about 1 GB RSS but only about 120 MB PSS, because most of that is shared pages. So on a 512 MiB cell, the server side is what pushes memory past the limit. **Fix:** the default is one transfer per 64 MiB of the cgroup/host memory, clamped to 2–16 (`memoryLimitBytes`). That gives 8 on 512 MiB and leaves every host with 1 GiB or more at 16. `FORGE_GIT_TRANSFER_CONCURRENCY` still overrides it. A queued push waits for its permit before its hook starts, so the wait does not count against the relay's deadline. With 8, the measured peak is 7 hooks and 8 receive-packs, 155 MB, and the time is unchanged (2.1 s). The fifty-push test is unchanged, and a unit test covers `transfersForMemory`. `tsc --noEmit` is clean, and agents.test.ts is 29/29.

grokbot added 1 commit

2026-10-18 04:30:35 +00:00

Pack transfers at once follow the host's memory: one per 64 MiB, two to sixteen, so a 512 MiB cell does not run sixteen pushes and their hooks past its limit

check / gate (3) (pull_request) Successful in 4m5s

Details

check / gate (2) (pull_request) Successful in 5m25s

Details

check / gate (1) (pull_request) Successful in 24m4s

Details

check / check (pull_request) Successful in 1s

Details

preview / preview (pull_request) Successful in 22s