The hollow audit fits a 512 MiB check runner: it collects each file's syntax tree as it goes, and the command's test runs in its own process #44

Merged

grokbot merged 2 commits from grokbot/forge:fix/hollow-test-memory into main 2026-10-17 21:27:55 +00:00

Conversation 0 Commits 2 Files changed 3 +47 -18

grokbot commented 2026-10-17 19:19:30 +00:00

Owner

Copy link

Stacked on fix/ci-fork-pr-checkout (#40): its commit is in this diff until it merges.

On main's check, test/hollow.test.ts was the gate's most frequent red: shard 2
OOM-killed at it twice (runs 874 and 883, exit 137 with memory.current pinned
at memory.max 512 MiB) and once swapped until "bun scripts/hollow.ts says so
and exits 0" took 450 s against its 60 s timeout (run 859; spawnSync blocked
the timer).

Two causes, both fixed without touching what the tests assert:

• scripts/hollow.ts parses 838 files with the TypeScript API and dropped each
  tree as garbage, but nothing collected it, so one audit peaked at ~442 MiB
  RSS for ~20 MiB of live heap. It now runs a full collection every 64 files:
  ~270 MiB peak, same result, no measurable time cost.
• The command's test spawned a second whole-tree audit beside the test
  process's own two audits. It moves to test/hollow-cli.test.ts, whose
  process audits nothing itself, and awaits Bun.spawn so its timeout can end
  a stuck child. The guard test now keeps both files out of EXCLUDED.

Peak memory of the whole process tree (PSS summed over test-shards, bun test
and the child, sampled every 50 ms), bun 1.3.14, FORGE_CHECK_SHARDS=1:
test/hollow.test.ts on main 504 MiB of the runner's 512; after, 276 MiB for
test/hollow.test.ts and 257 MiB for test/hollow-cli.test.ts. The audit
alone: ~442 MiB RSS before, ~270 MiB after.

Evidence

pipeline/status keeps one main event per finished push gate. Over the last 37 runs main was green in 22 (59%). test/hollow.test.ts is its most frequent single cause: OOM-killed in gate shard 2 in runs 874 and 883 (shard 20/160 done (exit 137), mem_current=536870912), and swapped until it timed out in run 859 ("bun scripts/hollow.ts says so and exits 0" timed out after 60000ms, 449,848 ms elapsed).

Not loosened

No assertion, timeout or exclusion changes. The command's test moves to its own file unchanged, apart from Bun.spawn + await replacing spawnSync, so its 60 s timeout can fire.

Validation note

The fast tier's other reds are not this change. They show up in the table below as FLAKY or BASELINE:

• vitals "errors are counted…" and tui-sentinel "an admin has it…" fail only beside test/drill-board-line.test.ts in one process. That file leaks a notifier pager, fixed on branch fix/tests-close-notifiers.
• contain "an undeclared public host…" fails only on the validation box, whose DNS returns 198.18.x.
• following, tui-workflows and usage-count pass when run alone on this branch: FLAKY, cause not diagnosed yet.
• Gate (2) stopped at following (fail-fast) before reaching the hollow files. The same commit's earlier validation, before it was stacked on #40 (9d3fe1f6, which differs only in the workflow files), ran gate (2) green through test/hollow.test.ts and test/hollow-cli.test.ts.

Validation

pipeline/validate on bc87716f: flaky in 476s — bun 1.3.14, CI-shaped env (scrubbed vars, runner PATH), exact check.yml shard + forge.json commands.

step	result	time
bun install --frozen-lockfile	ok (exit 0)	0s
forge.json check.fast	FLAKY (exit 1)	110s
check.yml gate (1) pull_request	BASELINE (exit 1)	22s
check.yml gate (2) pull_request	FLAKY (exit 1)	27s
check.yml gate (3) pull_request	ok (exit 0)	317s

• ⚠️ node v20.19.2 here, v22 on CI (tsc runs under node)
• ⚠️ check.yml gate (1) pull_request stopped early at a BASELINE failure (fail-fast): its later files were not run in this step
• ⚠️ check.yml gate (2) pull_request stopped early at a FLAKY failure (fail-fast): its later files were not run in this step

test output (failures, totals)

▶ bun install --frozen-lockfile
  ok  exit 0  0s
▶ forge.json check.fast
(fail) the sentinel command > an admin has it, and it says what the board says [16.23ms] [shard 1]
(fail) egress, reported: an undeclared public host goes through and is counted; a private one never does > an undeclared public host is let through and counted [0.89ms] [shard 10]
Ran 3064 tests across 303 files in 103.8s; 2 failed.
classify: reran test/contain.test.ts test/tui-sentinel.test.ts test/tui-workflows.test.ts test/usage-count.test.ts  alone on origin/main 1051d451 and on this branch
classify: not red on main:
  FLAKY  exit 1  110s
▶ check.yml gate (1) pull_request
(fail) egress, reported: an undeclared public host goes through and is counted; a private one never does > an undeclared public host is let through and counted [1.74ms] [shard 3]
Ran 110 tests across 3 files in 20.7s; 1 failed.
classify: reran test/contain.test.ts  alone on origin/main 1051d451 and on this branch
  BASELINE  exit 1  22s
▶ check.yml gate (2) pull_request
(fail) a fork that follows its parent > a proposal names only what is actually ahead: work the parent already has is not listed again [654.26ms] [shard 3]
Ran 61 tests across 3 files in 23.8s; 1 failed.
classify: reran test/following.test.ts  alone on origin/main 1051d451 and on this branch
classify: not red on main:
  FLAKY  exit 1  27s
▶ check.yml gate (3) pull_request
Ran 1721 tests across 165 files in 317.2s (165 shards).
  ok  exit 0  317s

Stacked on `fix/ci-fork-pr-checkout` (#40): its commit is in this diff until it merges. On main's check, test/hollow.test.ts was the gate's most frequent red: shard 2 OOM-killed at it twice (runs 874 and 883, exit 137 with memory.current pinned at memory.max 512 MiB) and once swapped until "`bun scripts/hollow.ts` says so and exits 0" took 450 s against its 60 s timeout (run 859; spawnSync blocked the timer). Two causes, both fixed without touching what the tests assert: - scripts/hollow.ts parses 838 files with the TypeScript API and dropped each tree as garbage, but nothing collected it, so one audit peaked at ~442 MiB RSS for ~20 MiB of live heap. It now runs a full collection every 64 files: ~270 MiB peak, same result, no measurable time cost. - The command's test spawned a second whole-tree audit beside the test process's own two audits. It moves to test/hollow-cli.test.ts, whose process audits nothing itself, and awaits Bun.spawn so its timeout can end a stuck child. The guard test now keeps both files out of EXCLUDED. Peak memory of the whole process tree (PSS summed over test-shards, bun test and the child, sampled every 50 ms), bun 1.3.14, FORGE_CHECK_SHARDS=1: test/hollow.test.ts on main 504 MiB of the runner's 512; after, 276 MiB for test/hollow.test.ts and 257 MiB for test/hollow-cli.test.ts. The audit alone: ~442 MiB RSS before, ~270 MiB after. ### Evidence `pipeline/status` keeps one `main` event per finished push gate. Over the last 37 runs main was green in 22 (59%). `test/hollow.test.ts` is its most frequent single cause: OOM-killed in gate shard 2 in runs 874 and 883 (`shard 20/160 done (exit 137)`, `mem_current=536870912`), and swapped until it timed out in run 859 (`"`bun scripts/hollow.ts` says so and exits 0" timed out after 60000ms`, 449,848 ms elapsed). ### Not loosened No assertion, timeout or exclusion changes. The command's test moves to its own file unchanged, apart from `Bun.spawn` + `await` replacing `spawnSync`, so its 60 s timeout can fire. ### Validation note The fast tier's other reds are not this change. They show up in the table below as FLAKY or BASELINE: - `vitals` "errors are counted…" and `tui-sentinel` "an admin has it…" fail only beside `test/drill-board-line.test.ts` in one process. That file leaks a notifier pager, fixed on branch `fix/tests-close-notifiers`. - `contain` "an undeclared public host…" fails only on the validation box, whose DNS returns 198.18.x. - `following`, `tui-workflows` and `usage-count` pass when run alone on this branch: FLAKY, cause not diagnosed yet. - Gate (2) stopped at `following` (fail-fast) before reaching the hollow files. The same commit's earlier validation, before it was stacked on #40 (`9d3fe1f6`, which differs only in the workflow files), ran gate (2) green through `test/hollow.test.ts` and `test/hollow-cli.test.ts`. <!-- pipeline:validation --> ### Validation `pipeline/validate` on `bc87716f`: **flaky** in 476s — bun 1.3.14, CI-shaped env (scrubbed vars, runner PATH), exact check.yml shard + forge.json commands. | step | result | time | |---|---|---| | `bun install --frozen-lockfile` | ok (exit 0) | 0s | | `forge.json check.fast` | FLAKY (exit 1) | 110s | | `check.yml gate (1) pull_request` | BASELINE (exit 1) | 22s | | `check.yml gate (2) pull_request` | FLAKY (exit 1) | 27s | | `check.yml gate (3) pull_request` | ok (exit 0) | 317s | - ⚠️ node v20.19.2 here, v22 on CI (tsc runs under node) - ⚠️ check.yml gate (1) pull_request stopped early at a BASELINE failure (fail-fast): its later files were not run in this step - ⚠️ check.yml gate (2) pull_request stopped early at a FLAKY failure (fail-fast): its later files were not run in this step <details><summary>test output (failures, totals)</summary> ``` ▶ bun install --frozen-lockfile ok exit 0 0s ▶ forge.json check.fast (fail) the sentinel command > an admin has it, and it says what the board says [16.23ms] [shard 1] (fail) egress, reported: an undeclared public host goes through and is counted; a private one never does > an undeclared public host is let through and counted [0.89ms] [shard 10] Ran 3064 tests across 303 files in 103.8s; 2 failed. classify: reran test/contain.test.ts test/tui-sentinel.test.ts test/tui-workflows.test.ts test/usage-count.test.ts alone on origin/main 1051d451 and on this branch classify: not red on main: FLAKY exit 1 110s ▶ check.yml gate (1) pull_request (fail) egress, reported: an undeclared public host goes through and is counted; a private one never does > an undeclared public host is let through and counted [1.74ms] [shard 3] Ran 110 tests across 3 files in 20.7s; 1 failed. classify: reran test/contain.test.ts alone on origin/main 1051d451 and on this branch BASELINE exit 1 22s ▶ check.yml gate (2) pull_request (fail) a fork that follows its parent > a proposal names only what is actually ahead: work the parent already has is not listed again [654.26ms] [shard 3] Ran 61 tests across 3 files in 23.8s; 1 failed. classify: reran test/following.test.ts alone on origin/main 1051d451 and on this branch classify: not red on main: FLAKY exit 1 27s ▶ check.yml gate (3) pull_request Ran 1721 tests across 165 files in 317.2s (165 shards). ok exit 0 317s ``` </details>

grokbot added 2 commits 2026-10-17 19:19:30 +00:00

A pull from a fork gets its check: the gate fetches refs/pull/<n>/head when the head is not on a branch, and the preview skips forks ... a979756022

check.yml cloned this repository and checked out GITHUB_SHA. A fork's head
commit is here only as refs/pull/<n>/head, which a clone does not fetch, so
every gate shard of a fork pull died at "clone source at this ref" with
"fatal: reference is not a tree" before running a test (PR #39, run 893).

preview.yml cloned --branch "$GITHUB_HEAD_REF", a branch that exists only in
the fork; a fork pull gets no secrets either, and a preview builds and pushes
an image of the pull's code, so it now skips pulls from forks.

The hollow audit fits a 512 MiB check runner: it collects each file's syntax tree as it goes, and the command's test runs in its own process ... bc87716fd1

On main's check, test/hollow.test.ts was the gate's most frequent red: shard 2
OOM-killed at it twice (runs 874 and 883, exit 137 with memory.current pinned
at memory.max 512 MiB) and once swapped until "`bun scripts/hollow.ts` says so
and exits 0" took 450 s against its 60 s timeout (run 859; spawnSync blocked
the timer).

Two causes, both fixed without touching what the tests assert:
- scripts/hollow.ts parses 838 files with the TypeScript API and dropped each
  tree as garbage, but nothing collected it, so one audit peaked at ~442 MiB
  RSS for ~20 MiB of live heap. It now runs a full collection every 64 files:
  ~270 MiB peak, same result, no measurable time cost.
- The command's test spawned a second whole-tree audit beside the test
  process's own two audits. It moves to test/hollow-cli.test.ts, whose
  process audits nothing itself, and awaits Bun.spawn so its timeout can end
  a stuck child. The guard test now keeps both files out of EXCLUDED.

Peak memory of the whole process tree (PSS summed over test-shards, bun test
and the child, sampled every 50 ms), bun 1.3.14, FORGE_CHECK_SHARDS=1:
test/hollow.test.ts on main 504 MiB of the runner's 512; after, 276 MiB for
test/hollow.test.ts and 257 MiB for test/hollow-cli.test.ts. The audit
alone: ~442 MiB RSS before, ~270 MiB after.

grokbot referenced this pull request 2026-10-17 19:47:33 +00:00

The ring's 31-day test holds its 5 s budget scaled to this machine, measured against plain inserts of the same points #48

grokbot referenced this pull request from a commit 2026-10-17 20:16:10 +00:00

The ring's 31-day test holds its 5 s budget scaled to this machine, measured against plain inserts of the same points

grokbot added 1 commit 2026-10-17 20:16:28 +00:00

Merge branch 'main' into fix/hollow-test-memory 6ccf361228

grokbot merged commit 0a79877b0c into main 2026-10-17 21:27:55 +00:00

grokbot referenced this pull request from a commit 2026-10-17 21:27:56 +00:00

Merge pull request 'The hollow audit fits a 512 MiB check runner: it collects each file's syntax tree as it goes, and the command's test runs in its own process' (#44) from grokbot/forge:fix/hollow-test-memory into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

agent-work

Queue this for an agent: it will build it and open a PR

  

agent-working

An agent is actively working on this

  

ultracode

Operators: run a Fable ultracode orchestrator on this issue

No labels

agent-work

agent-working

ultracode

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

trevato-apps/forge!44

No description provided.