The snapshot store tries again: five attempts with doubling waits, and a failure names its key

plat commented

2026-10-12 19:19:42 +00:00

Contributor

The forge's nightly snapshot has not landed since 2026-09-29: the in-cluster MinIO it wrote to stopped accepting its credentials at the move, and the Backups page only said the store "could not be listed just now". Tonight the destination was sealed to Backblaze (off the box at last) and a snapshot taken by hand; it wrote a dozen objects and then failed on one put with S3Error: an unexpected error has occurred and no key named. The same write from the same host, with the same Bun 1.3.14 and the same credentials, succeeded a minute later, in every size and in parallel: an object store's ordinary 5xx, and a store that gave up on the first one.

withRetries wraps the S3 store: every call is tried five times with waits of 0.5, 1, 2 and 4 seconds, and what still fails says which key and why. test/snapshot-store.test.ts (new, in the gate) pins the retries, the waits and the message against a store that fails twice and one that never works.

Toward trevato/forge#1147 (recovery you can print on a notecard): with this live, "Snapshot now" is retried on the Backups page and the first restore drill runs from it.

🤖 Generated with Claude Code

The forge's nightly snapshot has not landed since 2026-09-29: the in-cluster MinIO it wrote to stopped accepting its credentials at the move, and the Backups page only said the store "could not be listed just now". Tonight the destination was sealed to Backblaze (off the box at last) and a snapshot taken by hand; it wrote a dozen objects and then failed on one `put` with `S3Error: an unexpected error has occurred` and no key named. The same write from the same host, with the same Bun 1.3.14 and the same credentials, succeeded a minute later, in every size and in parallel: an object store's ordinary 5xx, and a store that gave up on the first one. `withRetries` wraps the S3 store: every call is tried five times with waits of 0.5, 1, 2 and 4 seconds, and what still fails says which key and why. `test/snapshot-store.test.ts` (new, in the gate) pins the retries, the waits and the message against a store that fails twice and one that never works. Toward trevato/forge#1147 (recovery you can print on a notecard): with this live, "Snapshot now" is retried on the Backups page and the first restore drill runs from it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

plat added 1 commit

2026-10-12 19:19:43 +00:00

The snapshot store tries again: five attempts with doubling waits, and a failure names its key

check / gate (1) (pull_request) Failing after 17m30s

Details

check / gate (2) (pull_request) Failing after 22s

Details

check / gate (3) (pull_request) Failing after 12s

Details

preview / preview (pull_request) Successful in 8m54s

Details

check / check (pull_request) Failing after 1s

Details

dce8f95074

Tonight's first snapshot to Backblaze failed on one put with 'S3Error: an
unexpected error has occurred' and no key named; the same write from the
same host and Bun version succeeded moments later. Object stores answer a
burst or a moment of maintenance with a 5xx. Every store call is now tried
five times with waits of 0.5, 1, 2 and 4 seconds, and what still fails says
which key and why. test/snapshot-store.test.ts pins the retries, the waits
and the message.

plat referenced this pull request

2026-10-12 22:30:52 +00:00

The snapshot store tries again: five attempts with doubling waits, and a failure names its key #25

plat commented

2026-10-12 22:30:53 +00:00

Author

Contributor

Superseded by #25: the same change on a fresh branch, because this pull's gate checked out a sha that no longer existed after the reopen.

plat closed this pull request

2026-10-12 22:30:54 +00:00