The snapshot store tries again: five attempts with doubling waits, and a failure names its key

plat commented

2026-10-12 22:30:52 +00:00

Contributor

The forge's nightly snapshot has not landed since 2026-09-29: the in-cluster MinIO it wrote to stopped accepting its credentials at the move, and the Backups page only said the store "could not be listed just now". Tonight the destination was sealed to Backblaze (off the box at last) and a snapshot taken by hand; it wrote a dozen objects and then failed on one put with S3Error: an unexpected error has occurred and no key named. The same write from the same host, with the same Bun 1.3.14 and the same credentials, succeeded a minute later, in every size and in parallel: an object store's ordinary 5xx, and a store that gave up on the first one.

withRetries wraps the S3 store: every call is tried five times with waits of 0.5, 1, 2 and 4 seconds, and what still fails says which key and why. test/snapshot-store.test.ts (new, in the gate) pins the retries, the waits and the message against a store that fails twice and one that never works.

Toward trevato/forge#1147 (recovery you can print on a notecard): with this live, "Snapshot now" is retried on the Backups page and the first restore drill runs from it.

🤖 Generated with Claude Code

Replaces #23, whose gate checked out a stale sha after the pull was reopened (fatal: reference is not a tree).

The forge's nightly snapshot has not landed since 2026-09-29: the in-cluster MinIO it wrote to stopped accepting its credentials at the move, and the Backups page only said the store "could not be listed just now". Tonight the destination was sealed to Backblaze (off the box at last) and a snapshot taken by hand; it wrote a dozen objects and then failed on one `put` with `S3Error: an unexpected error has occurred` and no key named. The same write from the same host, with the same Bun 1.3.14 and the same credentials, succeeded a minute later, in every size and in parallel: an object store's ordinary 5xx, and a store that gave up on the first one. `withRetries` wraps the S3 store: every call is tried five times with waits of 0.5, 1, 2 and 4 seconds, and what still fails says which key and why. `test/snapshot-store.test.ts` (new, in the gate) pins the retries, the waits and the message against a store that fails twice and one that never works. Toward trevato/forge#1147 (recovery you can print on a notecard): with this live, "Snapshot now" is retried on the Backups page and the first restore drill runs from it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Replaces #23, whose gate checked out a stale sha after the pull was reopened (`fatal: reference is not a tree`).

plat added 1 commit

2026-10-12 22:30:52 +00:00

The snapshot store tries again: five attempts with doubling waits, and a failure names its key

check / gate (1) (pull_request) Successful in 13m12s

Details

check / gate (2) (pull_request) Successful in 10m5s

Details

check / gate (3) (pull_request) Successful in 13m58s

Details

check / check (pull_request) Successful in 1s

Details

preview / preview (pull_request) Successful in 6s

Details

e583ec4e4a

Tonight's first snapshot to Backblaze failed on one put with 'S3Error: an
unexpected error has occurred' and no key named; the same write from the
same host and Bun version succeeded moments later. Object stores answer a
burst or a moment of maintenance with a 5xx. Every store call is now tried
five times with waits of 0.5, 1, 2 and 4 seconds, and what still fails says
which key and why. test/snapshot-store.test.ts pins the retries, the waits
and the message.

plat referenced this pull request

2026-10-12 22:30:53 +00:00

The snapshot store tries again: five attempts with doubling waits, and a failure names its key #23

plat merged commit e303edc23a into main

2026-10-13 00:03:19 +00:00

plat referenced this pull request from a commit

2026-10-13 00:05:27 +00:00

Release 0.7.62: the snapshot store tries again, a kept clone's remotes are set right on every visit

Rows
Columns

The snapshot store tries again: five attempts with doubling waits, and a failure names its key #25