open-platform/mitosis
3
0
Fork 0
Code Issues 27 Pull requests 9 Releases 4 Packages 1 Wiki Activity Actions

robustness polish: reviewer waitForPreview locks onto stale PR sha → ~10min spurious fallback on fresh clusters #57

New issue
Closed
opened 2026-07-01 18:23:00 +00:00 by plat · 2 comments
plat commented 2026-07-01 18:23:00 +00:00
Owner
Copy link

Found during the agents 0.7.4 first-try-ship validation (wb-rob2). The dispatcher reviewer flow: startReview→waitForPreview locked its liveness probe onto an EARLIER PR sha and waited the full ~10min fallback before logging "preview liveness unconfirmed; running reviewer anyway" and proceeding — while the preview was actually serving 200 with the correct /version sha the whole time. Self-heals (app ships correctly), but adds a ~10min spurious delay per build on fresh/slow clusters. For sellable-grade smoothness: fix the probe to track the current PR head sha (not a stale one) and/or shorten the fallback when the preview is already 200. Non-blocking backlog polish.

Found during the agents 0.7.4 first-try-ship validation (wb-rob2). The dispatcher reviewer flow: startReview→waitForPreview locked its liveness probe onto an EARLIER PR sha and waited the full ~10min fallback before logging "preview liveness unconfirmed; running reviewer anyway" and proceeding — while the preview was actually serving 200 with the correct /version sha the whole time. Self-heals (app ships correctly), but adds a ~10min spurious delay per build on fresh/slow clusters. For sellable-grade smoothness: fix the probe to track the current PR head sha (not a stale one) and/or shorten the fallback when the preview is already 200. Non-blocking backlog polish.
plat commented 2026-07-02 02:13:13 +00:00
Author
Owner
Copy link

Root-caused live on a fresh daughter (plat.local, agents 0.7.5): the ~10-min stall is NOT sha-locking — probePreview curls --resolve <host>:443:127.0.0.1, and from the agents POD 127.0.0.1 is the pod itself, so the liveness probe gets connection-refused on every poll and every review burns the full 10-min fallback. Plain in-cluster DNS (the CoreDNS rewrite germinate writes) returns 200 with /version == head sha. Fix: plat/agents#11 — DNS-first vantage with loopback fallback (dev mode preserved). Validated from both vantages on the live daughter.

Root-caused live on a fresh daughter (plat.local, agents 0.7.5): the ~10-min stall is NOT sha-locking — `probePreview` curls `--resolve <host>:443:127.0.0.1`, and from the agents POD 127.0.0.1 is the pod itself, so the liveness probe gets connection-refused on every poll and every review burns the full 10-min fallback. Plain in-cluster DNS (the CoreDNS rewrite germinate writes) returns 200 with `/version` == head sha. Fix: plat/agents#11 — DNS-first vantage with loopback fallback (dev mode preserved). Validated from both vantages on the live daughter.
plat commented 2026-07-02 13:06:29 +00:00
Author
Owner
Copy link

Fixed in agents v0.7.6 (plat/agents#11 → release #13, on prod + in the genesis seed). Root cause was the loopback-only probe, not sha-locking. Validated live twice: fresh daughter reviewer engaged in seconds after a 21-min build, zero liveness-unconfirmed lines (val076 run 3, plat/mitosis#66).

Fixed in agents v0.7.6 (plat/agents#11 → release #13, on prod + in the genesis seed). Root cause was the loopback-only probe, not sha-locking. Validated live twice: fresh daughter reviewer engaged in seconds after a 21-min build, zero liveness-unconfirmed lines (val076 run 3, plat/mitosis#66).
plat closed this issue 2026-07-02 13:06:29 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
reseed/r1043
proposal-the-wardrobe
seed-embedded-etcd
docs-app-roles
docs/aws-eks-build-history
audit/aws-2026-08-25
azure-docs-truth-pass
aks-coredns-hairpin-fix
fix/aks-ground-trust-cidrs
fix/aks-cert-zone-doubling
fix/azure-gate-nixos-bicep
feat/azure-gate
azure-aks-mode
feat/deploy-platform
feat/seed-release
seed/packages-integration-spec
feat/seed-workflow
resilience/standby-site
resilience/backup-born-on
fix/germinate-timeout-contention
feat/reseed-noop-guard
fix/reseed-setg
fix/reseed-script
fix/reseed-yaml
feat/reseed-workflow
fix/seed-createdfrom-durable
feat/gate-parallel-nightly
fix/gate-a4-detection
feat/fork-on-germinate
feat/ci-1-coldstart-gate
chore/reseed-genesis-post-61
fix/germinate-prerelease-tags
feat/wb-gate-seed-refresh
feat/dom-2-cilium-fqdn-egress
feat/rbac-5-webhook-hmac-seal
feat/robust-seed-refresh
feat/rbac-7-postgres-rls-design
feat/sec-3-secrets-threat-model
feat/dom-1a-fork-reseal-cloudflare-token
feat/sec-1-age-key-rotation
fix/germinate-python3-runtimedep
docs/pol-0-policy-governance
docs/hyb-3-opentofu-substrate
chore/refresh-genesis-seed
docs/enterprise-platform
docs/design-proposals
feat/observability-grafana
fix/cold-start-seed-branch-and-mtu
seed-r897
seed-r886
seed-r873
seed-r829
Labels
Clear labels   
bug
Something is not working

  
discussion
An idea that is still being discussed

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
goal

   
help wanted
Need some help

  
horizon:backlog

   
horizon:governance

   
horizon:mvp

   
invalid
Something is wrong

  
operator-decision
Blocked on the owner's call — no agent work until decided

  
question
More information is needed

  
roadmap

   
wontfix
This won't be fixed

No labels
bug
discussion
duplicate
enhancement
goal
help wanted
horizon:backlog
horizon:governance
horizon:mvp
invalid
operator-decision
question
roadmap
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
open-platform/mitosis#57
No description provided.