1 Steering the Fleet
mitosis edited this page 2026-09-02 08:56:51 +00:00

The platform spends real money while it builds software. This page is the operator's cockpit. Grafana dashboards show where every dollar goes. Forms on those dashboards steer the fleet live. Two Forgejo issues report fleet health without a single graph.

Where the meters live

Open https://grafana.open-platform.sh. Sign in with the Forgejo OAuth button. Members of the plat org get the Grafana Admin role. Everyone else signs in as a Viewer. Four dashboards live in the Platform folder. Prometheus keeps 60 days of history behind them. That history is the platform's durable spend record.

The table maps each dashboard to the lever it serves.

Dashboard What it tells you The lever it maps to
Fleet — Allocation & Burn Hourly burn by org, 24h spend by org, in-flight runs vs the per-org limit, flow holds by reason, live org allocations (including who is paused) The Set flow and Fleet governor forms on this same board
Flow Dams & Compute "The dams — where work is held/wasted", run success rate, setup-failure rate, cost truthfulness (% measured), and the three governor gauges: session-window, weekly-budget, burst-brake spend vs share Governor sizing — whether your caps bind before the provider does
Dispatcher & Provider Health Provider backoff (seconds remaining), freezers, global runner slots executing vs queued Global concurrency; clearing a provider backoff
Runs & Repos Runs per hour by role, run outcomes, top repos by 24h spend, mean duration and cost per run Where your attention and budget go next

Steer from the dashboard

The "Set flow — $org" panel on Fleet — Allocation & Burn is a Business Forms panel. It does not only read. It POSTs a live flow override for the selected org.

Field What it does
daily cap $ Emergency budget lid over 24h
ceiling $/h Soft hourly pacing ceiling — holds new work, self-heals when the window rolls
max runs Max concurrent runs for the org — the steady-state pacer (spend falls out of concurrency x cost x time)
min gap ms Minimum gap between run starts
breaker $ Emergency stop: org spend in 15 min above this trips a 20 min zero-dispatch cooldown — size it to ~half the hourly ceiling
pause Holds NEW work for the org; finishing reviewers still ship what is already built

The form pre-fills from the org's current effective values.

Note: Blank fields are omitted from the payload. Blank means keep — nothing is zeroed by accident.

Submit the form. The "Live — $org" readback row updates within one scrape (~30s). The dispatcher honors the override at its next dispatch decision. A DELETE clears the whole org override. The org then reverts to the env base in gitops.

Your Grafana session is the credential. The forms POST same-origin to /flow-api. A traefik forwardAuth middleware asks Grafana whether your session is valid before the request reaches the dispatcher. No session means 401. You manage no extra token.

The "Fleet governor" panel works the same way for the fleet-wide knobs — hourly / session / weekly $. Its DELETE reverts to the env base.

Dams and gauges

Flow control is one idea. The dispatcher consults a rolling-window USD spend ledger before every run start. Each check is a dam. The dispatcher meters every held item by reason. The "dams" panel stacks those reasons. Per org, the gates run in order: breaker (tight-window kill switch, then cooldown) → pause → concurrency → daily cap → ceiling → minimum gap. One deliberate bypass exists. A finishing reviewer — one that ships already-built, CI-green work — skips every gate. A built PR stranded unreviewed wastes the money already spent.

Above the org gates sits the global governor: fleet-wide dollar shares of your one Anthropic subscription. Its windows run shortest first. Session $ covers a 5-hour window — the provider's real-time wall. Weekly $ covers 7 days — endurance. Hourly $ is the burst brake. This is the point of the whole system. Spend is paced to your subscription, not to the crew's appetite.

The tuning doctrine is built in. Raise a knob until global-* holds stay near zero and provider backoff trips start to appear. Then back the knob off by about 20%. The governor should bind strictly before the provider does.

Provider backoff is the other breaker. Three consecutive fast startup failures signal a credential or usage-limit problem. The dispatcher then pauses the whole crew with exponential backoff, 5m→60m. Any healthy run clears it. The countdown shows on Dispatcher & Provider Health.

The fleet issue and the episode log

You do not need a graph to know how the fleet is doing. The 🛰 Fleet status issue is a self-updating table. The default location is plat/fleet#1; FLEET_REPO / FLEET_ISSUE set the location. Its columns: Org, Health, Episode, Progress bar, Backlog, In-flight, 24h spend. The dispatcher refreshes the table in place every few hours. The glyph legend: 🟢 working / ⏳ queued / ⚠️ stalled / ⏸ paused / 🔴 breaker / 🔌 provider backoff / ⚪ idle. Banner lines appear when provider backoff or a global pause is active. The dispatcher serves the same snapshot as JSON at GET /fleet.

The episode log (plat/fleet#2 by default) is the narrative feed. When every issue in a repo's open milestone closes, the dispatcher closes the milestone. It then posts one digest: ✅ <org>/<repo> / <milestone> — episode complete. The digest lists the issues shipped and the real agent-compute time from Forgejo's native time tracker. Orgs and Planning covers how milestones become episodes. Building with Issues covers the lifecycle of one issue.

Without Grafana

Every platform ships this surface from birth. Germinate — the process that births a platform — seals a fresh Grafana admin password and prints it on the YOUR PLATFORM card. Germinate also registers a sovereign "Sign in with Forgejo" OAuth client. The dashboards arrive with the seed. The fleet issue and the flow API work everywhere too.

The dispatcher's operator API covers every lever above: GET /fleet, GET/POST/DELETE /flow/:org, DELETE /flow/:org/cooldown, POST/DELETE /pause (global pause; finishing reviewers still ship), GET/POST/DELETE /governor, DELETE /provider/backoff. Commands require a signed-in operator session. Two wrappers carry that authority: the in-cluster plat-flow CLI, and Grafana's session-gated /flow-api. The CLI verbs: plat-flow fleet | get <org> | set <org> '<json>' | clear <org> | clear-cooldown <org> | pause | resume | clear-backoff. Endpoint and metric tables live in Reference.