Stream agent transcripts live into the Forgejo UI (full observability; heartbeat v1 shipped) #17

Open
opened 2026-06-30 16:04:46 +00:00 by plat · 1 comment
Owner

The owner's idea: stream the agent's transcript LIVE into the Forgejo UI so a long (now-uncapped) run reads as 'actively building,' watchable from a phone — and so debugging needs no cluster access. The 🔄 heartbeat comment shipped in v0.7.2 is observability v1; this is the full version.

Implementation is already complete and green on feat/agent-resilience. Build, typecheck, and the 47 tests all pass. Here is my return payload: a short status block, then the issue-ready design proposal.


STATUS — resilience + observability (implemented on feat/agent-resilience)

Branch feat/agent-resilience (3 commits atop the merged dev-scope code) is built, typechecks clean (bunx tsc --noEmit), and bun test → 47 pass / 0 fail. Working tree clean, not pushed (human deploys via tag).

What the code now does, mapped to the root cause:

  • Uncapped active runs (the actual bug fix). The single hard wall-clock setTimeout(SIGTERM, timeoutMs) is gone. runner.ts now runs a polling idle watchdog (idleVerdict, src/runner.ts:154): a run is reaped only on idleTimeoutMs of zero log growth (genuinely hung — a stuck process emits no stream-json) or an absoluteMaxMs backstop (builder/worker ≈24h ≈ uncapped; reviewer bounded). The adventuring run (92 tool calls, log growing every few seconds) would now never be killed.
  • Label honesty. RunResult.timeoutKind: "idle" | "absolute" flows into describeExit (src/dispatcher.ts:127): the word "crash" is reserved for a true non-zero exit; a reaped run reads "paused after Nm with no activity" / "stopped at the absolute cap." No more "crashed twice" for a working agent.
  • No lost work. Builder resume-on-timeout (resumeBuilder, bounded at 2) re-spawns against the 🏗️ prime PR; pushed commits survive, so an idle-reaped builder continues rather than restarting.
  • Live observability v1 (heartbeat). Runner.activeRuns() exposes a runId→{logPath,startedAt} index; extractProgress tail-parses the stream-json firehose (steps, last tool, last text); the dispatcher upserts ONE HB_MARKER comment per run every HEARTBEAT_INTERVAL_MS (default 3m). This is the v1 the proposal below supersedes.
  • Playwright MCP -32000 fix. The in-cluster Chromium launch is hardened (--disable-dev-shm-usage primary, plus --no-sandbox/--disable-gpu), addressing the Connection closed sidecar drop.

Key files: /tmp/agents-impl/src/runner.ts, /tmp/agents-impl/src/dispatcher.ts, tests in /tmp/agents-impl/src/runner.test.ts + /tmp/agents-impl/src/dispatcher.test.ts.


Proposal: Live agent transcripts streamed into the Forgejo UI

Problem

Runs are now effectively uncapped for genuinely-active agents (good — slash-/goal + ultracode builds will run for hours). But "uncapped" only works if a user can see it working. The v1 heartbeat (one comment edited every few minutes with step count + last tool) proves liveness but is not watchable — it's a number that ticks, not a transcript. The owner wants the agent's actual transcript, pretty-printed, streaming live into the Forgejo UI, watchable from a phone, reading unmistakably as "actively building."

Options evaluated

A — Run the agent as a Forgejo Actions job (stream into the native job-log UI)

The dispatcher stops spawning claude locally and instead triggers a Forgejo Actions workflow; act_runner checks out the repo and runs claude -p --output-format stream-json | pretty so stdout lands in Forgejo's existing live, tailing, ANSI, collapsible, phone-friendly job-log viewer.

  • Trigger: dispatcher fires a workflow_dispatch (needs a new Forgejo.dispatchWorkflow() method — not in the client today) or pushes a control branch/tag that a workflow on: matches, passing role/issue/PR/model as inputs.
  • Credential + MCP + clone move into the job: claude OAuth token, PLAT_TOKEN, and the per-run dev secret map become Forgejo Actions secrets — this is exactly the self-serve-secrets (#15) plumbing, reused. The job container must be the agents image (Chromium + MCP baked), so the playwright/plat MCP sidecars run in-job. Clone becomes native checkout — a real win.
  • Concurrency via the runner pool: MAX_CONCURRENT + the builder>worker>reviewer priority queue have no Actions-native equivalent — concurrency becomes "number of act_runner replicas," priority becomes separate labeled runner pools. The priority semantics you just built are lost unless re-modeled.
  • Security tie-in (#16): strong fit — the job runs under a repo/org-scoped Forgejo Actions identity + token (agent-as-user), not the global PLAT_TOKEN. This is the adversary-safe-multitenancy north star.
  • Cost: this relocates the dispatcher's entire brain — idle watchdog, resume-on-timeout, reviewer Stop-hook enforcement, priority — into workflow logic or a thinner split-brain. Adds act_runner as a new cluster subsystem. Large, multi-month re-architecture; also risks Forgejo's per-job log-size caps on multi-hour firehoses.

B — Log-shipping sidecar (tail stream-json → a Forgejo-visible surface, run model unchanged)

Keep dispatcher-spawn; a sidecar tails /data/logs/<run>.log and writes elsewhere. The three candidate surfaces all disappoint as live surfaces:

  • b1 heartbeat comment — that's v1; comments aren't a live-tail (edit churn, one comment bloats).
  • b2 committed transcript file (.plat/transcripts/<run>.md) — viewable in Forgejo's file browser but not live, requires commits, pollutes the branch.
  • b3 Actions log API — Forgejo exposes no public write API to a job log from outside a job.

Verdict: B as a standalone degrades to "v1++." But its b2 idea (render to a committed .md) is the right permanent-retention artifact, so we fold it into the winner rather than ship it alone.

C — Keep dispatcher-spawn, expose a live transcript via a small platform endpoint/app

The dispatcher already holds everything: the running map, activeRuns() (runId→logPath), extractProgress, and a stream-json firehose persisted on the PVC (the code comment at runner.ts:330 literally anticipates "the future transcript SSE endpoint"). Add a small HTTP server in the dispatcher process that tails the log, pretty-prints each stream-json line, and pushes it over SSE to a tiny static SPA at transcripts.${appsDomain}. The PR/issue heartbeat becomes a "▶ Watch live" link.

  • Pros: reuses 100% of what was just shipped; true live surface in days, not months; full control of the pretty-printer and retention; no act_runner, no brain-relocation, no risk to the idle-watchdog/resume you just built.
  • Cons: it's a new surface to operate (endpoint + ingress + auth) and it lives in platform chrome adjacent to Forgejo, not inside Forgejo's own page. Auth must be implemented (it doesn't inherit Forgejo RBAC for free) — but #16's Forgejo-OAuth/scoped-identity work supplies the pieces.

Recommendation — C now, A as the north star (phased C→A); fold B's committed-transcript in as retention

Ship C as v2 because it is the fastest path to the real thing and it reuses the resilience code just landed rather than fighting it. Keep A as the explicit v3 target — its end-state (native Forgejo job-log UI, native artifacts/retention, repo-scoped identity) is strictly better, but only once act_runner + secrets(#15) + scoped-identity(#16) are mature enough that the orchestration brain can be expressed as a trigger layer. B is rejected standalone, but its committed-.md becomes C's permanent record.

Rationale for not leading with A despite its "native UI" appeal: A demands relocating the idle watchdog, resume-on-timeout, reviewer Stop-hook, and the builder>worker>reviewer priority queue — all freshly built, none with Actions-native equivalents — and adds a whole act_runner subsystem. C delivers the owner's actual goal (live, pretty, phone-watchable) on top of code that already exists.

Winner (C) — design detail

Data flow (unchanged spine, two new read-only consumers):

claude -p --output-format stream-json
  → logFd → /data/logs/<runId>.log  (PVC, persisted; only work/<runId> is GC'd)
       ├─ heartbeat (extractProgress, low-freq)  → "▶ Watch live" comment on PR/issue
       └─ transcript server: fs.watch + incremental read from last offset
              → renderEvent(jsonl) → SSE → static SPA (live)
              → on completion: render once → commit .plat/transcripts/<runId>.md (permanent)

Runner changes: essentially none — activeRuns()/logPath are already exposed and logs already survive run cleanup (run() deletes only work/<runId>, not logs/). Add only a terminal "run done + final logPath" signal so the server flips a live stream to replay.

Dispatcher changes: add a Bun HTTP server with three routes — GET /runs (from activeRuns()), GET /runs/:id/stream (SSE live tail), GET /runs/:id (replay a finished run from PVC or the committed .md). Change the heartbeat to upsert a low-churn "▶ Watch live: · last:

" comment (keep HB_MARKER for find/replace upsert; drop beat frequency to ~10m since detail now lives in the SPA). Gate the whole thing behind TRANSCRIPT_URL_BASE so it's inert until deployed.

Pretty-printer (renderEvent(json) → TranscriptLine, pure + unit-tested, generalizing extractProgress's parser):

  • system/init → ▶ starting <model>
  • assistant text → markdown bubble · assistant tool_use → 🔧 <tool>(<short args>)
  • user/tool_result → ✓ result (<n> lines) / ✗ error (long outputs collapsed)
  • result → ✅ done — <summary> · <tokens>/<cost> or the timeout/error subtype

Retention: stream-json stays on the PVC with an N-day TTL (live + replay); on completion, render once to .plat/transcripts/<runId>.md committed to the PR branch — a permanent, in-Forgejo, infra-free record that survives dispatcher restarts (this is B's good idea). No object store needed for v2.

Migration from v1 heartbeat: purely additive — no data migration. v1's churning progress comment becomes a static "Watch live" link comment (same HB_MARKER, same upsert path, lower frequency); the live detail moves to SSE. If TRANSCRIPT_URL_BASE is unset, behavior falls back to exactly today's v1 heartbeat.

Deploy path: the transcript server is the same Bun process as the dispatcher (no new image). Add a Service + Ingress (transcripts.${appsDomain}) to charts/agents; ship via tag → open-platform.sh CI builds image+chart → plat/gitops pins the new agents OCIRepository/HelmRelease tag.

Phased plan

  • Phase 0 — done: idle watchdog (uncapped active runs), label honesty, resume-on-timeout, Chromium hardening, v1 heartbeat (feat/agent-resilience).
  • Phase 1 — C core: SSE endpoint + renderEvent pretty-printer + minimal SPA; heartbeat becomes the live link; charts/agents Service+Ingress; behind TRANSCRIPT_URL_BASE.
  • Phase 2 — C hardening: auth via Forgejo OAuth / signed per-PR links (#16), committed-.md retention on completion, replay of finished runs.
  • Phase 3 — A pilot (north star): move the reviewer only (bounded, no resume, no priority dependence — the simplest role) to a Forgejo Actions job to prove act_runner + Actions-secrets(#15) + scoped identity(#16) + MCP-in-job; evaluate migrating builder/worker once orchestration-in-Actions is proven.

Decisions for the owner

  1. Native-chrome (A) vs fast-linked-app (C) for v2. Recommendation: C now, A piloted at Phase 3. Confirm we're OK with the live surface being a platform app linked from each PR/issue rather than inside Forgejo's own page for v2.
  2. Auth model for the transcript surface. Public-with-unguessable-link, Forgejo-OAuth-gated (inherits repo RBAC, ties to #16), or signed-token links posted into the PR. Drives adversary-safe multitenancy — this is the one real security decision.
  3. Retention. TTL for the raw stream-json on the PVC (replay window) and whether we commit the permanent .plat/transcripts/<run>.md to the PR branch (permanent + in-Forgejo, slight branch noise) vs PVC-only (cleaner branch, lost on GC/restart).
The owner's idea: stream the agent's transcript LIVE into the Forgejo UI so a long (now-uncapped) run reads as 'actively building,' watchable from a phone — and so debugging needs no cluster access. The 🔄 heartbeat comment shipped in v0.7.2 is observability v1; this is the full version. Implementation is already complete and green on `feat/agent-resilience`. Build, typecheck, and the 47 tests all pass. Here is my return payload: a short status block, then the issue-ready design proposal. --- ## STATUS — resilience + observability (implemented on `feat/agent-resilience`) Branch `feat/agent-resilience` (3 commits atop the merged dev-scope code) is built, typechecks clean (`bunx tsc --noEmit`), and `bun test` → 47 pass / 0 fail. Working tree clean, **not pushed** (human deploys via tag). What the code now does, mapped to the root cause: - **Uncapped active runs (the actual bug fix).** The single hard wall-clock `setTimeout(SIGTERM, timeoutMs)` is gone. `runner.ts` now runs a polling **idle watchdog** (`idleVerdict`, `src/runner.ts:154`): a run is reaped only on `idleTimeoutMs` of *zero log growth* (genuinely hung — a stuck process emits no stream-json) or an `absoluteMaxMs` backstop (builder/worker ≈24h ≈ uncapped; reviewer bounded). The adventuring run (92 tool calls, log growing every few seconds) would now never be killed. - **Label honesty.** `RunResult.timeoutKind: "idle" | "absolute"` flows into `describeExit` (`src/dispatcher.ts:127`): the word "crash" is reserved for a true non-zero exit; a reaped run reads "paused after Nm with no activity" / "stopped at the absolute cap." No more "crashed twice" for a working agent. - **No lost work.** Builder resume-on-timeout (`resumeBuilder`, bounded at 2) re-spawns against the 🏗️ prime PR; pushed commits survive, so an idle-reaped builder continues rather than restarting. - **Live observability v1 (heartbeat).** `Runner.activeRuns()` exposes a `runId→{logPath,startedAt}` index; `extractProgress` tail-parses the stream-json firehose (steps, last tool, last text); the dispatcher upserts ONE `HB_MARKER` comment per run every `HEARTBEAT_INTERVAL_MS` (default 3m). This is the v1 the proposal below supersedes. - **Playwright MCP -32000 fix.** The in-cluster Chromium launch is hardened (`--disable-dev-shm-usage` primary, plus `--no-sandbox`/`--disable-gpu`), addressing the `Connection closed` sidecar drop. Key files: `/tmp/agents-impl/src/runner.ts`, `/tmp/agents-impl/src/dispatcher.ts`, tests in `/tmp/agents-impl/src/runner.test.ts` + `/tmp/agents-impl/src/dispatcher.test.ts`. --- # Proposal: Live agent transcripts streamed into the Forgejo UI ## Problem Runs are now effectively uncapped for genuinely-active agents (good — `slash-/goal` + ultracode builds will run for hours). But "uncapped" only works if a user can *see* it working. The v1 heartbeat (one comment edited every few minutes with step count + last tool) proves liveness but is not watchable — it's a number that ticks, not a transcript. The owner wants the agent's actual transcript, pretty-printed, streaming live into the Forgejo UI, watchable from a phone, reading unmistakably as "actively building." ## Options evaluated ### A — Run the agent **as a Forgejo Actions job** (stream into the native job-log UI) The dispatcher stops spawning `claude` locally and instead triggers a Forgejo Actions workflow; `act_runner` checks out the repo and runs `claude -p --output-format stream-json | pretty` so stdout lands in Forgejo's existing live, tailing, ANSI, collapsible, **phone-friendly** job-log viewer. - **Trigger:** dispatcher fires a `workflow_dispatch` (needs a new `Forgejo.dispatchWorkflow()` method — not in the client today) or pushes a control branch/tag that a workflow `on:` matches, passing role/issue/PR/model as inputs. - **Credential + MCP + clone move into the job:** claude OAuth token, `PLAT_TOKEN`, and the per-run dev secret map become **Forgejo Actions secrets** — this is exactly the self-serve-secrets (#15) plumbing, reused. The job container must be the **agents image** (Chromium + MCP baked), so the playwright/plat MCP sidecars run in-job. Clone becomes native `checkout` — a real win. - **Concurrency via the runner pool:** `MAX_CONCURRENT` + the builder>worker>reviewer **priority queue** have no Actions-native equivalent — concurrency becomes "number of `act_runner` replicas," priority becomes separate labeled runner pools. The priority semantics you just built are lost unless re-modeled. - **Security tie-in (#16):** strong fit — the job runs under a **repo/org-scoped Forgejo Actions identity + token** (agent-as-user), not the global `PLAT_TOKEN`. This is the adversary-safe-multitenancy north star. - **Cost:** this relocates the dispatcher's *entire brain* — idle watchdog, resume-on-timeout, reviewer Stop-hook enforcement, priority — into workflow logic or a thinner split-brain. Adds `act_runner` as a new cluster subsystem. Large, multi-month re-architecture; also risks Forgejo's per-job log-size caps on multi-hour firehoses. ### B — Log-shipping sidecar (tail stream-json → a Forgejo-visible surface, run model unchanged) Keep dispatcher-spawn; a sidecar tails `/data/logs/<run>.log` and writes elsewhere. The three candidate surfaces all disappoint as *live* surfaces: - **b1 heartbeat comment** — that's v1; comments aren't a live-tail (edit churn, one comment bloats). - **b2 committed transcript file** (`.plat/transcripts/<run>.md`) — viewable in Forgejo's file browser but not live, requires commits, pollutes the branch. - **b3 Actions log API** — Forgejo exposes no public *write* API to a job log from outside a job. **Verdict:** B as a standalone degrades to "v1++." But its b2 idea (render to a committed `.md`) is the right **permanent-retention artifact**, so we fold it into the winner rather than ship it alone. ### C — Keep dispatcher-spawn, expose a live transcript via a small platform endpoint/app The dispatcher already holds everything: the `running` map, `activeRuns()` (`runId→logPath`), `extractProgress`, and a stream-json firehose persisted on the PVC (the code comment at `runner.ts:330` literally anticipates "the future transcript SSE endpoint"). Add a small HTTP server in the dispatcher process that **tails the log, pretty-prints each stream-json line, and pushes it over SSE** to a tiny static SPA at `transcripts.${appsDomain}`. The PR/issue heartbeat becomes a **"▶ Watch live"** link. - **Pros:** reuses 100% of what was just shipped; true live surface in days, not months; full control of the pretty-printer and retention; no `act_runner`, no brain-relocation, no risk to the idle-watchdog/resume you just built. - **Cons:** it's a *new* surface to operate (endpoint + ingress + auth) and it lives in *platform chrome adjacent to* Forgejo, not inside Forgejo's own page. Auth must be implemented (it doesn't inherit Forgejo RBAC for free) — but #16's Forgejo-OAuth/scoped-identity work supplies the pieces. ## Recommendation — **C now, A as the north star (phased C→A); fold B's committed-transcript in as retention** Ship **C** as v2 because it is the fastest path to the real thing and it *reuses the resilience code just landed* rather than fighting it. Keep **A** as the explicit v3 target — its end-state (native Forgejo job-log UI, native artifacts/retention, repo-scoped identity) is strictly better, but only once `act_runner` + secrets(#15) + scoped-identity(#16) are mature enough that the orchestration brain can be expressed as a trigger layer. **B is rejected standalone**, but its committed-`.md` becomes C's permanent record. Rationale for not leading with A despite its "native UI" appeal: A demands relocating the idle watchdog, resume-on-timeout, reviewer Stop-hook, and the builder>worker>reviewer priority queue — all freshly built, none with Actions-native equivalents — and adds a whole `act_runner` subsystem. C delivers the owner's actual goal (live, pretty, phone-watchable) on top of code that already exists. ### Winner (C) — design detail **Data flow** (unchanged spine, two new read-only consumers): ``` claude -p --output-format stream-json → logFd → /data/logs/<runId>.log (PVC, persisted; only work/<runId> is GC'd) ├─ heartbeat (extractProgress, low-freq) → "▶ Watch live" comment on PR/issue └─ transcript server: fs.watch + incremental read from last offset → renderEvent(jsonl) → SSE → static SPA (live) → on completion: render once → commit .plat/transcripts/<runId>.md (permanent) ``` **Runner changes:** essentially none — `activeRuns()`/`logPath` are already exposed and logs already survive run cleanup (`run()` deletes only `work/<runId>`, not `logs/`). Add only a terminal "run done + final logPath" signal so the server flips a live stream to replay. **Dispatcher changes:** add a Bun HTTP server with three routes — `GET /runs` (from `activeRuns()`), `GET /runs/:id/stream` (SSE live tail), `GET /runs/:id` (replay a finished run from PVC or the committed `.md`). Change the heartbeat to upsert a low-churn **"▶ Watch live: <url> · last: <summary>"** comment (keep `HB_MARKER` for find/replace upsert; drop beat frequency to ~10m since detail now lives in the SPA). Gate the whole thing behind `TRANSCRIPT_URL_BASE` so it's inert until deployed. **Pretty-printer** (`renderEvent(json) → TranscriptLine`, pure + unit-tested, generalizing `extractProgress`'s parser): - `system/init` → `▶ starting <model>` - `assistant` text → markdown bubble · `assistant` `tool_use` → `🔧 <tool>(<short args>)` - `user`/`tool_result` → `✓ result (<n> lines)` / `✗ error` (long outputs collapsed) - `result` → `✅ done — <summary> · <tokens>/<cost>` or the timeout/error subtype **Retention:** stream-json stays on the PVC with an N-day TTL (live + replay); on completion, render once to `.plat/transcripts/<runId>.md` committed to the PR branch — a permanent, in-Forgejo, infra-free record that survives dispatcher restarts (this is B's good idea). No object store needed for v2. **Migration from v1 heartbeat:** purely additive — no data migration. v1's churning progress comment becomes a static "Watch live" link comment (same `HB_MARKER`, same upsert path, lower frequency); the live detail moves to SSE. If `TRANSCRIPT_URL_BASE` is unset, behavior falls back to exactly today's v1 heartbeat. **Deploy path:** the transcript server is the same Bun process as the dispatcher (no new image). Add a `Service` + `Ingress` (`transcripts.${appsDomain}`) to `charts/agents`; ship via tag → open-platform.sh CI builds image+chart → plat/gitops pins the new agents OCIRepository/HelmRelease tag. ### Phased plan - **Phase 0 — done:** idle watchdog (uncapped active runs), label honesty, resume-on-timeout, Chromium hardening, v1 heartbeat (`feat/agent-resilience`). - **Phase 1 — C core:** SSE endpoint + `renderEvent` pretty-printer + minimal SPA; heartbeat becomes the live link; `charts/agents` Service+Ingress; behind `TRANSCRIPT_URL_BASE`. - **Phase 2 — C hardening:** auth via Forgejo OAuth / signed per-PR links (#16), committed-`.md` retention on completion, replay of finished runs. - **Phase 3 — A pilot (north star):** move the **reviewer only** (bounded, no resume, no priority dependence — the simplest role) to a Forgejo Actions job to prove `act_runner` + Actions-secrets(#15) + scoped identity(#16) + MCP-in-job; evaluate migrating builder/worker once orchestration-in-Actions is proven. ## Decisions for the owner 1. **Native-chrome (A) vs fast-linked-app (C) for v2.** Recommendation: C now, A piloted at Phase 3. Confirm we're OK with the live surface being a platform app *linked from* each PR/issue rather than inside Forgejo's own page for v2. 2. **Auth model for the transcript surface.** Public-with-unguessable-link, Forgejo-OAuth-gated (inherits repo RBAC, ties to #16), or signed-token links posted into the PR. Drives adversary-safe multitenancy — this is the one real security decision. 3. **Retention.** TTL for the raw stream-json on the PVC (replay window) and whether we commit the permanent `.plat/transcripts/<run>.md` to the PR branch (permanent + in-Forgejo, slight branch noise) vs PVC-only (cleaner branch, lost on GC/restart).
Author
Owner

Observability v1 SHIPPED this session: agents 0.7.x posts a single edit-in-place 🔄 working… heartbeat comment (elapsed/steps/last-action) on the PR/issue during a run, plus the idle-based uncapped timeout (long runs live as long as they emit; killed only when genuinely idle). This issue tracks the FULL version (streaming the live transcript into the Forgejo Actions UI). Design/options in the issue body; recommended path = run the agent AS a Forgejo Actions job so its stream-json stdout streams to the native job-log UI.

Observability v1 SHIPPED this session: agents 0.7.x posts a single edit-in-place `🔄 working…` heartbeat comment (elapsed/steps/last-action) on the PR/issue during a run, plus the idle-based uncapped timeout (long runs live as long as they emit; killed only when genuinely idle). This issue tracks the FULL version (streaming the live transcript into the Forgejo Actions UI). Design/options in the issue body; recommended path = run the agent AS a Forgejo Actions job so its stream-json stdout streams to the native job-log UI.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
open-platform/mitosis#17
No description provided.