F FourIA latest 2026.10.29 GitHub ↗

Cold-start latency: keep the full setup + gateway boot inside the start budget

On this page

Status: in progress — Phase 1 (plugin detection) and the bootup-UI slow-step alert delivered; measurement and budget re-evaluation pending. This plan reduces the Worker/container cold-start latency so a full (non-restored) boot comfortably fits the Worker’s start budget, and makes a slow boot legible in the loading page instead of an unchanging stepper.

Background

A start-openclaw.sh start wrapper spans the whole cold boot in one supervised process: the setup body (R2 restore → onboard → plugin detection → moltlazy patch → validate) plus the gateway’s own bootstrap before it binds the port. The Worker’s STARTUP_TIMEOUT_MS (300s) and the owner stuck watchdog (STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS = 360s) both bound that process.

On a fresh / no-backup tenant the setup body alone runs ~290s, so setup (~290s) + gateway boot (~45s) ≈ 335s sits right at the 360s watchdog grace and beyond the 300s non-owner budget. A non-owner isolate that observed the wrapper early timed out mid-boot and killed the gateway it had execed into, releasing the startup lock and forcing the waiting second wrapper to re-run the entire setup (observed on staging-client-dev: a ~14-minute cold boot).

Evidence (Workers Observability, staging-client-dev, 2026-10-12)

Log messageCount
Startup script present but gateway never became reachable … killing and restarting1
Existing process is startup script, waiting for gateway readiness5
warm-probe start/catch (Probing existing gateway / Existing process not reachable after)0 / 0
gateway.start.stuck0

Setup phase timings from /tmp/startup.log (process started 20:07:32, gateway execed 20:12:22 → ~290s):

PhaseCost
lock acquire + credits + package resolution + onboard block~107s
ai-gateway plugin detection~57s
unified-billing plugin detection~7s
diagnostics-otel plugin detection~74s
moltlazy patch → validate → tunnel → dev agents~45s

Root cause

  1. Plugin detection read the registry, not the loaded artifact. plugin_is_installed ran openclaw plugins list --json, which performs a plugin-registry refresh that hangs for its full CLI_TIMEOUT (30s) when ClawHub/npm are slow/unreachable. A hung/partial list returns “not installed”, so the pre-installed-plugin guard fell through and triggered a needless network install — the two dominant setup costs (ai-gateway ~57s, otel ~74s).
  2. The non-owner kill window was shorter than the wrapper’s real boot. ensureGatewayImpl measured a fresh STARTUP_TIMEOUT_MS from when a non-owner observed the wrapper rather than from the wrapper’s start, so it expired ~35s before the wrapper’s full boot completed.

Delivered

Phase 1 — fast plugin detection (start-openclaw.sh)

plugin_is_installed (registry read) → plugin_extension_present <id> (the extensions/<id> directory the gateway actually loads, which the codebase already documents as authoritative). The three guards (ensure_cloudflare_ai_gateway_provider_plugin, unified-billing, diagnostics-otel) now use the instant dir check, removing up to ~120s of bounded-hang + needless-install time on a full cold boot. Corpus: test/business-rules/operational-modes.txt.

Phase 1b — non-owner kill deferred to the owner watchdog (src/gateway/process.ts)

STARTUP_WRAPPER_STUCK_BUDGET_MS = STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS; both the fast stuck check and the readiness wait use it, measured from the wrapper’s startedAt, and a /health re-probe runs immediately before any kill. A non-owner slice can no longer kill before the owner’s own watchdog would.

Phase 1c — bootup-UI slow-step alert (src/assets/loading.html)

The loading page now shows a non-fatal notice with the current step name and elapsed seconds once a step overruns SLOW_STEP_MS (120s), driven by the script’s own progress timestamp. It is hidden on ready/error. Test: src/assets/loading.test.ts.

Remaining follow-up

  1. Measure, then decide. Add per-phase wall-clock timing to log_boot so the boot log / observability carries each phase’s duration, and confirm the Phase 1 saving on a real full cold boot (target: setup ≤ ~180s).
  2. Bound / shortcut the onboard block. Onboard is capped at 60s and is the other large cost; confirm whether it is hitting its cap and whether the pre-onboard provider pre-install is still needed once detection is instant.
  3. Re-evaluate the budgets. Once measured, decide whether STARTUP_TIMEOUT_MS / the wrapper-stuck budget need adjustment, or whether the restored-tenant skip path should be the norm for warm tenants (it already bypasses setup entirely).
  4. Surface recovery state in the alert. Wire gateway.start.stuck / late-ready / recovery events into the startup snapshot so the loading page can say “recovering” rather than only “slow step”.

Acceptance criteria

  • A full (non-restored) cold boot reaches gateway-ready well inside STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS with measurable headroom.
  • No non-owner isolate kills a wrapper before the owner watchdog’s budget.
  • The loading page names the current slow step and its elapsed time.

Out of scope

  • SandboxV1 migration (docs/plans/sandbox-v1-scaffold.md).
  • R2 restore latency (separate GATEWAY_RESTORE_PHASE_BUDGET_MS budget).
  • Changing the plugin pre-install set in the Dockerfile.