Cold-start latency: keep the full setup + gateway boot inside the start budget
Status: in progress — Phase 1 (plugin detection) and the bootup-UI slow-step alert delivered; measurement and budget re-evaluation pending. This plan reduces the Worker/container cold-start latency so a full (non-restored) boot comfortably fits the Worker’s start budget, and makes a slow boot legible in the loading page instead of an unchanging stepper.
Background
A start-openclaw.sh start wrapper spans the whole cold boot in one supervised
process: the setup body (R2 restore → onboard → plugin detection → moltlazy patch
→ validate) plus the gateway’s own bootstrap before it binds the port. The
Worker’s STARTUP_TIMEOUT_MS (300s) and the owner stuck watchdog
(STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS = 360s) both bound that
process.
On a fresh / no-backup tenant the setup body alone runs ~290s, so
setup (~290s) + gateway boot (~45s) ≈ 335s sits right at the 360s watchdog grace
and beyond the 300s non-owner budget. A non-owner isolate that observed the wrapper
early timed out mid-boot and killed the gateway it had execed into, releasing the
startup lock and forcing the waiting second wrapper to re-run the entire setup
(observed on staging-client-dev: a ~14-minute cold boot).
Evidence (Workers Observability, staging-client-dev, 2026-10-12)
| Log message | Count |
|---|---|
Startup script present but gateway never became reachable … killing and restarting | 1 |
Existing process is startup script, waiting for gateway readiness | 5 |
warm-probe start/catch (Probing existing gateway / Existing process not reachable after) | 0 / 0 |
gateway.start.stuck | 0 |
Setup phase timings from /tmp/startup.log (process started 20:07:32, gateway
execed 20:12:22 → ~290s):
| Phase | Cost |
|---|---|
| lock acquire + credits + package resolution + onboard block | ~107s |
| ai-gateway plugin detection | ~57s |
| unified-billing plugin detection | ~7s |
| diagnostics-otel plugin detection | ~74s |
moltlazy patch → validate → tunnel → dev agents | ~45s |
Root cause
- Plugin detection read the registry, not the loaded artifact.
plugin_is_installedranopenclaw plugins list --json, which performs a plugin-registry refresh that hangs for its fullCLI_TIMEOUT(30s) when ClawHub/npm are slow/unreachable. A hung/partial list returns “not installed”, so the pre-installed-plugin guard fell through and triggered a needless network install — the two dominant setup costs (ai-gateway ~57s, otel ~74s). - The non-owner kill window was shorter than the wrapper’s real boot.
ensureGatewayImplmeasured a freshSTARTUP_TIMEOUT_MSfrom when a non-owner observed the wrapper rather than from the wrapper’s start, so it expired ~35s before the wrapper’s full boot completed.
Delivered
Phase 1 — fast plugin detection (start-openclaw.sh)
plugin_is_installed (registry read) → plugin_extension_present <id> (the
extensions/<id> directory the gateway actually loads, which the codebase already
documents as authoritative). The three guards (ensure_cloudflare_ai_gateway_provider_plugin,
unified-billing, diagnostics-otel) now use the instant dir check, removing up to
~120s of bounded-hang + needless-install time on a full cold boot. Corpus:
test/business-rules/operational-modes.txt.
Phase 1b — non-owner kill deferred to the owner watchdog (src/gateway/process.ts)
STARTUP_WRAPPER_STUCK_BUDGET_MS = STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS;
both the fast stuck check and the readiness wait use it, measured from the
wrapper’s startedAt, and a /health re-probe runs immediately before any kill.
A non-owner slice can no longer kill before the owner’s own watchdog would.
Phase 1c — bootup-UI slow-step alert (src/assets/loading.html)
The loading page now shows a non-fatal notice with the current step name and
elapsed seconds once a step overruns SLOW_STEP_MS (120s), driven by the
script’s own progress timestamp. It is hidden on ready/error. Test:
src/assets/loading.test.ts.
Remaining follow-up
- Measure, then decide. Add per-phase wall-clock timing to
log_bootso the boot log / observability carries each phase’s duration, and confirm the Phase 1 saving on a real full cold boot (target: setup ≤ ~180s). - Bound / shortcut the onboard block. Onboard is capped at 60s and is the other large cost; confirm whether it is hitting its cap and whether the pre-onboard provider pre-install is still needed once detection is instant.
- Re-evaluate the budgets. Once measured, decide whether
STARTUP_TIMEOUT_MS/ the wrapper-stuck budget need adjustment, or whether the restored-tenant skip path should be the norm for warm tenants (it already bypasses setup entirely). - Surface recovery state in the alert. Wire
gateway.start.stuck/ late-ready / recovery events into the startup snapshot so the loading page can say “recovering” rather than only “slow step”.
Acceptance criteria
- A full (non-restored) cold boot reaches gateway-ready well inside
STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MSwith measurable headroom. - No non-owner isolate kills a wrapper before the owner watchdog’s budget.
- The loading page names the current slow step and its elapsed time.
Out of scope
- SandboxV1 migration (
docs/plans/sandbox-v1-scaffold.md). - R2 restore latency (separate
GATEWAY_RESTORE_PHASE_BUDGET_MSbudget). - Changing the plugin pre-install set in the Dockerfile.