# Cold-start latency: keep the full setup + gateway boot inside the start budget

> Status: **in progress — Phase 1 (plugin detection) and the bootup-UI slow-step
> alert delivered; measurement and budget re-evaluation pending.**
> This plan reduces the Worker/container cold-start latency so a full (non-restored)
> boot comfortably fits the Worker's start budget, and makes a slow boot legible in
> the loading page instead of an unchanging stepper.

## Background

A `start-openclaw.sh start` wrapper spans the **whole** cold boot in one supervised
process: the setup body (R2 restore → onboard → plugin detection → `moltlazy patch`
→ validate) **plus** the gateway's own bootstrap before it binds the port. The
Worker's `STARTUP_TIMEOUT_MS` (300s) and the owner stuck watchdog
(`STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS` = 360s) both bound that
process.

On a **fresh / no-backup** tenant the setup body alone runs ~290s, so
`setup (~290s) + gateway boot (~45s) ≈ 335s` sits right at the 360s watchdog grace
and beyond the 300s non-owner budget. A non-owner isolate that observed the wrapper
early timed out mid-boot and killed the gateway it had `exec`ed into, releasing the
startup lock and forcing the waiting second wrapper to re-run the entire setup
(observed on `staging-client-dev`: a ~14-minute cold boot).

### Evidence (Workers Observability, staging-client-dev, 2026-10-12)

| Log message                                                                                  | Count |
| -------------------------------------------------------------------------------------------- | ----- |
| `Startup script present but gateway never became reachable … killing and restarting`         | 1     |
| `Existing process is startup script, waiting for gateway readiness`                          | 5     |
| warm-probe start/catch (`Probing existing gateway` / `Existing process not reachable after`) | 0 / 0 |
| `gateway.start.stuck`                                                                        | 0     |

Setup phase timings from `/tmp/startup.log` (process started 20:07:32, gateway
`exec`ed 20:12:22 → ~290s):

| Phase                                                       | Cost  |
| ----------------------------------------------------------- | ----- |
| lock acquire + credits + package resolution + onboard block | ~107s |
| ai-gateway plugin detection                                 | ~57s  |
| unified-billing plugin detection                            | ~7s   |
| diagnostics-otel plugin detection                           | ~74s  |
| `moltlazy patch` → validate → tunnel → dev agents           | ~45s  |

## Root cause

1. **Plugin detection read the registry, not the loaded artifact.**
   `plugin_is_installed` ran `openclaw plugins list --json`, which performs a
   plugin-registry refresh that hangs for its full `CLI_TIMEOUT` (30s) when
   ClawHub/npm are slow/unreachable. A hung/partial list returns "not installed",
   so the pre-installed-plugin guard fell through and triggered a **needless
   network install** — the two dominant setup costs (ai-gateway ~57s, otel ~74s).
2. **The non-owner kill window was shorter than the wrapper's real boot.**
   `ensureGatewayImpl` measured a fresh `STARTUP_TIMEOUT_MS` from when a non-owner
   _observed_ the wrapper rather than from the wrapper's start, so it expired
   ~35s before the wrapper's full boot completed.

## Delivered

### Phase 1 — fast plugin detection (`start-openclaw.sh`)

`plugin_is_installed` (registry read) → `plugin_extension_present <id>` (the
`extensions/<id>` directory the gateway actually loads, which the codebase already
documents as authoritative). The three guards (`ensure_cloudflare_ai_gateway_provider_plugin`,
unified-billing, diagnostics-otel) now use the instant dir check, removing up to
~120s of bounded-hang + needless-install time on a full cold boot. Corpus:
`test/business-rules/operational-modes.txt`.

### Phase 1b — non-owner kill deferred to the owner watchdog (`src/gateway/process.ts`)

`STARTUP_WRAPPER_STUCK_BUDGET_MS = STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS`;
both the fast stuck check and the readiness wait use it, measured from the
wrapper's `startedAt`, and a `/health` re-probe runs immediately before any kill.
A non-owner slice can no longer kill before the owner's own watchdog would.

### Phase 1c — bootup-UI slow-step alert (`src/assets/loading.html`)

The loading page now shows a non-fatal notice with the **current step name and
elapsed seconds** once a step overruns `SLOW_STEP_MS` (120s), driven by the
script's own progress timestamp. It is hidden on ready/error. Test:
`src/assets/loading.test.ts`.

## Remaining follow-up

1. **Measure, then decide.** Add per-phase wall-clock timing to `log_boot` so the
   boot log / observability carries each phase's duration, and confirm the Phase 1
   saving on a real full cold boot (target: setup ≤ ~180s).
2. **Bound / shortcut the onboard block.** Onboard is capped at 60s and is the
   other large cost; confirm whether it is hitting its cap and whether the
   pre-onboard provider pre-install is still needed once detection is instant.
3. **Re-evaluate the budgets.** Once measured, decide whether
   `STARTUP_TIMEOUT_MS` / the wrapper-stuck budget need adjustment, or whether the
   restored-tenant skip path should be the norm for warm tenants (it already
   bypasses setup entirely).
4. **Surface recovery state in the alert.** Wire `gateway.start.stuck` /
   late-ready / recovery events into the startup snapshot so the loading page can
   say "recovering" rather than only "slow step".

## Acceptance criteria

- A full (non-restored) cold boot reaches gateway-ready well inside
  `STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS` with measurable headroom.
- No non-owner isolate kills a wrapper before the owner watchdog's budget.
- The loading page names the current slow step and its elapsed time.

## Out of scope

- SandboxV1 migration (`docs/plans/sandbox-v1-scaffold.md`).
- R2 restore latency (separate `GATEWAY_RESTORE_PHASE_BUDGET_MS` budget).
- Changing the plugin pre-install set in the Dockerfile.
