# FourIA — Worker Dashboard Architecture

Deep-dive into the `apps/fouria` component, the core Tenant Worker of FourIA.

> See [../docs/DEVELOPMENT.md](../../docs/DEVELOPMENT.md) for project-level overview and [../docs/ENV-VARIABLES.md](../../docs/ENV-VARIABLES.md) for environment variable reference.

---

## Architecture Overview

```
Browser
   │
   ▼
┌─────────────────────────────────────────┐
│       Cloudflare Worker (index.ts)       │
│                                          │
│  Middleware: Logging → Auth → RBAC       │
│                                          │
│  Routes:                                 │
│    /api/*     → Hono API routes          │
│    /_admin/*  → React SPA (Vite build)   │
│    /debug/*   → Debug endpoints (ROOT)   │
│    /*         → Proxy to OpenClaw GW     │
│                                          │
│  Activation: Worker restore → onboard    │
│              → patch → launch gateway    │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│   Cloudflare Sandbox Container           │
│                                          │
│  start-openclaw.sh:                      │
│    1. openclaw onboard (auth setup)      │
│    2. Enable plugins (unified billing)   │
│    3. moltlazy patch (config injection)  │
│    4. Start Cloudflare Tunnel (optional) │
│    5. background gateway + health poll   │
│                                          │
│  ┌────────────────────────────────────┐  │
│  │     OpenClaw Gateway               │  │
│  │  - Control UI on port 18789        │  │
│  │  - WebSocket RPC protocol          │  │
│  │  - Agent runtime                   │  │
│  └────────────────────────────────────┘  │
└─────────────────────────────────────────┘
```

---

## Frontend Architecture (Vite + React SPA)

The dashboard is a React 19 SPA built with Vite 6 and served from `/_admin/`.

### Build & Dev Flow

```
bun run dev           → Vite dev server on :8787 (vite.ui.config.ts)
    │                     Hot-reload for frontend changes
    │                     Proxies /api/*, /debug/* to :8788 (worker)
    │
bun run dev:worker    → Vite dev server on :8788 (vite.config.ts)
    │                     Runs the actual Worker in workerd via the
    │                     Cloudflare Vite plugin; Worker changes hot-reload
    │
bun start             → bun run build:docker && vite dev
                          Full local stack (build + Worker dev server)
```

> **Key insight:** both dev servers are Vite. `bun run dev` only handles the
> admin SPA; the Worker (routes, auth, gateway, DOs, containers) runs under the
> Cloudflare Vite plugin via `bun run dev:worker`, the same plugin used by
> `vite build`. The plugin has no `wrangler dev` flags: local E2E vars are
> injected through its `config` hook and `.dev.vars` is loaded from this
> directory.

### Vite Configuration

```typescript
// vite.config.ts — Worker dev server (bun run dev:worker)
base: '/_admin/'              // All assets served under /_admin/
server: {
  port: 8788,
  allowedHosts,               // DEV_ALLOWED_HOSTS (tunnel hostnames)
}

// vite.ui.config.ts — admin SPA dev server (bun run dev, :8787)
server: {
  port: 8787,
  proxy: {
    '/api':   'http://localhost:8788',
    '/debug': 'http://localhost:8788',
    '/cdp':   'http://localhost:8788',
  },
}
```

> Exposing the Worker through a tunnel (e.g. Telegram webhooks): Vite blocks
> unknown `Host` headers, so pass the public hostname explicitly —
> `DEV_ALLOWED_HOSTS=foo.trycloudflare.com bun run dev:worker` (comma-separated;
> a leading dot allows a whole domain). Point the tunnel at `:8788`, not the SPA
> at `:8787` (it does not proxy `/telegram-webhook`).

### Client Module Structure

```
src/client/
├── App.tsx              # Root component with React Router
├── main.tsx             # Entry point — mounts React app
├── api.ts               # Typed API client → /api/admin/*
├── gateway-gate.ts      # Health circuit breaker (park/resolve gateway queries)
├── GatewayHealthContext.tsx  # App-wide passive health subscription + gate wiring
├── pages/               # Route-level pages
│   ├── HomePage.tsx       # Dashboard overview
│   ├── AgentsPage.tsx     # Agent management
│   ├── ChannelsPage.tsx   # Connections/integrations
│   ├── KnowledgePage.tsx  # Knowledge base/wiki
│   ├── AccountPage.tsx    # User/team management
│   ├── IntegrationsPage.tsx   # Integration config
│   ├── SecretsPage.tsx    # Vault management
│   ├── BackupsPage.tsx    # Backup management
│   ├── DiagnosticDashboard.tsx  # Costs + diagnostics
│   ├── AuditLogsPage.tsx  # Gateway ledgers: audit activity + background tasks
│   └── AutomationEditorPage.tsx # Automation workflows
├── components/          # Shared UI components
└── hooks/               # Custom React hooks
```

### Frontend ↔ Backend Communication

The frontend (`src/client/api.ts`) calls `/api/admin/*` endpoints. These are served by the Worker's Hono routes (`src/routes/api.ts`), which:

1. Verify RBAC (`requireWriteRole()` for mutations)
2. Execute OpenClaw CLI commands inside the Sandbox container
3. Parse CLI output and return JSON

#### Client circuit breakers (health-gated gateway queries)

Two client-side circuit breakers keep the dashboard responsive when the OpenClaw gateway is not ready (issue #371):

- **#1 — gateway queries are asynchronous.** Pages paint immediately and never block first paint on a gateway response; mutations are fire-and-notify through `NotificationProvider`. `api.ts` routes every gateway-touching call through `gatewayApiRequest`.
- **#2 — gateway queries require the passive health stream to be UP.** The single app-wide `GatewayHealthProvider` (`src/client/GatewayHealthContext.tsx`) subscribes once to `/api/health-ws?passive=1` and drives a module-singleton `GatewayGate` (`src/client/gateway-gate.ts`). **UP** is defined as `snapshot.readiness.ok === true`; `starting`/`stopped`/`sleeping` are **not** UP (`sleeping` is healthy-idle but the gateway is not running). Before dispatching, `apiRequest` awaits `whenGatewayUp(signal)`; while down the query is parked and the `GatewayGateNotice` indicator shows, then the queue flushes automatically on the next UP snapshot. A transient (`503`/transport) failure is retried once after the next UP.

The gateway-touching boundary is the `GATEWAY_TOUCHING_PREFIXES` manifest in `api.ts` (`/agents`, `/models`, `/channels`, `/config`, `/devices`, `/integrations`, `/secrets`, `/knowledge`, `/audit-logs`, `/dreaming`, `/gateway/restart`). Diagnostics (`/debug/*`, `/logs`, `/gateway-api`, `/gateway/status/polling`, `/gateway-token`, `/version`) and worker-local surfaces (`/storage`, `/costs`, `/billing`, `/users`, `/vault`, `/releases`, `/fouria-docs`, `/whoami`) are deliberately **not** gated, so an operator can still inspect a down gateway and non-gateway features keep working.

**Diagnostics (`DiagnosticDashboard`).** The ROOT dashboard is explicitly _skip-not-park_: `useGatewayRunning()` (`GatewayHealthContext.tsx`) reports `snapshot.readiness.ok` **ignoring the ROOT bypass**, and the page only dispatches its container reads (`/config/raw`, `/config`) while it is `true`; they re-run automatically when the gateway transitions back to running. The debug-routes flag is Worker-local and still loads. Every call is dispatched after paint (`scheduleAfterPaint` → `requestIdleCallback`, `setTimeout(0)` fallback) and the widgets render their waiting state first, so nothing is blocked on a container response: the Configuration Editor renders its tabs/editor with a non-blocking "Loading configuration…" / "Waiting for the gateway to start…" notice (`gatewayReady` prop on `ConfigSection`) and the device list shows a waiting state until Refresh. Gateway-only actions (device list/approve, config save, channel probe) are disabled/guarded while down; **`doctor --fix` repair is NOT one of them**: it is the recovery action for a broken/down gateway (the backend stops the gateway itself), so a dedicated **Repair Environment** control sits outside the Configuration Editor (always rendered for ADMIN/ROOT, disabled only while a repair is in flight) and the in-editor repair button stays enabled while the gateway is down. The Debug Tools that call the container (logs, processes, version, sqlite integrity) and the recovery actions (restart/rescue/destroy) stay available on purpose — they are how an operator inspects and revives a down gateway — while the ones that require a running gateway (`gateway-api`, `stop-gateway`) are skipped with an inline explanation.

**ROOT bypass.** ROOT operators are never gated: `App.tsx` passes `bypass={whoami?.role === 'ROOT'}` to the provider, which flips the gate's bypass (`GatewayGate.setBypass`). This preserves the "ROOT can always reach a tenant" rule and keeps the operator's **non-passive** device-pair request (`GET /api/admin/devices/ws` → `listDevices` → `ensureGateway({ waitForReady: false })`) able to wake a sleeping container. The notification bell is **passive for every non-ROOT role** (`subscribeDeviceNotifications(..., { passive: true })` → `?passive=1`), so a regular dashboard load never wakes a sleeping tenant; only ROOT's stream is non-passive. Server-side, a passive `/api/admin/devices/ws` first does the non-waking `readRuntimeActive()` read and, when the runtime is not live, emits an empty snapshot and skips the gateway event stream. Once the ROOT wake brings the gateway UP, the shared health stream flips to `up` and every parked query (for all roles) flushes. The `GatewayGateNotice` is suppressed while bypassed, since ROOT actions are never queued.

The gate is fail-open when no provider is mounted (non-SPA callers, unit tests): the module default is `up`. The provider installs the shared gate during its first render, before any child page effect runs. It is a module singleton rather than a `useState` initializer because React StrictMode double-invokes initializers in dev — two gates would leave `api.ts` parked on a different instance than the provider drives.

---

## Backend Architecture (Hono Worker)

### Request Pipeline

```
Incoming Request
    │
    ├─► Logging middleware (all routes)
    ├─► Sandbox init (getSandbox() into context)
    │
    ├─► Public routes (no auth):
    │     /sandbox-health, /api/otel/*
    │
    ├─► CF Access Auth middleware:
    │     Extract JWT → validate → set accessUser
    │
    ├─► RBAC resolution:
    │     resolveCustomerRole → set userRole, customerSlug
    │
    ├─► Route dispatch:
    │     /api/*     → API routes (auth + RBAC)
    │     /_admin/*  → Admin UI (SPA assets)
    │     /debug/*   → Debug routes (ROOT only)
    │     /*         → Catch-all proxy to gateway
    │
    └─► Gateway proxy:
          HTTP proxy → containerFetch(port 18789)
          WS proxy   → WebSocket relay with token injection
```

### API Route Structure

```typescript
// src/routes/api.ts — route aggregation
api.route('/admin', adminApi)
  ├── /whoami, /version, /logs, /gateway-api, /debug-status, /gateway-token  (misc)
  ├── /devices/*      → devicesRoutes
  ├── /storage/*      → storageRoutes
  ├── /releases/*     → releasesRoutes
  ├── /gateway/*      → gatewayRoutes
  ├── /config/*       → configRoutes
  ├── /cron/*         → cronRoutes
  ├── /dreaming/*     → dreamingRoutes
  ├── /users/*        → usersRoutes
  ├── /integrations/* → integrationsRoutes
  ├── /knowledge/*    → knowledgeRoutes
  ├── /secrets/*      → secretsRoutes
  ├── /agents/*       → agentsRoutes
  ├── /models/*       → modelsRoutes
  ├── /channels/*     → channelsRoutes
  ├── /costs/*        → costsRoutes
  ├── /vault/*        → vaultRoutes
  ├── /billing/*      → billingRoutes (plan, checkout, portal, invoices, freeze-status, freeze, unfreeze)
  └── /audit-logs/*   → auditLogsRoutes (audit activity ledger + task ledger)

// Write gate: POST/PUT/DELETE/PATCH require ADMIN or ROOT
// Read gate: GET/HEAD available to all authenticated roles
```

> **Build identity.** `GET /api/admin/version` and `GET /debug/version` report the
> FourIA Worker's own `fouria_version` (component CalVer from
> `apps/fouria/package.json`) and `fouria_commit` (short git SHA) alongside the
> container's OpenClaw/Node versions. Both are injected into the bundle by
> `vite.config.ts` (`__FOURIA_VERSION__` / `__FOURIA_COMMIT__`, consumed in
> `src/build-info.ts`; `GITHUB_SHA` wins in CI, `git rev-parse --short HEAD`
> locally). The DiagnosticDashboard **Version Info** tool surfaces them for
> operators.

### Cloudflare error and trace correlation

The Worker emits structured `worker.request.started` and
`worker.request.completed` records with the method, path, status, duration, and
Cloudflare Ray ID (`cfRay`). Unhandled exceptions are logged as
`worker.request.failed` with the exception name/message/stack and the same
request fields. Cloudflare also attaches its invocation `$metadata.requestId`
and `$metadata.traceId` to each Worker log, so the invocation's request, logs,
and trace can be opened together in Workers Observability. Error records pass an
`Error` object to `console.error` as well, so Cloudflare populates its exception
metadata (including `$metadata.error`) alongside the indexed event fields.

Sandbox command failures are emitted as `sandbox.process.failed`. These records
include a stable operation label and phase, the Sandbox `processId` when a
process was launched, and available exit-code/timeout/duration fields. The
`processId` is the join key to the `processId` field on the Containers dataset's
`process.complete` event; inspect `exitCode` and `timedOut` there rather than
trusting the generic event message, which can say `success` when the process
itself exited non-zero. Raw shell commands, stdout, and stderr are deliberately
omitted because they can contain tenant config or credentials.

Persistence failures use stable events such as `persistence.backup.failed`,
`persistence.config-version.failed`, `persistence.restore.failed`, and
`storage.restore.failed`. Config-version and restore errors include `backupId`
when an archive exists; use it to join to the `backup.create` / `backup.restore`
container events. A config-version capture failure is best-effort: it identifies
the archive whose supplemental config snapshot was not captured, but does not
mean the archive itself failed to be created.

Backup and restore work also emits paired `persistence.phase` records. Each
record has an `operationId`, `operation` (`backup` or `restore`), `phase`, and
`status` (`started`, `completed`, or `failed`); terminal records include
`durationMs`, and phases after an archive handle exists include `backupId`.
`source` names the route or job, `cfRayId` correlates request-triggered work with
`worker.request.*`, and `jobId` correlates work running later in the
ActivityScheduler. Use `operationId` to group the phases; if a started phase has
no terminal record, the Worker may have been interrupted during that phase.
Useful phases include `sandbox.createBackup`, `archive.encrypt`,
`catalog.commit`, `sandbox.restoreBackup`, and `legacy-state.materialize`.

For a failure investigation, filter the `cloudflare-workers` dataset to
`event = "sandbox.process.failed"`, record `processId` and the Worker
`$metadata.requestId`/`$metadata.traceId`, then search the `containers` dataset
for the same `processId` (and the tenant's `$container.applicationId`). For a
request-level failure, filter to `event = "worker.request.failed"` or use
`cfRay` to pivot to the matching edge request. For backup/restore CPU-limit
investigations, filter to `event = "persistence.phase"`, find the last
`started` event for the relevant `cfRayId` or `jobId`, then group by its
`operationId` to see which phases completed before it.

### Worker observability export destinations

Account-level Cloudflare **Observability → Destinations** export this Worker's
logs and traces to Lerma over OTLP/HTTP JSON (issue #337). The destination names
are referenced in the `observability` block:

```jsonc
// staging
"observability": {
  "logs":   { "destinations": ["lerma-worker-logs"],   "head_sampling_rate": 0.1, "persist": false },
  "traces": { "destinations": ["lerma-worker-traces"], "head_sampling_rate": 0.1, "persist": false }
}
// production
"observability": {
  "logs":   { "destinations": ["lerma-worker-logs-prod"],   "head_sampling_rate": 0.1, "persist": false },
  "traces": { "destinations": ["lerma-worker-traces-prod"], "head_sampling_rate": 0.1, "persist": false }
}
```

The `-prod` destinations point at `https://lerma.paso4.io/api/ingest/otel/v1/*`;
the un-suffixed pair points at `https://lerma-dev.paso4.io/...`.

- `apps/fouria/wrangler.jsonc` carries the matching names in each env block
  (`staging` and `production`, direct deploys).
- WFP-deployed tenant workers get the names from Lerma's
  `Lerma.Cloudflare.Observability.worker_metadata/0`
  (`WORKER_OBSERVABILITY_LOGS_DESTINATIONS` /
  `WORKER_OBSERVABILITY_TRACES_DESTINATIONS`). The CD sets the staging names for
  staging and the `-prod` names for production, so a redeploy inherits them.
- Lerma is the destination of record: `persist: false` stops the duplicate
  Cloudflare-side copy, and the head sampling rate (10% at rollout) bounds export
  volume. Lerma drops message text, stacks, URLs/queries, user agents,
  request/response bodies and DB query text at ingestion; only allowlisted
  metadata is stored.
- Cloudflare gzips each OTLP/HTTP JSON batch (`Content-Encoding: gzip` with
  `Content-Type: application/json`). `LermaWeb.Plugs.OtlpBodyReader` runs before
  `Plug.Parsers` on the ingest routes and decompresses it; without that step the
  normal JSON parser returns `400` and the destination retries forever.
- The export only feeds diagnostics + the Lerma instance **Worker telemetry**
  widget. It never replaces `HealthMonitor` (still the source of
  `instances.status`) or billing, and it is not a probing path — no container is
  woken to produce it.

Ops: create the two destinations under **Observability → Destinations** with the
custom header `Authorization: Bearer <OTEL_INGEST_TOKEN>`, set the token and the
destination names on Lerma (staging first), then redeploy the tenants. Delivery
status shows **Last: n minutes ago**; the uptime numbers come from
`GET /api/analytics/v1/uptime/<script_name>?window=24h`.

### Gateway start lifecycle + stuck-start watchdog

A cold start is one shared promise (`ensureGateway`) spanning the restore
barrier → the supervised `start-openclaw.sh start` launch → the port wait. A
wrapper that never `exec`s the gateway, or a container control-channel drop, can
leave that promise "in progress" for many minutes while every request logs
`Gateway start already in progress … waiting...` and waits on it.

The start is now tracked per sandbox key with a phase (`probe` → `restore` →
`launch` → `port-wait`) and a per-phase timer. `post-start` runs **detached**
after readiness (see below), so it is deliberately not part of the awaited start
promise:

- `gateway.start.begin` / `gateway.start.phase` structured events record the
  attempt, phase, elapsed time, and (after launch) the process id; the
  "already in progress" log carries `phase`, `age`, and `attempt`.
- `isGatewayStartStuck` flags a phase that overruns its budget +
  `GATEWAY_START_STUCK_GRACE_MS` (the `restore` phase gets
  `GATEWAY_RESTORE_PHASE_BUDGET_MS` because the legacy stage may take 10 min; all
  other phases use `STARTUP_TIMEOUT_MS`).
- When the watchdog trips, `recoverStuckGatewayStart` emits a
  `gateway.start.stuck` structured event, kills the stuck supervised process
  (`killGateway`), clears the startup/readiness caches, and returns so the next
  request starts clean. After `GATEWAY_START_STUCK_MAX_ATTEMPTS` consecutive
  resets it reprovisions the container (`recoverStaleSandbox`).
- Separately, `findExistingGatewayProcess` uses `classifyGatewayStart` to detect a
  `start-openclaw.sh start` wrapper older than the cold-start budget with the
  port closed and kills it immediately (source `startup-wrapper-age`) instead of
  waiting out another full readiness budget.
- A **non-owner** isolate that merely _observes_ a running wrapper waits to the
  same deadline the owner's watchdog uses — `STARTUP_WRAPPER_STUCK_BUDGET_MS`
  (`STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS`) measured from the
  wrapper's start — and re-probes `/health` before any kill. A fixed
  `STARTUP_TIMEOUT_MS` measured from observation expired while the wrapper had
  already `exec`ed the gateway and was still binding, killing it mid-boot,
  releasing the startup lock, and making the waiting second wrapper re-run the
  entire setup (observed on `staging-client-dev`: a ~14-minute cold boot). The
  kill now only fires once the wrapper has genuinely overrun the owner's own
  budget with the port still closed.

**The `port-wait` also completes on a positive `/health` probe.** The SDK's
`waitForPort` drives a container-side port-watch + log-readiness subscription
that can hang indefinitely on some boots **even after the gateway is serving**.
Waiting on it alone let the shared start promise overrun its budget, so the
watchdog force-killed a **healthy** gateway and reprovisioned the container
(observed on `tenant-client-dev`: `gateway.start.stuck phase=port-wait` after
368 s while HTTP proxying returned 200). The wait races `waitForPortResilient`
against the same positive `/health` probe the public proxy uses. If that waiter
reaches its deadline and parks, the watchdog makes one fresh bounded health
check before resetting: a late-positive response emits
`gateway.start.late-ready` and is adopted without killing the gateway. A failed
health RPC is treated as **unknown**, not closed, and defers the reset until a
later request can probe again; only a confirmed closed port follows the normal
stuck-start reset path.

**Process count is not a health signal.** `listProcesses()` returns every
tracked process for the container's lifetime — including `exited`/`error`
entries the Worker never garbage-collects — so the raw length only grows. The
log line now reads `listProcesses returned <total> processes (<running> running)`
and every health decision (`classifyGatewayStart`, the gateway matcher) considers
**only `running` entries**. The SDK exposes no completed-process cleanup, so the
list is append-only; never gate health on the total.

#### Gateway protocol ledgers (`/api/admin/audit-logs`)

The `AuditLogsPage` reads the two append-only ledgers from the Gateway protocol
([docs.openclaw.ai/gateway/protocol/ledgers](https://docs.openclaw.ai/gateway/protocol/ledgers)):

- `GET /api/admin/audit-logs` — audit **activity** ledger (`audit.activity.list`,
  `operator.read`), newest first. Supports `agentId`, `sessionKey`, `runId`,
  `kind`, `status`, `direction`, `channel`, `after`, `before`, `limit`, and
  `cursor`; returns `{ events, nextCursor? }`. The bundled `admin-http-rpc`
  plugin allowlists `tasks.*` but **not** `audit.*`, so this read goes over the
  Gateway **WebSocket** RPC using OpenClaw's reference client
  (`@openclaw/gateway-client/browser` + `@openclaw/gateway-protocol`) bridged
  through `sandbox.wsConnect`, authenticating with the shared gateway token as
  an `operator` (see `src/gateway/gateway-rpc.ts`). It falls back to
  `openclaw audit --json` when the WS path is unavailable (older Gateway); the
  CLI fallback gets the resolved `OPENCLAW_GATEWAY_TOKEN` injected because a
  fresh `sandbox.exec` shell does not inherit the gateway process env.
- `GET /api/admin/audit-logs/tasks` — **task** ledger (`tasks.list`,
  `operator.read`), newest first. Supports `status` (repeatable), `agentId`,
  `sessionKey`, `limit`, `cursor`, and `sortBy`; returns
  `{ tasks, nextCursor? }`. RPC-first over the admin-http-rpc plugin, with an
  `openclaw tasks list --json` CLI fallback (same token injection).

Both surfaces are ADMIN-only (absent from `DIAGNOSTICS_PATHS`) and return an
empty page with an `error` string when the ledger is unavailable, so the
dashboard degrades gracefully instead of failing the request.

> **Why a WebSocket client.** `admin-http-rpc` only exposes an allowlisted
> method subset. Methods outside it (notably `audit.*`, `plugins.*`,
> `secrets.*`, `agents.files.*`) require the wire-v4 Gateway WebSocket
> handshake (`connect.challenge` → `connect` → `hello-ok`).
> `src/gateway/gateway-rpc.ts` adapts `sandbox.wsConnect` to the reference
> client's `GatewayProtocolSocket`; the client version sent in the connect frame
> is injected at build time from `catalog.openclaw` (`__OPENCLAW_VERSION__`).

#### RPC-first agent board + integrations

Eligible dashboard routes try Admin HTTP RPC first, the official Gateway
WebSocket RPC second, and keep the CLI as the final fallback for older
Gateways (helpers return `null`/`false` on transport or method failure):

- **Agent board (`/api/admin/agents`)** — list reads `agents.list` and channel
  bindings in `config.get` over Admin HTTP RPC. If either result is unavailable,
  the Gateway WebSocket client fetches the missing read(s) over one connection;
  the CLI remains the final fallback. Create → `agents.create`; delete →
  `agents.delete`; identity (name/emoji/avatar) → `agents.update`; model (+
  fallbacks) → `config.patch` on `agents.entries.<id>.model`. Explicit
  bind/unbind operations have no matching RPC method and stay on the CLI;
  persona read/write are filesystem operations.
- **Devices (`/api/admin/devices`)** — list, approve, reject, and approve-all
  retain Admin HTTP RPC as the first path, use Gateway WebSocket methods
  (`device.pair.list`, `.approve`, `.reject`) if HTTP RPC is unavailable, and
  retain the CLI as the compatibility fallback.
- **Models (`/api/admin/models`)** and **channel probe
  (`/api/admin/channels/probe`)** — `models.list` and `channels.status` use
  Admin HTTP RPC first, then Gateway WebSocket RPC, then the existing CLI path.
- **Integrations (`/api/admin/integrations`)** — the whole endpoint is
  **WebSocket-free**. `GET /` reads the allowlisted `channels.status` +
  `agents.list` over the **admin HTTP RPC**, and resolves plugin enabled-state
  from `config.get` (`plugins.entries.<id>.enabled`) merged with the static
  `PLUGIN_FIELDS` catalog — no `plugins.list` socket. When no channel accounts
  are configured the supported-channel catalog is synthesized locally from
  `CHANNEL_FIELDS` instead of shelling out to `openclaw channels list --all`.
  Plugin/channel toggles and non-secret configure fields → `config.patch`;
  secrets reload → the `openclaw secrets reload` CLI. `secrets.apply` and agent
  bindings have no 1:1 RPC method and stay on the CLI. CLI fallbacks remain for
  a degraded HTTP RPC transport. The admin RPC fetcher also cancels a late
  `containerFetch` body when its timeout wins, preventing the Sandbox DO's
  "ReadableStream received over RPC disconnected prematurely" warning and the
  churn that dropped later sockets. The same cleanup is shared by the public
  proxy and the `gateway-api`/diagnostics probes via `src/utils/container-rpc.ts`
  (`containerFetchWithTimeout` cancels a late body, `wsConnectWithTimeout` closes
  a late socket). Reading a **running** gateway's logs must not open its live SDK
  stream (it never settles and is later disconnected): `GatewayProcess.getLogs()`
  tails `/tmp/gateway-stdout.log`, and `/debug/processes?logs=true` /
  `/api/admin/logs?id=` refuse a live stream for a still-running process. The
  catch-all proxy logs only an allowlisted header subset (`safeHeaderLog`) —
  never the forwarded map, which carries CF Access credentials and cookies.
- **Config / secrets / agent resolution** — config set/unset and dreaming
  settings use `config.patch` over Admin HTTP RPC, then Gateway WebSocket RPC,
  then the CLI fallback. `POST /api/admin/secrets/reload` uses
  `secrets.reload` (WebSocket); `resolveAgentFlag` (shared by the
  `models`/`knowledge` CLI calls) resolves the roster with `agents.list` over
  Admin HTTP RPC, then WebSocket RPC, then CLI.

> **`secrets apply` has no `--yes`.** The current CLI accepts only `--from
> <plan.json>`, `--dry-run`, `--allow-exec`, and `--json`; the plan itself is the
> confirmation. Both `integrations` and `dreaming` apply without `--yes`.

#### Realtime device-pairing notifications (`/api/admin/devices/ws`)

The dashboard `NotificationBell` subscribes to `GET /api/admin/devices/ws`:

- On connect it pushes a `{ type: "device.pair.snapshot", pending, paired }`
  frame (best-effort, non-blocking), then relays Gateway
  `device.pair.requested` / `device.pair.resolved` / `device.pair.changed` events
  as `{ type: "device.pair.event", event, payload }`.
- The Gateway subscription uses `openGatewayEventStream` (the official
  `@openclaw/gateway-client` browser client over `sandbox.wsConnect`), which
  reconnects with backoff until the browser socket closes.
- Worker-hosted notification sockets rotate after 10 minutes with close code
  `1012`; this is intentionally below the observed Cloudflare load-shed lifetime
  for long-lived Worker invocations. The dashboard reconnects and receives a
  fresh snapshot, releasing the old Worker invocation and Gateway subscription.
- Admins approve/reject directly from the bell:
  `POST /api/admin/devices/:requestId/approve` / `.../reject` (RPC-first
  `device.pair.approve` / `device.pair.reject`, CLI fallback). The client
  helper is `subscribeDeviceNotifications` in `src/client/api.ts`.

**Dashboard notification centre.** `NotificationBell` also renders application
notifications (backup created/failed, backup restored/failed) from the
`NotificationProvider` context (`src/client/NotificationContext.tsx`) that wraps
the admin layout. Long-running admin actions push their result there instead of
blocking on a modal, so the outcome still reaches the admin if they navigate away
before the request settles. The bell badge counts unread app notifications plus
pending device-pairing requests; opening the panel marks app notifications read
and each can be dismissed.

**Persistence-lock indicator.** `PersistenceLockIndicator` (rendered beside the
bell) shows a **closed red lock** while a dashboard-initiated backup or restore
is in flight, and an **open green lock** otherwise. The state comes from
`PersistenceLockContext` (`src/client/PersistenceLockContext.tsx`): the
`BackupHistoryPanel` raises it (`begin('backup' | 'restore')`) when it fires an
async snapshot/restore and drops it (`end()`) when its completion check settles
(success, failure, or timeout), so the indicator is **synchronous with the
operation lifecycle — there is no polling and no lease read**. A counter keeps
the lock held if two operations overlap. The widget is **ADMIN/ROOT only** (a
BASE_USER cannot trigger a backup).

### OTel Ingest (`/api/otel/*`, shared-secret auth via `OTEL_SHARED_SECRET`)

- `POST /api/otel/v1/metrics` — OTLP/HTTP JSON metrics (cost/tokens). AI cost
  rows keep `openclaw.provider` + `openclaw.model` so lerma can attribute the
  spend (unified-billing gateway-routed vs. direct provider / OTel-only).
- `POST /api/otel/v1/traces` — OTLP/HTTP JSON traces (`{resourceSpans:[...]}`).
  Only `openclaw.message.processed`, `openclaw.harness.run`, `openclaw.tool.execution`
  spans are approved; others are counted as `ignored`. Each approved span is
  classified (`src/billing-classification.ts`) into a **semantic** billing
  action (`telegram_query`, `signal_analysis`, `email_draft`, …) — an explicit
  `openclaw.billing.action_type` marker wins, span attributes are the fallback.
  Approved-but-unmapped spans are accepted and counted as `nonBillable`.
  Stable `action_id = lowercase(traceId) + lowercase(spanId)` (no separator);
  `action_key` dedupes a business unit and `trace_id` drives precedence
  (`email_draft`/`review_analysis` suppress `telegram_query`).
  Tenant is required per span (`customer.slug` et al.; one batch = one tenant).
  `application/x-protobuf` is rejected with `415` (use `http/json`).
  See `docs/contracts/otel-trace-action-id.md`. Normalized traces are held
  in-memory for downstream reconcile; Analytics Engine stays source of truth.
  **Emission guard:** `isBillableEventsEnabled(env)` (`src/dev-mode.ts`) decides
  whether an instance may mint billable actions. **Billable production always
  emits and cannot be opted out**, and **the staging platform always emits** in
  every environment so the billing pipeline runs with production fidelity before
  release (its rows land in the staging Analytics Engine dataset). A **dev
  environment** is non-emitting by default — it accepts spans (200) and the cron
  drain drops buffered traces — and emits only when `BILLABLE_EVENTS=true` opts
  it in (e.g. to exercise billing end-to-end from a dev instance). The override
  is emission-only — `getCostContext` still labels rows with the real
  `platform`/`environment` (`staging`/`dev` stay non-billable _cost_ axes), and
  `shouldAutoSleep` is unchanged, so an opted-in dev instance keeps its idle
  timeout and staging keeps its 15m one.

#### Cost-tracking dimensions & development expense

Every Analytics Engine cost-tracking row (`ai-query`, `container-active`,
`r2-operation`, `cron-trigger`, `ws-session`, `action-completed`, `action-step`,
`health-check`) carries two trailing blobs — `platform` (`production` |
`staging`) and `environment` (`production` | `dev`) — resolved by
`getCostContext(env)` (`src/metrics.ts` → `resolveInstanceDimensions` in
`src/dev-mode.ts`). The canonical `customer` key is the **base client slug**
(`CUSTOMER_SLUG`, else `tenant-{slug}[-dev]` with the prefix/suffix stripped),
so worker-side rows join against the `client.slug` lerma reconciles. `health-check`
rows (emitted from non-passive `GET /api/health`) feed lerma's uptime/downtime
statistics. Dev/staging rows are non-billable **development expense** — infra +
AI cost of building for a client, tracked in AE and pulled asynchronously by
lerma, never charged to the client's credit pool.

---

## Software Provisioning (declared, re-applied every reprovision)

Software that agent skills install at runtime (a plugin, an `apt`/`npm`
package, a downloaded binary) lives **only** in the container filesystem, so it
is lost when the image is upgraded or the container is destroyed on idle-sleep.
The `softwareProvisioning` feature flag makes that software declarative:

| Step              | Where                                                                                                                                                                             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Declaration       | `customers/<slug>/software.json` in `BACKUP_BUCKET` (`PUT /api/admin/software/manifest`), `{ id, check, install, verify?, timeoutSec? }` per package                              |
| Materialization   | `materializeSoftwareManifest()` → `/home/openclaw/.openclaw/software.json` (inside the backup root, so a warm restore also carries it)                                            |
| Reconcile         | `moltlazy software provision` (`packages/moltlazy/src/software/`): `check` → skip, else `install` → optional `verify`; per-package timeout, **non-fatal**, exit 0                 |
| Boot trigger      | `ensureGatewayImpl` dispatches the CLI **detached** (`void spawnSoftwareProvision(...)`) just before `start-openclaw.sh start`, so it runs in parallel and never blocks readiness |
| On-demand trigger | `POST /api/admin/software/provision` (ADMIN/ROOT) runs the same CLI to completion, bounded by `SOFTWARE_PROVISION_RUN_TIMEOUT_MS` (5 min)                                         |

It applies on **every provisioning cycle** — cold start, restored boot, idle
wake, image upgrade, and `/debug/restart-openclaw` — because it lives in the
Worker's `ensureGatewayImpl` rather than in the (skippable) setup body of
`start-openclaw.sh`. A soft reload (`SIGUSR2`) does not re-run it; nothing is
lost there.

Outputs: `/tmp/software-report.json` (structured; read by
`GET /api/admin/software/report`) and `/tmp/software-progress.log` (ticks; read
by `/api/startup-progress` as `softwareLog`). The CLI truncates the progress log
at the start of every run because `/tmp` survives an in-place restart, so
otherwise ticks from earlier boots would be interleaved.

Operational notes:

- Both `GET` endpoints return `enabled` so the dashboard can distinguish
  "flag off", "instance asleep", "not run yet" and "nothing declared"; the
  homepage widget's **Run provisioning** button (ADMIN/ROOT) calls the on-demand
  endpoint and renders the fresh report.
- A failed package stays `failed` in the report and does not abort the rest of
  the manifest or the boot. A report is only rewritten when the CLI actually
  runs, so a stale report can exist inside one container after the flag is
  switched off — the `enabled: false` marker is what the UI keys on.
- The manifest is **arbitrary shell** executed in the tenant container. The
  upload and on-demand routes are ADMIN/ROOT-only; there is no command allowlist
  by design (the container is single-tenant).

---

## Container Startup Flow (`start-openclaw.sh`)

`start-openclaw.sh` takes a phase argument (`$1`):

| Mode             | Behaviour                                                                                                                                                                                                                |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `start`          | Production cold-start (single entry point): acquire the container-wide startup lock **first**, then run the full setup body, then become the single OpenClaw owner — `exec`ing the gateway while still holding the lock. |
| `setup`          | Standalone/smoke only: run everything up to (not including) the gateway launch, then exit.                                                                                                                               |
| `launch`         | Standalone/smoke only: acquire the container-wide launch lock and become the single OpenClaw owner (or wait for the current owner to become healthy), then `exec` the gateway.                                           |
| `post-start`     | After the gateway is ready — dev automation + boot-log reset, then exit. Dispatched **detached** (fire-and-forget) by the Worker so it never extends the shared start promise.                                           |
| `full` (default) | Manual/back-compat + boot smoke: run setup, then launch + poll the gateway in the background, then exit.                                                                                                                 |

The Worker's `ensureGateway` drives production cold-start as a **single**
`/usr/local/bin/start-openclaw.sh start` call followed by `post-start`. Unlike
the standalone phases, `start` acquires the container-wide startup lock **before
running setup and holds it through the Gateway `exec`**, so setup and launch are
serialized under the same owner — the Worker no longer invokes `setup` and
`launch` separately. The gateway is **not** a shell background job: the `start`
owner `exec`s `openclaw gateway run` while holding the lock. The lock is held for
the **Gateway process lifetime** via the inherited fd (released exactly when the
gateway exits), so the supervised process _is_ the gateway — tracked by
`listProcesses`/`getProcess` for the life of the container and killable via its
handle. A concurrent invocation that loses the lock polls `/health` and exits 0
without starting a duplicate owner. The serial
`test/e2e/ac_gateway_startup_concurrency.txt` suite checks that exactly one owner
exists and that neither the migration-input-change error nor the SQLite
offline-maintenance error appears. The standalone `setup`/`launch` modes (used
only by smoke/manual runs) keep the older split behaviour — a `setup` `flock`
that serializes the slow setup body and is released before a separate `launch` —
which is why ownership there is enforced solely by the dedicated launch lock.
`full` keeps the legacy background-launch behaviour for manual runs and
`test:gateway:boot`.

**Restored containers skip the setup body (zero-data-loss).** The R2 restore
happens in the Worker (`restoreIfNeeded` → `sandbox.restoreBackup()`), _before_
`start-openclaw.sh` runs — the script never performs the restore. When that
restore actually restored a backup, `ensureGateway` passes
`OPENCLAW_RESTORED=true` in the launch env (set only when the restore reports
`hadBackup && restored`, never for a brand-new tenant), and the script skips the
entire setup body — onboard → plugin installs → `moltlazy patch` → config
validate — emitting those phases as skipped and launching the gateway directly
from the restored filesystem. The restored state is therefore authoritative: the
cold-start budget is spent launching it, not re-mutating it (which could diverge
from or clobber the snapshot, and which wastes startup time). A brand-new tenant
(no backup) still runs the full setup body and onboards normally. Each boot
rewrites a durable receipt at `/tmp/openclaw-restored-setup-skipped` (written on
the skip path, removed on the setup path) so an operator can confirm which path
ran; the local backup E2E
(`test/e2e-backup/zzz_backup_restore_skip_setup.txt`, opt-in via
`E2E_BACKUP_RESTORE=1`) asserts the receipt and that the restored setup state is
left byte-for-byte intact.

`OPENCLAW_RESTORED` is keyed on the **in-memory** restore status
(`getRestoreStatus()` — `hadBackup && restored`, set by `restoreIfNeeded` in the
same isolate that launches the gateway), **not** on the durable R2
`restore-needed` latch. The two are deliberately separate signals:
`restore-needed` is a cross-isolate retry/fence latch (written before
`sandbox.restoreBackup()`, cleared only after restore + legacy-state
materialization succeed), while the skip-setup decision is simply the result of
the restore that just ran. A normal cold restore does **not** require
`restore-needed` to be present — its absence is normal while idle and after a
successful restore; the restore is triggered by the container-state probe
(`containerHasState`) plus the committed recovery point (`stable-backup.json`).
The loading page's "Restoring…" spinner observes the in-memory restore status the
same way, so it appears the moment a cold restore is in flight regardless of
whether the marker was present in R2.

**One targeted exception: the restored boot re-applies only the `gateway`
section.** The Worker-critical `gateway` config (`trustedProxies`, `auth`,
`controlUi.allowedOrigins`) must self-heal even on a restored boot: a
warm-restored `openclaw.json` can carry a stale `gateway.trustedProxies` (e.g.
the pre-2026-09 default `['127.0.0.1','::1']`) that omits the Cloudflare Sandbox
bridge range (`10.1.0.x`) the Worker proxy connects from, so OpenClaw rejects
every proxied request with `proxy_attribution_required` (symptom: the tenant
serves the gateway's JSON error `{"error":{"type":"proxy_attribution_required"}}`
on the root HTML/WS proxy). The skip path therefore runs
`moltlazy patch --only gateway --file "$CONFIG_FILE"` (bounded, non-fatal): it
heals `trustedProxies` (drop catch-alls, union the required ranges — see
`healTrustedProxies`) and re-asserts auth/origins, but never regenerates the
user-editable `agents`/`session`/`tools`/`logging` sections nor re-seeds
workspaces, so the snapshot's user data is untouched (the backup E2E fingerprints
`moltlazy-session.json` + a sentinel, both unaffected).

**The skip is guarded on the restored config actually existing.** A "restored"
boot can carry partial container state but no materialized `openclaw.json` — an
interrupted restore (a container-image rollout changes the runtime mid-restore),
or `restoreIfNeeded`'s "container already holds tenant state; skipping duplicate
restore" short-circuit, which skips the legacy-state materialization yet still
reports `restored=true` and therefore sets `OPENCLAW_RESTORED=true`. Skipping the
setup body then launches OpenClaw with **no config at all** (observed on
`tenant-tarjetero-dev` after the `v2026.10.13` rollout). `start-openclaw.sh`
therefore only honours `OPENCLAW_RESTORED=true` when `$CONFIG_FILE` exists;
otherwise it logs a warning and falls through to the setup body, which
onboards/repairs the config instead of launching a configless gateway.

**The cold boot is a durable `ActivityScheduler` DO job, not a request
`waitUntil()`.** Cloudflare cancels a request's background (`waitUntil`) work a
short time after the response, so a cold boot driven from a request was killed
mid-restore and re-attempted forever: `[Gateway] Gateway start already in
progress … (phase=restore, age=686365ms)` with `start-openclaw.sh` never run,
141 `waitUntil() tasks did not complete` warnings, 14× re-decrypt of the same
archive, and 64 probe processes hammering the Sandbox DO (observed on
`tenant-client-dev`). The boot is now a durable job (`src/activity/boot-job.ts`)
on the `ActivityScheduler` DO: `POST /api/warmup`, non-passive `GET /api/health`
and `/api/startup/ws` only **request** it (`requestBoot`, deduped) and return;
the alarm performs `restoreIfNeeded` → `ensureGateway` and retries on a bounded
lease independent of any request. When the `ACTIVITY` binding is absent (local
dev / unit tests) the routes fall back to the request-scoped `waitUntil` boot.

**A request never owns a destructive restore.** Even with the durable boot job,
every `ensureGateway` call on a cold path re-asserts the restore barrier. A
request that would actually need an archive restore now **defers** it:
`restoreIfNeeded(..., { restoreOwner: 'request' })` (the default for
`ensureGateway`) performs only the cheap non-destructive checks — the
`restore-needed` latch and `containerHasState()` — returns immediately when the
container already holds tenant state, and otherwise throws
`RestoreDeferredToBootJobError`. `ensureGateway` catches it, calls
`requestDurableBoot()`, and throws `GatewayBootPendingError`, so the proxy/route
answers `503` (or the loading page) while the DO alarm performs the restore +
launch. Durable callers (the `ActivityScheduler` boot/wake jobs and the queue
consumer) pass `restoreOwner: 'durable'` and run it inline. The proxy no longer
calls `restoreIfNeeded` itself. Motivation: on `tenant-tarjetero-dev` a 350 MB
encrypted archive was re-decrypted and `sandbox.restoreBackup` aborted at
~30.1 s (`backup.restore error … The operation was aborted`) on every cold
request, because the restore ran under a request execution context that
Cloudflare cancels shortly after the response — a reboot/restore loop while the
tenant was actively used.

Three supporting bounds keep a stalled boot from hiding behind the long restore
phase: the SDK `sandbox.restoreBackup` RPC is bounded by
`RESTORE_BACKUP_TIMEOUT_MS` (fail fast on a hung control channel); the restore
reports sub-phase progress (`onProgress`) so `isGatewayStartStuck` judges the
`restore` phase on `GATEWAY_RESTORE_NO_PROGRESS_MS` of **no progress** rather
than the full 11-min budget; and the persistence `withLease` releases the shared
R2 lease when its enclosing request aborts (`PersistenceDiagnosticContext.signal`),
so a cancelled operation cannot leave `restore-needed` + a held `backup-lock`
re-triggering the loop.

**Snapshot lease renewal and admin snapshot mode.** The shared R2
`backup-lock` TTL is `STEP_DEADLINE_MS` (12 minutes), with a heartbeat every
quarter-TTL (3 minutes). Snapshot work explicitly renews before/after legacy
staging, `sandbox.createBackup`, and encryption, and before final config capture;
restore renews around decrypt and archive application. Renewals are serialized on
one promise chain, awaited before the release tombstone, and best-effort (a failed
renewal is logged but does not abort the snapshot). Structured events report
renewed, late (at least half a TTL), and failed renewals. A 12-minute TTL trades a
longer stale-holder recovery window for avoiding false lease expiry during a
legitimate long step; lock waiters can still hit their bounded acquisition timeout.

`POST /api/admin/storage/sync` uses the durable backup job by default when the
`ACTIVITY` binding is configured (`202` with `jobId`, `409` if another backup/restore
owns the slot). `?inline=1` explicitly requests synchronous completion; `?async=1`
keeps the accepted-then-background behavior. Environments without `ACTIVITY` retain
the inline fallback. E2E scenarios that need an immediate backup id must use
`?inline=1`.

**Never call `output()`/`runShell` on the live gateway handle.** The gateway never
exits, so the SDK's buffered `output()` only opens an RPC log stream, waits for
the local timeout, and disconnects it, surfacing workerd's `ReadableStream
received over RPC disconnected prematurely`. Read diagnostics from the
`/tmp/gateway-stdout.log` / `/tmp/startup.log` tails (`readLogTail`) instead.

**The `setup`/`post-start` phases must stay child-free.** The Worker waits on
them with `waitForExit`, which — despite its name — only settles once the
supervised _terminal_ exits, which never happens while the script leaves a live
background child. `setup` is awaited on the critical path; `post-start` is
dispatched detached but still waited with a bounded `waitForExit` (a lingering
child would keep the post-start process alive until the
`GATEWAY_POST_START_TIMEOUT_MS` kill). So neither phase may start a `tail -f` log
mirror or background `cloudflared`: the mirror is started by the native launch
(`buildGatewayRunScript`), and cloudflared is detached with `setsid`. A lingering
child in `setup` stalls cold start for the full `STARTUP_TIMEOUT_MS` and makes the
Worker re-run setup on retry (observed: ~11 min to first gateway launch).

**`post-start` never blocks readiness.** Once the port wait succeeds and
`markGatewayReady` runs, the gateway is serving; `ensureGatewayImpl` dispatches
`start-openclaw.sh post-start` with `void runGatewayPostStart(...)` instead of
awaiting it. Dev instances run `openclaw automations list/create` there (30–60s),
and awaiting it previously held the shared start promise so every request logged
`Gateway start already in progress … waiting...` while `/health` already returned 200. `startShell` dispatches the `sandbox.exec` synchronously before its first
`await`, so the work still runs server-side even if the isolate is evicted;
`runGatewayPostStart` swallows its own errors (best-effort).

The entrypoint runs sequentially:

```
1. Fast-path check: Is gateway already running? → exit 0 (skipped for `post-start`)

2. Serialize concurrent startup/setup/full invocations (exclusive `flock` on
   /tmp/openclaw-setup.lock, ~900s wait). The Worker's in-process
   `gatewayStartPromises` guard is per-isolate, so bursty cold-start polling can
   land several runs in different isolates at once; without the lock they
   all run the slow setup body concurrently, contend on OpenClaw's native
   config/state locks, and stall cold start for many minutes (observed: ~18 min
   to gateway-ready in run 36692775633). Only one body runs; a waiter re-checks
   the gateway fast-path after acquiring and exits 0 once the winner is up.
   In production, `start` takes this lock **before** the setup body and holds it
   through the Gateway `exec`; `post-start` is outside the lock (it runs after
   the gateway is reachable). The standalone `launch` mode has its own dedicated
   launch lock and must not double-lock. `full` releases fd 9 before
   backgrounding the gateway so the child does not inherit the setup lock.

3. Worker-side lifecycle coordination
   - Restore, snapshot, restart, and destroy are coordinated by the Worker.
   - OpenClaw's native lock protects gateway ownership; the startup `flock`
     above serializes onboarding/config/plugin filesystem mutations **and** the
     launch across concurrent Worker isolates — `start` holds it from before
     setup until the gateway is `exec`'d, so a single owner covers both. The
     standalone `setup`/`launch` smoke modes still split the lock (setup lock
     released before the separate launch lock), so within that path single-owner
     launch is enforced by the dedicated launch lock alone.

4. Pre-backup optimization:
   - Move logs to /tmp (excluded from squashfs snapshots)
   - Clean stale backup files

5. Onboard (if needed):
    - Detect Cloudflare AI Gateway auth from env vars
    - Ensure the `cloudflare-ai-gateway` provider plugin is present before
      onboard. It is **pre-installed into the image at build time** (see
      Dockerfile), so this is normally a no-op detection; the install is a
      resilience fallback. Non-interactive onboarding cannot approve plugin
      capabilities, so without the plugin its plugin phase fails and burns the
      60s onboard cap on every cold start. With it present, the phase is a no-op
      and onboarding completes cleanly.
    - Run: openclaw onboard --non-interactive --accept-risk

6. Enable plugins (single working `config patch`):
   - All three plugins are **pre-installed into the image at build time**
     (Dockerfile, unpinned → latest ClawHub release). The installs below run
     only as a fallback when the extension directory is missing (e.g. a warm R2
     restore replaced state). Detection (`plugin_extension_present`) checks that
     directory — the artifact the gateway actually loads — and deliberately does
     **not** read `openclaw plugins list --json`: the registry refresh hangs for
     its full `CLI_TIMEOUT` when ClawHub/npm are slow, and a hung/partial list
     reads as "not installed" and triggers a needless network install (~57s
     ai-gateway + ~74s otel on a full cold boot on `staging-client-dev`). See
     `docs/plans/cold-start-latency.md`.
   - **Build/boot guard:** every ClawHub install is best-effort. A ClawHub
     outage, a transient network error, or a plugin that requires a newer plugin
     API than the pinned OpenClaw runtime (observed: `@openclaw/cloudflare-ai-gateway-provider`
     resolving to 2026.9.9 against the 2026.9.8 runtime) logs a `WARNING` and
     continues — the Docker build no longer fails, and the boot proceeds. The
     enablement patch below only enables a plugin whose extension directory is
     present, so a failed install cannot invalidate the whole patch.
   - cloudflare-unified-billing (when installed) — the Paso4 plugin, served from
     https://clawhub.ai/paso4/plugins/cloudflare-unified-billing. Entry opts
     into `hooks.allowConversationAccess=true` so the
     `agent_end`/`message_received` hooks run (OpenClaw 2026.8.2 blocks
     non-bundled conversation hooks without it)
   - cloudflare-ai-gateway (when installed)
   - admin-http-rpc (always) — the Worker's RPC-first admin read path
   - diagnostics-otel (enabled only when `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`
     is set and not in E2E mode; otherwise explicitly disabled). Exporter
     protocol is always `http/json`. When `WORKER_URL` is set (and
     not in E2E mode) `buildEnvVars()` also derives
     `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT={WORKER_URL}/api/otel/v1/traces`.
     `diagnostics.otel.traces` is opt-in via `OTEL_TRACES_ENABLED=true`
     (default: disabled); the config shape is unit-tested in
     `src/gateway/diagnostics-otel.ts` (`buildDiagnosticsOtelConfig`).
   - The patch must NOT carry `plugins.bundledDiscovery` (removed in OpenClaw
     2026.9.x — including it fails schema validation and drops the whole patch)
     nor `plugins.allow` (an allowlist disables every bundled provider not
     listed). Memory search is disabled via `memory.search.enabled=false`.
   - **memory-core dreaming cadence (dev):** the same patch slows the dreaming
     sweep to weekly (`plugins.entries.memory-core.config.dreaming.frequency =
     "0 3 * * 1"`, default `0 3 * * *`) when `OPENCLAW_DEV_MODE=true`. The full
     sweep is a heavy isolated agent turn (~6 min, `toolsAllow: ["*"]`) that the
     cron replays on every wake (`cron: running missed jobs after restart`) — a
     major RSS spike source on the 1-vCPU dev sandbox (observed 1.6 → 2.08 GiB
     during a replayed sweep). Production keeps the OpenClaw default cadence.
   - **Version drift:** the plugin-API mismatch above means a build on the
     pinned runtime ships **without** the Cloudflare provider plugins until
     OpenClaw is bumped (see the openclaw-migration skill) or the specs are
     pinned. The guard keeps the image and boot healthy in the meantime; it does
     not restore the missing provider by itself.

7. Patch config via moltlazy:
   - Run: node /app/moltlazy/dist/cli.js patch
   - Generates per-section includes (moltlazy-session/tools/logging.json) injected
     via per-section $include; gateway/agents written inline (no root $include)
   - References the gateway token as an env SecretRef (kept in the container environment)
   - Run: node /app/moltlazy/dist/cli.js validate
   - **Validation is FATAL**: an invalid config would prevent the gateway from
     booting, so the script aborts (exit 1) instead of proceeding.

8. Harden file permissions:
   - chmod 600 moltlazy-*.json, chmod 700 .openclaw

9. Cloudflare Tunnel (optional):
   - Start cloudflared for obsidian-sync (port 18790), detached with `setsid`
     (stdout/stderr → `/tmp/cloudflared.log`) so `setup` stays child-free

10. Start gateway:
    - `start` mode (production): after setup, `exec` the gateway under the same
      startup lock the phase acquired at step 2 — the Worker no longer calls
      `launch` separately (its stdout log mirror / `post-start` run as usual)
    - `setup` mode: smoke/manual-only standalone path — `mark_step gateway`,
      then exit 0 without launching the Gateway. The production Worker uses the
      atomic `start-openclaw.sh start`, which holds the startup lock and `exec`s
      the Gateway, followed by `post-start`
    - `full` mode: background `openclaw gateway run --port 18789 --verbose
      --allow-unconfigured --bind lan`, poll `/health`, then exit while the
      gateway child remains detached
```

### Startup Progress (`/api/startup-progress`)

The loading page shows a transparent boot stepper + live log while the gateway
starts. `start-openclaw.sh` declares its boot phases in a single `STEPS` array
(single source of truth) and emits progress to two `/tmp` files on each phase
(kept in `/tmp` so they stay out of squashfs backups and R2):

| File                         | Content                                                                       |
| ---------------------------- | ----------------------------------------------------------------------------- |
| `/tmp/startup-progress.json` | `{ steps:[{id,label}], currentStep, currentStepId, totalSteps, skipped, ts }` |
| `/tmp/startup.log`           | Clean, **secret-free**, chronological boot detail (verbosity-gated)           |

`mark_step <id> [skipped]` resolves the label/index from `STEPS`, writes the
progress JSON atomically (`mktemp` + `mv`), and echoes the transition to the
container stdout. `log_boot <level> <msg>` writes clean boot detail to
`/tmp/startup.log`. Step transitions are intentionally **not** written to the
log — the loading page's stepper shows step progress, so the log stays clean
(no duplicated step/checked info). Adding a phase = one `STEPS` row + one
`mark_step` call — the loading page renders the list dynamically, so no other
code changes.

`GET /api/startup-progress` (public, no warm-up) reads both files and returns
`{ available, steps, currentStep, currentStepId, totalSteps, skipped, ts, log }`.
It is bounded by `STARTUP_PROGRESS_TIMEOUT_MS` and never calls
`restoreIfNeeded`/`ensureGateway`, so it adds no load to the health path.

The loading page consumes this data over the **`GET /api/startup/ws`** WebSocket
(not by polling). Each push is a combined `{ ...HealthSnapshot, progress }`
payload — the health snapshot (which drives readiness/redirect) plus the boot
steps + log. On socket open the endpoint registers **one** `ensureGateway` warm-up
with `waitUntil`; every subsequent health snapshot is **passive/read-only**. It
does not re-enter `restoreIfNeeded` on every push. This distinction matters when
restore is blocked on an R2 `backup-lock`: the former per-push warm-up queued a
new 120-second restore-lock waiter every 5–30 seconds, which could starve gateway
startup and the pre-replacement checkpoint while the UI kept its socket open.

The loading page shows the boot log panel **while the gateway is starting up** —
both while `start-openclaw.sh` is running its setup body (`processType ===
'script'`) and while the gateway binary is binding its port (`processType ===
'gateway'`). During the earlier restore phase (no process yet) the log is empty,
so the page shows the FourIA logo instead. The `processType === 'gateway'` state
no longer hides the log: `start-openclaw.sh start` `exec`s the gateway and a
`start_boot_log_stream` mirror tails the gateway's own stdout
(`/tmp/gateway-stdout.log`) into the curated boot log, so the log keeps filling
with real launch progress (SQLite migration, plugin load, port bind) instead of
going blank for the minutes the gateway takes to bind — the exact "Launching the
gateway — no progress" window that previously looked stuck. The mirror is killed
(`stop_boot_log_stream`) and the buffer cleared once the gateway is ready.

While a cold restore is in flight, the page also shows a **restoring spinner**
beside the status line (title "Restoring FourIA" / status "Restoring data from
backup…"). It is driven by the health snapshot's in-memory `restoreStatus`
(`hadBackup && !restored`) — the same signal the `startup.ok` probe uses to
report "Backup found but not restored" — or the durable `recovery.restoring`
marker (below), and is cleared the moment the restore commits (or the start
fails/redirects). It is intentionally **not** keyed on the R2 `restore-needed`
latch, so the spinner shows for a restore that would have run anyway (a fresh
container with a committed recovery point).

The page also **names the backup + date being loaded**, so a cold restore is
legible instead of a generic spinner. `/api/health` (non-passive, not-yet-ready)
and `/api/startup/ws` attach a durable `recovery` object read from R2
(`readRecoveryState`): `{ restoring: { id, created } | null, committed: { id,
created } | null }`. `restoring` is the in-flight `restore-target.json` written
by `restoreIfNeeded`/`restoreBackupById` just before `sandbox.restoreBackup` and
cleared on success; `committed` is the `stable-backup.json` recovery point a cold
boot restores. It is read from R2 (not the per-isolate `restoreStatus`) because
the durable boot/restore runs in the `ActivityScheduler` DO, whose in-memory
state is invisible to the request isolate serving the page. Lerma's frequent
**passive** poll never pays the two R2 reads; they are bounded by
`RECOVERY_STATE_TIMEOUT_MS` and fail soft to `null`.

During a restore the loading UI is a **progress bar synchronized with the DO's
own phase events** (not a client-side timer). `restoreIfNeeded` /
`restoreBackupById` call their `onProgress(phase)` hook on every
`persistence.phase`; the DO wires that to `recordRestoreProgress`, which maps the
phase to a label + a monotonic fraction (`describeRestoreProgress`, in
`activity/restore-progress.ts`) and exposes it via `getRestoreProgress()`.
`/api/startup/ws` pushes it as `restoreProgress: { phase, label, fraction }`;
`loading.html` renders the bar and hides the stepper while a restore is in
flight, then switches back to the container's `startup-progress.json` stepper
once the gateway launch begins. A request-scoped restore with no DO feed shows an
indeterminate bar. `/api/admin/boot-state` also returns `restoreProgress` for
operators.

The label stays honest when a restore is interrupted. A failed attempt is marked
before the durable job rethrows (`recordRestoreFailure`), so the bar reads
`Restore attempt failed during "<phase>" — retrying` instead of the last phase.
A kill that skips that catch (e.g. a Worker CPU-limit kill) leaves the last
durable phase in place, so `getRestoreProgress()` appends
`— no progress for N min` once the record is idle for 2 minutes. The fraction
never moves, so a stale bar is labelled rather than presented as live.

Archive decryption (`decryptStream` / `encryptStream` in
`crypto/backup-encryption.ts`) assembles each 4 MiB chunk from R2 reads with a
byte queue that copies each byte once. The earlier `concat`-per-read buffer
recopied the pending chunk on every ~64 KB read, which exhausted the Worker CPU
budget on large recovery archives (`Worker exceeded CPU time limit` during
`Decrypting backup …`). Measured in isolation on a 96 MiB archive: decrypt
1507 ms → 140 ms at 64 KB reads, and 5614 ms → 137 ms at 16 KB reads.

The R2 binding delivers the archive in **4 KiB** reads (local workerd: 64 MiB
arrives as 16,385 reads), so the old per-read `concat` was the dominant cost.
Each restore logs `[crypto] Decrypt read profile` with the read count, bytes,
largest and average read, and chunk count, so the read pattern can be checked
on a live tenant.

The loading page uses a **single push channel** — `GET /api/startup/ws` — and
does not poll while the WebSocket is open. Flow:

1. **Handshake** — the first WS push reports whether the gateway is already
   running; if so the page redirects immediately, otherwise the stream keeps
   warming and pushing step updates.
2. **Boot phase** — while boot steps are in progress, the sandbox pushes the
   health snapshot + `progress` (steps + log) every
   `STARTUP_WS_POLL_INTERVAL_MS` (~5s). No client polling. If a transient
   DO↔container transport drop makes the probe fail, the stream re-sends the last
   known state and backs off exponentially up to `STARTUP_WS_MAX_BACKOFF_MS`
   (30s) until the control session reconnects, so the page keeps waiting and the
   same error is not surfaced/re-logged every 5s.
3. **Fallback** — if the WebSocket drops (e.g. local dev server, a network
   blip), the page falls back to slow HTTP polling (`/api/health` every 5s)
   **indefinitely**. The page has no wall-clock budget or retry cap of its own: the
   cold boot is driven out-of-band by the `ActivityScheduler` DO, which owns
   retries and terminal failure. A run of failed polls does **not** hard-fail the
   page ("Unable to reach server"): a container-image / Worker deploy briefly
   resets the DO↔container control session, so the page keeps retrying and shows
   "Reconnecting — the instance may be updating…".

The page only declares **failure** on a _permanent_ container error
(`degraded: true` in the `/api/health` snapshot — deleted image, missing app,
where a rescue is required), surfaced immediately via the container error, or on
the durable **boot job's** terminal `boot.status === 'failed'` state. Every other
state — still booting, gateway reachable but warming up, restore in progress —
keeps waiting, because a cold start legitimately takes 3–5+ minutes (onboard +
gateway warmup; the provider plugins are pre-installed in the image, so plugin
installs only run on the fallback path). Mixed probe states (e.g. `startup.ok`
lagging behind `readiness.ok` while a backup restore is pending) are **not**
treated as failures, and there is no client-side timeout that interrupts a
long-but-progressing boot.

The loading page treats the canonical gateway status (`status === 'running'`,
driven by the readiness probe) as "ready" rather than trusting `startup.ok`
alone, because `startup.ok` is gated on backup-restore state that can lag a
healthy, already-serving gateway.

**A slow boot step is surfaced as an alert.** Once the current boot step has run
longer than `SLOW_STEP_MS` (120s), the loading page shows a non-fatal notice
naming the step and its elapsed time (e.g. "Still working on "Installing
plugins" (185s)…"), driven by the script's own `progress.ts`. It is **not** a
failure — cold starts legitimately take minutes — and it is hidden on ready/error.
It exists so an operator can tell a slow-but-progressing boot from a stuck one
without opening logs or the Workers Observability console. (See
`docs/plans/cold-start-latency.md`.)

The Worker's root HTML route (`app.all('*')` in `src/index.ts`) uses the **same**
ready predicate (`snapshot.readiness.ok`, i.e. `status === 'running'`) — it must
**not** additionally require `startup.ok`. Requiring both made the two disagree
whenever `readiness.ok` was true but the startup probe lagged (a restore still
settling, a stale cached startup result, or a `start-openclaw.sh start` wrapper
not yet replaced by the gateway binary): the loading page redirected to `/`
while the root route re-served the loading page — an **infinite reload loop**
where OpenClaw never appeared to boot (observed on `tenant-client-dev`). Keep the
two predicates in lockstep.

**The gateway WebSocket proxy gates on the same readiness predicate.** The
catch-all WS path in `src/index.ts` must call `isGatewayReadyForProxy()` before
`sandbox.wsConnect()`. `wsConnect` (and `containerFetch`) wait for their target
port **inside the Sandbox DO** for `containerTimeouts.portReadyTimeoutMS`
(`CONTAINER_PORT_TIMEOUT_MS`, 90s). During a container-image rollout the
container is healthy — Cloudflare reports it active and the container logs are
clean — but the gateway has not bound 18789 yet, so that one RPC parks for the
full 90s and every other RPC (health, components, other proxies) queues behind
it, surfacing as "Unable to reach server" on the loading page. When the gateway
is not ready the WS path returns a fast, retriable `503` + `Retry-After: 5`
(and fires `ensureGateway({ waitForReady: true })` via `waitUntil` so the cold
start still progresses) instead of calling `wsConnect`. The HTML and non-HTML
HTTP proxy paths already gate the same way.

A `"Timed out waiting for runtime port"` error — the DO's port wait expiring
because the container is up but the service has not begun listening — is
classified by `isRuntimePortTimeoutError()` (`src/gateway/sandbox-recovery.ts`)
as **transient**: the WS path returns a retriable `503`, never a `502`, and it
must **not** trigger `killGateway`/stale-container recovery, which would discard
a healthy container mid-rollout. It is deliberately excluded from
`isStaleContainerError`.

Likewise, `Network connection lost`, `WebSocket connection failed`, and the
Worker's bounded `Operation timed out after …` from `sandbox.wsConnect()` are
transient Sandbox DO WebSocket-transport failures, not proof that the gateway
process died. The proxy returns `503` with `Retry-After: 5` and leaves the
container and R2 state untouched. This matters when the admin HTTP RPC
health/status path still works: it uses `containerFetch`, a separate transport
from the browser's WebSocket connection.

**Security invariant:** the boot log contains curated `log_boot` lines — the
script's raw stdout (which can contain the AI Gateway API key as an `openclaw
onboard --cloudflare-ai-gateway-api-key …` argument) must never be written to
`/tmp/startup.log`. In addition, once the gateway launches, a
`start_boot_log_stream` mirror tails the **gateway's** own `--verbose` stdout
(`/tmp/gateway-stdout.log`) into the boot log so the loading page can follow
launch progress; this is the same output already surfaced to ADMIN/ROOT via
`/debug/gateway-logs` and `/api/admin/logs`, and it is stopped (`stop_boot_log_stream`)
and the buffer truncated at readiness. The gateway's startup output is a distinct
stream from the script's onboard stdout (which is where the API key appears as a
CLI argument and is still never captured).

Verbosity is controlled by `STARTUP_LOG_VERBOSITY` (`steps` | `detailed` (default) | `verbose`),
passed through from the Worker via `buildEnvVars()`.

### `moltlazy patch` — Config Injection

`moltlazy patch` is the **mandatory config system**. It runs every startup and generates:

| Config Section     | Type          | Content                                                             |
| ------------------ | ------------- | ------------------------------------------------------------------- |
| `gateway.*`        | Mandatory     | Port, mode, bind, auth                                              |
| `session.*`        | Mandatory     | Session scope, DM settings                                          |
| `tools.toolSearch` | Mandatory     | Tool search configuration                                           |
| `logging.*`        | Mandatory     | Log levels, format, destinations                                    |
| `agents.defaults`  | Always-on     | Default model, system prompt                                        |
| `agents.entries`   | Always-on     | Premade agents keyed by id (dev-only `fouria-builder`, `fouria-qa`) |
| `skills.entries`   | Always-on     | Built-in skills                                                     |
| `memory.*`         | Feature-gated | Memory backends (knowledge graph)                                   |
| `channels.*`       | Env-gated     | Channel tokens (Telegram, Discord, etc.)                            |
| `plugins.*`        | Feature-gated | Plugin management config                                            |

The moltlazy-owned sections (`session`/`tools`/`logging`) are generated into per-section include files and injected into `openclaw.json` via per-section `$include` directives — **not a root `$include`** (root includes fail closed for Control UI saves / `config.apply`). They are regenerated from scratch every startup — never edit manually. `gateway.*` and `agents.*` are written inline into `openclaw.json`; the gateway token is an **env SecretRef** (resolved from the container environment, which `start-openclaw.sh` keeps set).

### Admin RPC (`admin-http-rpc`) — RPC-first read path

The Worker reaches the gateway's admin control plane over the gateway's loopback
HTTP listener using OpenClaw's bundled **`admin-http-rpc`** plugin
(`POST /api/v1/admin/rpc`). The typed client lives in `packages/moltlazy/sdk`
(`createClient`, HTTP `RpcClient`); `apps/fouria/src/gateway/admin-rpc.ts`
supplies the transport — a `fetch`-shaped adapter over `sandbox.containerFetch`
(global `fetch` cannot resolve the container's localhost).

This replaced the CLI-first read path for the hot dashboard routes. CLI
invocations (`startShell(sandbox, 'openclaw ...')`) reap a fresh process and
re-establish a WebSocket on every call (~10–15s); the HTTP RPC path is a single
container request. Routes attempt RPC first and fall back to the CLI
(`runCliFallback` / the existing parsers) whenever the transport is unavailable
or returns an unexpected shape:

- `GET /api/admin/devices` → `device.pair.list`
- `POST /api/admin/devices/:id/approve` → `device.pair.approve`
- `GET /api/admin/models` → `models.list`
- `GET /api/admin/channels/probe` → `channels.status { probe: true }`

**Enablement:** `start-openclaw.sh` enables the plugin with a dedicated
`plugins.entries["admin-http-rpc"].enabled = true` config patch. It is kept
separate from the batched plugin patch because OpenClaw 2026.9.x rejects that
batch (`plugins.bundledDiscovery` was removed upstream), and setting
`plugins.allow` would disable every bundled provider that is not listed. Enablement
applies to a running gateway automatically; no restart.

**Security:** the plugin exposes a full operator control-plane surface on the
gateway listener. It is only reached in-process via `sandbox.containerFetch`;
`src/index.ts` shadows `/api/v1/admin/rpc` with a `404` before the public
catch-all gateway proxy, so it is never reachable through a tenant hostname.
Never remove that guard.

**Cold-start guard (health probe).** `sandbox.containerFetch` auto-starts the
container and **blocks until the requested port is ready**. The RPC-first health
probe (`getRpcHealthSnapshot`, `src/gateway/health-probes.ts`) therefore only
issues the RPC once a gateway process is actually detected; while the container
is cold, or the production `start-openclaw.sh start` wrapper is still running
setup, it returns `null` — RPC health is skipped until the local Gateway
health/port is open — so `/api/health` falls back to the exec-based
`getHealthSnapshot` (which never waits on the gateway port). Callers that already hold a process record pass it as
`{ proc }` to skip the extra scan — `buildHealthSnapshot` does this with the
`findExistingGatewayProcess` result it already fetched.

**Cold-start boot lifetime.** #318 split the boot so the **Worker** launched the
gateway _after_ `start-openclaw.sh setup` exited. `/api/health` is bounded to a
5s snapshot, so the request returns long before the (multi-second) setup
finishes; workerd then cancels the unregistered `ensureGateway` continuation and
the container-side setup completes with **no gateway ever launched** (the
observed cold-start stall: 68 processes, setup exit 0, every `curl /health` exit
7). The fix was to keep the boot alive past the response, and the current runtime
contract collapses the production path into a single
`/usr/local/bin/start-openclaw.sh start` call (startup lock acquired before
setup and held through the Gateway `exec`) followed by `post-start` — the Worker
no longer invokes `setup` and `launch` separately. Any route that triggers a cold
boot must still keep it alive past the response with
`c.executionCtx.waitUntil(ensureGateway(sandbox, env, { waitForReady: true }))`
— `waitForReady: true` returns the shared start promise unwrapped (no read
budget) and dedupes onto the promise the bounded snapshot already started.
`/api/health` and `/api/warmup` do this; do not replace it with a bare
`await ensureGateway(..., { waitForReady: false })`.

The SDK's WebSocket transport (`createWsClient`) is **not** used: the current
OpenClaw wire protocol (v4) requires a `connect` handshake and a paired
`operator.admin` device identity, which the shared-token loopback path does not
provide. HTTP RPC over the shared token is the least-effort RPC-first transport
and is what `admin-rpc.ts` implements.

---

## RBAC & Access Control

### Roles

| Role          | Permissions                                   | Access Scope                     |
| ------------- | --------------------------------------------- | -------------------------------- |
| **ROOT**      | `admin:platform`, `admin:customer`, `api:all` | All tenants, all debug endpoints |
| **ADMIN**     | `admin:customer`, `api:all`                   | Own tenant only                  |
| **BASE_USER** | `api:own`                                     | Own tenant, read-only            |

### Dashboard Pages by Role

| Page               | ROOT | ADMIN | BASE_USER |
| ------------------ | ---- | ----- | --------- |
| Dashboard Home     | Yes  | Yes   | Yes       |
| Agents             | Yes  | Yes   | Read-only |
| Connections        | Yes  | Yes   | Read-only |
| Knowledge          | Yes  | Yes   | Read-only |
| Account/Team       | Yes  | Yes   | Read-only |
| Integrations       | Yes  | Yes   | No        |
| Secrets Vault      | Yes  | Yes   | No        |
| Backups            | Yes  | Yes   | No        |
| Diagnostics        | Yes  | No    | No        |
| Debug (`/debug/*`) | Yes  | No    | No        |
| Audit Logs         | Yes  | No    | No        |

> **ROOT** is the maintainers' intended role. It has full access to debugging tools, container inspection, and platform-level diagnostics at `/_admin/diagnostic` and `/debug/*`. ROOT is restricted to diagnostics-only API paths (`DIAGNOSTICS_PATHS` in `src/routes/api.ts`) — management pages (Agents, Connections, Backups, etc.) are ADMIN-gated. As a diagnostic exception, ROOT can still verify backup connectivity (`Test Backup Connectivity` on the Diagnostic Dashboard → `POST /api/admin/storage/test-connectivity`) without the Backups page.

> **Workflow commit tree.** The tenant's workflow git history (commit tree, branches, tags, releases) is **client data**. It is served by `GET /api/admin/releases/commits`, which is deliberately **not** in `DIAGNOSTICS_PATHS`: ROOT is blocked at the diagnostics gate and BASE_USER by `requireAdminOrRoot`, leaving ADMIN-only access. It is rendered by the `CommitTree` component on the ADMIN-gated Backups page. The tree is produced from the sandbox git repository by the release tooling; until that substrate ships the endpoint returns an empty tree.

> In a **harness/dev mode** (`DEV_MODE` or `E2E_TEST_MODE`), `/debug/*` additionally permits `ADMIN` so the automated E2E suite can drive CLI/operator routes from a bare `workers.dev` worker that has no Cloudflare Access or Dispatch Worker in front of it. Real production (`DEV_MODE=false`, `E2E_TEST_MODE=false`) stays ROOT-only per the table above.

### Agent Push (dev → production promotion)

A tenant owns two instances: its `dev` environment builds and validates agents
(the dev-only `fouria-builder` / `fouria-qa` personas), and its `production`
environment serves them. **Agent push** promotes the built content from dev to
production.

The promoted artifact is the **full promotable manifest**, owned by
`@moltlazy` (`packages/moltlazy/src/promotion/index.ts`): `openclaw.json`, the
moltlazy per-section `$include` files, and the agent `workspace/`. The dev-only
agents and all runtime state — `state/`, `credentials/`, `agents/`, `sessions/`,
`secrets/` — are **never** promoted. Provider/channel credentials carried _inside_
`openclaw.json` do travel; only the runtime secret stores are excluded.

The transport is **HTTP, Worker → Worker**, and the production instance applies
the release as a **custom backup**: it keeps its own runtime state (sessions,
transcripts, the control-plane SQLite) while overwriting config / agents / tools
and re-applying the `gateway` section for its own `WORKER_URL`, then soft-reloads
the gateway. The reload is **off the response path**, so neither dashboard is
blocked.

Routes (`src/routes/api/releases.ts`, `src/services/promotion/`):

| Route                              | Instance   | Behavior                                                                                                                                                               |
| ---------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET /api/admin/releases/commits`  | both       | Workflow commit tree (client data; ADMIN-only, not a ROOT diagnostic).                                                                                                 |
| `GET /api/admin/releases/status`   | production | The current release pointer (staged/applied + agents).                                                                                                                 |
| `POST /api/admin/releases/promote` | dev        | Export a sanitized bundle of the promotable manifest and POST it to `<targetUrl>/api/admin/releases/import`. Refuses a non-http target and a self-target.              |
| `POST /api/admin/releases/import`  | production | Receive + stage a bundle, acknowledge `202`, then apply + checkpoint in `executionCtx.waitUntil`. Requires the `PROMOTION_PUSH_TOKEN` bearer when that binding is set. |
| `POST /api/admin/releases/apply`   | production | Apply the staged release now (idempotent), checkpoint it as a recovery point, and reload the gateway.                                                                  |

Export builds a gzip bundle: the config files + workspace are staged in the
container (dev-agent workspaces skipped), `openclaw.json` is sanitized (dev-only
agent entries removed, ambient-owner pointers repaired), then archived and
streamed out with `sandbox.readFileStream`. Import writes the stream with
`sandbox.writeFile`, extracts over `/home/openclaw` (state untouched — it is not
in the archive), re-sanitizes, runs
`moltlazy patch --only gateway` (restored-boot self-heal), and soft-reloads.
`src/services/promotion/index.ts` keeps its own copy of the exclusion list so the
Worker bundle never depends on the moltlazy config module; a unit test pins both
lists together.

E2E: [`apps/fouria/test/e2e-push/`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/e2e-push/README.md) (local-only;
boots a dev + production Worker pair and pushes over HTTP).

### Auth Flow

1. Cloudflare Access intercepts request → user authenticates (email OTP, Google, GitHub)
2. Access injects `Cf-Access-Jwt-Assertion` header
3. Worker verifies JWT against Cloudflare Access JWKS endpoint
4. `resolveCustomerRole()` resolves the RBAC role — Paso4 ROOT users (platform Access AUD) are always `ROOT`; client users get their lerma-assigned role (`ADMIN`/`BASE_USER`) forwarded by the Dispatch Worker via `X-FourIA-Role`
5. `userRole` and `customerSlug` are set in Hono context
6. Route handlers check role before serving content

### Dev-instance role preview (ROOT → ADMIN)

On a **dev instance** — a `-dev` hostname, or an explicit `OPENCLAW_DEV_MODE`
override (see `isViewAsEnabled` in `src/dev-mode.ts`) — a Paso4 ROOT user may
**view and perform as** a lower role from the dashboard's "View as" switcher:
the `X-Mock-Role` cookie (`POST /api/admin/dev/set-mock`) downgrades a resolved
ROOT role to `ADMIN` (or `BASE_USER`) for that session. This lets support/ops
inspect and drive a tenant's dev dashboard as the client's ADMIN without the
production diagnostics-only restriction.

The preview is a **strict downgrade and dev-only**:

- `resolveCustomerRole()` only honours the cookie when the role it already
  resolved is `ROOT` (`src/auth/middleware.ts`). A non-ROOT user setting the
  cookie cannot escalate — e.g. `ADMIN` → `ROOT` is ignored.
- `POST /api/admin/dev/set-mock` returns `404` outside `DEV_MODE` /
  `E2E_TEST_MODE` / dev instances, and `403` for non-ROOT callers on a dev
  instance.
- The tenant slug is **pinned** to the instance's own tenant; `X-Mock-Customer`
  is ignored in preview mode, so the preview can never cross tenants.
- Production instances are unaffected — `canViewAs` stays false and ROOT keeps
  its diagnostics-only scope.

`GET /api/admin/whoami` reports `canViewAs: true` for a ROOT user on a dev
instance so the SPA renders the switcher (`DevUserSelectorCompact`); local
`DEV_MODE` keeps the existing full mock-role switcher (`devMode`).

### Billing Freeze Gate (F4)

Durable per-customer `billing_frozen` flag (`src/billing-freeze.ts`). Frozen blocks
**new** billable actions only — in-flight actions complete, basic non-billable service
continues. This is **distinct from tenant suspension** (Dispatch/Lerma-level), which
denies everything.

- Durability: persisted per-customer as `billing-freeze/<slug>.json` in `BACKUP_BUCKET`
  (R2 — survives cold starts; never in-memory only). No CustomerRegistry DO exists.
- Lerma → Fouria: Dispatch Worker propagates `billing_frozen` inside `X-FourIA-Metadata`
  (shared Postgres is the source of truth; Fouria never calls Lerma directly).
  Effective freeze = stored flag **OR** header flag (fail closed).
- Fouria → Lerma: `GET /api/admin/billing/freeze-status` (ROOT-readable diagnostics path)
  plus the R2 object itself.
- Writes: `POST /api/admin/billing/freeze` | `/unfreeze` (ADMIN-only; ROOT is blocked by
  the diagnostics-only gate).
- freezeGate(env, customer) is implemented in src/billable-actions.ts; it reads the durable per-customer R2 freeze record (no request headers available, fails open on read error); enforcement happens at emit time inside emitBillableActions, which filters billing-frozen customers' events before writing Analytics Engine and enqueueing to BILLING_QUEUE, so in-flight work completes and basic service continues.

---

## Development Tips

### Debugging the Container

```bash
# Get container ID from Worker logs
docker ps | grep sandbox

# Enter container shell
docker exec -it <container_id> bash

# Inside container: check openclaw status
openclaw status
openclaw devices list --json
openclaw config get gateway.port

# Check logs
tail -f /tmp/openclaw-logs/*.log
```

### Understanding Boot Logs

During cold start, these log patterns are normal:

```
[Gateway] listProcesses returned 0 processes
[Gateway] Starting new OpenClaw gateway...
[ERROR] Uncaught ProcessReadyTimeoutError: Waiting for: port 18789 (TCP)
```

The gateway takes **up to 3 minutes** to become ready. The Worker retries automatically. If errors persist beyond 3 minutes:

1. Check `docker logs <container_id>` for OpenClaw errors
2. Verify CF AI Gateway config (`CLOUDFLARE_AI_GATEWAY_API_KEY` + `CLOUDFLARE_API_TOKEN` + `CF_AI_GATEWAY_ACCOUNT_ID` + `CF_AI_GATEWAY_GATEWAY_ID`)
3. Run `bun run dev:worker` to see full Worker output

### Service Integrations (Cloudflare Tunnel)

The container can run Cloudflare Tunnels for exposing internal services:

```bash
# Obsidian sync on port 18790
# Requires CLOUDFLARE_TUNNEL_TOKEN secret
# start-openclaw.sh auto-starts cloudflared if token is set
```

### CRON and Scheduled Tasks

The Worker runs one `scheduled` handler against two crons (see
`wrangler.jsonc` → `triggers.crons`). The handler branches on `controller.cron`
(`CRON_BILLABLE_DRAIN` / `CRON_OPENCODE_CREDITS` in `src/config.ts`):

| Cron            | Job                                                                                          |
| --------------- | -------------------------------------------------------------------------------------------- |
| `*/2 * * * *`   | Drain buffered billable OTel traces to `BILLING_QUEUE` (retry safety net for the sync emit). |
| `0 */8 * * 1-5` | Enqueue the OpenCode credits check onto `TASKS` (the queue consumer runs it in the sandbox). |

The credits enqueue is deliberately **not** on the frequent tick — otherwise the
OpenCode credits notification would fire every couple of minutes. Staging and
production keep only the weekday 8-hourly cron; the frequent drain is a
local/E2E aid because the synchronous `/api/otel/v1/traces` emit already covers
the normal path.

Containers sleep per a **derived** idle timeout (staging 15m, dev 2h — the Sandbox SDK 1.0
`setInactivityTimeout()` platform cap). **The billable production environment does not
auto-sleep**; it is stopped only by an explicit lerma freeze (`POST /control` →
`sandbox.destroy()`), so a warmed gateway never incurs a cold-start/downtime window after
an idle period. There is no `SANDBOX_SLEEP_AFTER` binding; `resolveSleepAfter()` derives
the value from the instance's own `WORKER_URL` (`src/dev-mode.ts`), the same source the
dev/billable axes use. Waking is driven by the per-tenant `ActivityScheduler` Durable Object
alarm, which fires ahead of the next automation and then runs an interval-gated
R2 checkpoint.

The container's idle-sleep decision is owned by the Worker's Sandbox Durable
Object class, **`TenantSandbox`** (`src/sandbox-do.ts`), exported under the DO
binding name `Sandbox` (`wrangler.jsonc` → `class_name: "Sandbox"`). It extends
the SDK's legacy `Sandbox` and overrides `onActivityExpired()`: on a **non-billable**
instance the hook ignores the **persistent** gateway (its lock wrapper, the gateway
binary, and the `tail -f` stdout mirror) and stops the container once only that
persistent work remains — matching Sandbox SDK 1.0, where code inside the instance
does not keep it alive. Any _other_ running process (an in-flight CLI/backup
command) still counts as activity and defers to the SDK gate. On the **billable
production** environment the hook defers to the SDK's `keepAlive` gate instead,
so the container never stops itself.

When a non-billable instance is due to idle-sleep, the hook checks the R2
catalog first: a committed snapshot **inside the idle window** already captures
the unchanged filesystem (the container has had no activity for the whole
window), so the hook stops inline through `stopIfIdle()` without re-archiving.
That shortcut exists because the previous unconditional snapshot was itself
renewing the DO activity window, so the stop-time re-check refused and the next
idle expiry snapshotted again — a loop of 422 MB archives that kept
`tenant-tarjetero-dev` awake. Otherwise (no recent snapshot, no `BACKUP_BUCKET`,
or a catalog read failure) it enqueues a durable **deprovision** job (mode
`sleep`) on the `ActivityScheduler` DO, which snapshots the tenant and only then
calls `sandbox.stopIfIdle()` — fail closed, so a container is never slept without
a committed archive. Without the `ACTIVITY` binding (local/dev) it falls back to
stopping inline. See **Backup before deprovisioning** below.

Lerma's `HealthMonitor` probes the **passive** `GET /api/health?passive=1` every
few minutes (authenticated at the Cloudflare edge with a service token — see the
path-scoped public-health Access applications in `apps/fouria/iac/access.tf`).
Passive probes never run the exec RPCs that would cold-start a stopped
container: `getHealthSnapshot` first does the non-waking `isRuntimeActive()`
read and reports `status: "sleeping"` (HTTP 200, healthy-idle) for an idle
container. Health probes are observability-only and never schedule a backup.

The container runs scheduled tasks via OpenClaw's cron system.

### Task Buffer (Cloudflare Queues)

The Worker is a durable buffer for OpenClaw-side jobs (`src/tasks/`): it never
implements the job logic itself.

- **Producer** — `scheduled` enqueues the recurring OpenCode credits check onto
  `TASKS`; `POST /api/admin/tasks/opencode-credits-check/enqueue` is the manual
  trigger (ADMIN/ROOT).
- **Consumer** — `default.queue` wakes/heals the sandbox via `ensureGateway`,
  runs `openclaw automations run opencode-credits-check` inside it, then acks
  (Cloudflare Queues own retries/backoff; `max_retries: 5`).
- `POST /api/admin/tasks/opencode-credits-check` is the manual consumer trigger
  (runs the automation directly, bypassing the queue). It clamps its total
  wall-clock to `TASK_TOTAL_TIMEOUT_MS` (90s) — the gateway wake, exec launch,
  CLI wait, and log recovery are each clamped to the remaining budget — so a
  cold/contended boot always answers within the endpoint's ceiling instead of
  outliving the caller (the e2e suite's `curl --max-time` hit 120s exactly on
  cold runs). The shared gateway start promise keeps booting in the background.
- **Automation definition** — `start-openclaw.sh`'s `ensure_opencode_credits_automation`
  (dev-only) **creates** the recurring `opencode-credits-check` cron job; the
  Worker only **runs** it. The guard queries `openclaw cron list --all --json`
  in addition to the legacy `automations list --json`, which missed a prior
  boot's job and produced two duplicate `opencode-credits-check` jobs on a dev
  instance.
- **Heartbeat cadence** — `start-openclaw.sh`'s `ensure_dev_heartbeat_cadence`
  (dev-only) reconciles the `heartbeat-fouria-builder` job on every post-start.
  OpenClaw 2026.10.5 retired `agents.entries.*.heartbeat`: a heartbeat is now an
  ordinary automation job, and config reloads/restarts never rewrite an existing
  job, so the instance created by the migration kept the **default 30-minute**
  cadence even though moltlazy declares `heartbeat.every: 12h` (observed on
  `tenant-tarjetero-dev`: 30-minute heartbeats for 9 days, ~317k tokens each on
  the main session, until the OpenCode subscription hit 0%). The reconciler
  reads `openclaw automations get <id> --json`, compares
  `schedule.everyMs`/`sessionTarget`/`payload.lightContext`, and only then runs
  `automations edit --every 12h --session isolated --light-context` (split so a
  rejected `--light-context` cannot lose the cadence fix). It never resets the
  schedule anchor when the job already matches.

### Telegram Webhook Ingress (#364)

Managed tenants switch Telegram to **webhook mode** so an inbound update wakes a
sleeping container (long polling runs _inside_ OpenClaw and cannot wake it). The
Worker is the always-on ingress (`src/telegram-ingress/`):

- **Route** — `POST /telegram-webhook/:account` is mounted **before** Cloudflare
  Access (secret-auth, not JWT). It resolves the account's config from a
  Worker-readable F4E1 registry, constant-time validates
  `X-Telegram-Bot-Api-Secret-Token`, and rate-limits — all _before_ any container
  wake, so a rejected request never starts or renews the container.
- **Registry** — `telegram-ingress/<slug>/<account>` in `BACKUP_BUCKET`
  (F4E1-encrypted, `BACKUP_ENCRYPTION_KEY`). Holds the per-account webhook secret
  - bot token, readable by the Worker _without_ the container running. Populated
    by `POST /api/admin/telegram-ingress/:account` (multi-bot).
- **Hot path** — when the gateway is running and ready, the update is forwarded
  straight to OpenClaw's webhook route (port 18789) and its
  `x-openclaw-delivery-accepted` response is passed back to Telegram.
- **Cold path ("instance is not in hot mode")** — the Worker notifies the sender
  via `sendMessage` ("waking up…"), enqueues the update on the
  `TELEGRAM_INGRESS_QUEUE` (Cloudflare Queue), acks Telegram (the message is now
  durable), and kicks the cold start. The queue consumer wakes/heals the gateway,
  waits for readiness, and forwards — acking **only after OpenClaw confirms
  receipt**, so a message is never "popped" until it has arrived (at-least-once,
  mirroring Telegram's own delivery). `max_retries: 10, retry_delay: 60s` covers a
  multi-minute cold boot.
- **Freeze policy** — enforced upstream by the Dispatch Worker: a
  frozen/operator-stopped instance is never woken by a Telegram update (no user
  role → always non-ROOT); it gets an explicit `503` and the message stays in
  Telegram's retry queue.
- **Dispatch ingress** — a dedicated host (`hooks.fouria.io` /
  `hooks.dev.fouria.io`, `TELEGRAM_INGRESS_HOST`) excluded from the wildcard
  `*.fouria.io` Access app. It resolves `/v1/<slug>/<account>`, applies the freeze
  gate, and forwards (rewritten to `POST /telegram-webhook/:account`) with no
  Access JWT.
- **moltlazy** — `start-openclaw.sh` passes the Worker-injected
  `TELEGRAM_WEBHOOK_MANIFEST` to moltlazy, which writes
  `channels.telegram.accounts.<account>.webhookUrl/webhookSecret/webhookPath/legacyWebhook`
  (fill-only, preserving the user's `botToken`/`dmPolicy`).
- **Queue observability** — the per-tenant `TelegramIngress` DO records every
  ingress event (bounded counters + the last 20): `queued` (a cold update was
  enqueued and woke the container), `forwarded` (hot path), `delivered`
  (queue consumer confirmed receipt), `failed` (a delivery attempt failed and
  will retry). `GET /api/admin/telegram-ingress/status` returns the snapshot, and
  `GET /api/admin/telegram-ingress/ws` streams it (a frame per change) for the
  dashboard. The admin shell's `TelegramIngressProvider` opens that socket only
  while the feature is in use, raises a **notification-bell** entry ("Telegram
  message queued — waking your instance", or a delivery-failure alert), and feeds
  the **Telegram Queue** widget on the home page (state + queued/delivered/failed
  - last event), which is itself hidden until the ingress is in use.

### Backup and Hard-Restart Policy

Backups are snapshots of `/home/openclaw`, not a live filesystem journal. Agent
workspaces are therefore kept under `/home/openclaw/workspace`; files outside
that tree are not protected by the Sandbox snapshot.

The Sandbox SDK's `createBackup` returns a handle marked with `localBucket: true`
when local-bucket mode is enabled (`persistence.ts`). That mode is an explicit
`BACKUP_LOCAL_BUCKET=true` opt-in (falling back to `DEV_MODE` only when unset, for
local development) — it must **not** be keyed on `DEV_MODE` alone, because the
cloud E2E deploy sets `DEV_MODE=true` merely to bypass Access and its backups are
real remote-R2 snapshots (see `ENV-VARIABLES.md`). The flag is persisted in
`backup-handle.json` and forwarded to `restoreBackup()`, so a local backup is
always restored through the local binding — never through the remote
presigned-URL path, which 404s for archives that exist only in the local bucket.
Legacy catalog entries without the flag default to the current local-bucket mode,
and a presigned-URL 404 during restore is treated as a gone backup and pruned from
the catalog.

Health probes are **observability-only**: `GET /api/health` must never write the
R2 backup catalog — no `scheduleBackupIfNeeded`, passive or active. A snapshot is
only meaningful for a **live** instance (a sleeping container has no workload, so
its filesystem has no changes to store), and waking one purely to snapshot would
restart billing. Snapshots therefore run only against a running runtime:

- the **Dispatch Worker cron** sends a `backup` control action; the tenant
  asserts `sandbox.isRuntimeActive()` (a non-waking bridge read) and returns
  `skipped: 'not-running'` for a sleeping container instead of waking it. A
  running instance is acknowledged immediately (`202`) and the snapshot is
  enqueued as a **durable job** on the per-tenant `ActivityScheduler` DO — a
  `waitUntil` task is cancelled ~30s after the response (platform cap) and was
  observed dropping the snapshot mid-cold-boot, so the cron never blocks on it
  and the alarm runs it to completion;
- `POST /debug/restart-openclaw` quiesces the gateway and checkpoints inside a
  `restart` maintenance window (see the hard-restart paragraph below);
- `POST /api/admin/storage/sync` is the explicit admin checkpoint. `?async=1`
  acknowledges with `202` and snapshots in `waitUntil`; the default (no flag)
  blocks and returns the result, which the E2E checkpoint relies on;
- `POST /debug/backup/sync` and `POST /debug/backup/async` (ROOT) trigger the
  blocking and non-blocking checkpoint modes on demand;
- `POST /api/admin/storage/reconcile` (ADMIN/ROOT) runs the catalog heal
  (`reconcileBackupCatalog`) on demand: adopt orphaned `backups/*/meta.json`
  archives, sweep metadata-less uploads, re-encrypt plaintext archives, and
  enforce `BACKUP_MAX_VERSIONS`. The durable `repair` step runs the same work
  before every scheduled snapshot, and `GET /api/admin/storage/backups` is a
  **pure read** that never heals the catalog.

Scheduled snapshots additionally refuse to run while a `restore-needed` marker
is present, rechecked under the shared lease, so a wake that lands mid-restore
cannot snapshot and promote a partial filesystem over the committed recovery
point. The marker is the durable, R2-backed restore latch — the scheduled gate
does **not** use a container-state probe, because the legacy `exists` probe can
fail in a deployed container and failing the gate closed would silently disable
every scheduled backup.

Soft reload (`POST /api/admin/gateway/restart`) does not snapshot. When no live
gateway is found it starts the gateway through `ensureGateway` (never arming
`restore-needed`, so live changes survive) and returns `409` during a maintenance window.

Snapshot promotion is further gated. A snapshot taken while `restore-needed` is set
over a committed recovery point is archived but does not move the pointer, and a
pre-replacement `checkpoint` is refused with `409` (`RestorePendingError`) before the
gateway is touched. A tenant with no recovery point yet may promote its first snapshot.
A restore-only restore (`setRecoveryPoint: false`) writes `recovery-hold.json`: scheduled
snapshots archive without promoting until an explicit snapshot clears the hold.

All three snapshot paths — scheduled, manual sync, and the hard-restart
checkpoint — share **one tenant-scoped R2 `backup-lock` lease**, so only one
isolate ever stages the fixed legacy stage directory or calls
`sandbox.createBackup` at a time. The lease has a 60s TTL, is renewed every 20s
while a snapshot is in flight, and is released by compare-and-swap to a
tombstone (never by deleting the object, which would race a concurrent
takeover). The two entry paths differ only in how they acquire it:

- **Scheduled** acquisition is non-blocking — `scheduleBackupIfNeeded` takes the
  lease with `tryAcquireLock` and, if another isolate holds it, skips this tick
  (returns without waiting).
- **Manual** (`POST /api/admin/storage/sync`) and the **hard-restart checkpoint**
  wait for the lease with bounded exponential backoff (up to **120s**), then
  **fail with an error** instead of silently reporting success.

The lease is held across the legacy stage + `sandbox.createBackup` consumption
and released via the CAS tombstone when the snapshot finishes. The dashboard's
`PersistenceLockIndicator` mirrors the lock in the UI from the operation
lifecycle (`PersistenceLockContext`) rather than reading the R2 lease: the
`BackupHistoryPanel` raises it when it fires an async backup/restore and drops it
when the completion check settles, so there is no polling.

**Async admin operations + non-blocking Backups page.** A manual snapshot or a
restore can wait up to the 120s lease and then transfer a large archive, so
holding the admin HTTP request open is a poor UX and a needless failure mode (a
restore that lost the lease race surfaced as a bare `500` after ~132s, with the
underlying error dropped). Both admin routes therefore accept `?async=1`:
`POST /api/admin/storage/sync?async=1` and `POST /api/admin/storage/restore/:id?async=1`
return `202 { accepted: true }` and run the work in `executionCtx.waitUntil`. The
`BackupsPage` reads the fast raw config (`GET /api/admin/config/raw`, a single
`cat`) instead of the slow structured `/api/admin/config` (which awaits
`ensureGateway` + up to two 30s shell waits) and paints every section
independently, so no single request blocks the page. `BackupHistoryPanel` fires
the async endpoints, immediately pushes a "started" notification to the bell,
then polls the durable job (`GET /api/admin/storage/jobs/:id`, `getBackupJobStatus`)
while a snapshot is in flight (falling back to the history diff when the server
returns no job id) or `/api/health` (restore) and pushes success/failure when the
operation completes — the outcome reaches the admin via `NotificationProvider`
even after navigating away. Restore failures are now `console.error`'d for
Observability.

**Crash-consistency (issue #344).** If a Worker isolate is killed by its CPU
budget between `sandbox.createBackup()` uploading the archive and the catalog
write, the archive would otherwise be orphaned (present in R2 but absent from
`backup-handle.json`, hence invisible to `listBackups`/restore). Two guards
prevent/repair this: the catalog entry is committed **immediately after the
archive is created** (after encryption, before the config-version capture and
retention prune), and the catalog is **reconciled explicitly** from
`backups/*/meta.json` by `reconcileBackupCatalog` (the durable `repair` step,
`POST /api/admin/storage/reconcile`, `restoreIfNeeded`,
`scheduleBackupIfNeeded`, and the reprovision guard) whenever an archive is
missing from the handle. `getBackupHistory` is a **pure read**: a dashboard poll
never adopts, prunes, sweeps, or writes. The scheduled `backup` control action
also runs under a dedicated CPU budget (`BACKUP_CONTROL_CPU_MS` in
`workers/fouria-dispatch`) so the orchestration can finish; a killed isolate's
lease is reclaimed via the 60s TTL.

**Retention is enforced by the reconcile, not by a catalog read.** The two are
coupled: an isolate killed after `createBackup()` leaves an orphan that
reconciliation adopts, and if later snapshots then fail before their own prune
the catalog and R2 grow past `BACKUP_MAX_VERSIONS` indefinitely (observed on
`tarjetero-dev`: 7 archives under `maxVersions: 3`, the catalog written entirely
by reconcile, with the scheduled snapshots failing at the legacy stage).
`reconcileCatalogState` (behind `reconcileBackupCatalog` and
`reconcileOrphanedBackups`) runs `cleanupOldBackups` after merging, and the
durable `repair` step runs it before the `gate` step of every scheduled snapshot,
so the cap holds however the snapshot path ends. It still performs no
write/delete when the catalog is already within the cap, and no read path can
trigger it.

**The catalog is a compare-and-swap, not a last-writer-wins file.** Every
`backup-handle.json` write goes through `mutatePersistenceStore(bucket, mutate)`
(`persistence.ts`): it reads the handle and its etag, computes the caller's
mutation, and writes with `onlyIf: { etagMatches }` (`etagDoesNotMatch: '*'` for a
create-only handle). On a conflict it re-reads and **re-applies the mutation**
against the fresh catalog — never a blind merge, because retention and restore
mutations legitimately remove entries — up to five times before raising
`PersistenceStoreConflictError`. A mutation returns `null` to skip the write.
Reconciliation, snapshot finalization (`catalog.record-outcome`), the retention
prune and the schedule-timestamp commit all run inside this CAS, so a dashboard
poll overlapping a snapshot can no longer drop the newly committed archive (or
resurrect a pruned one).

**R2 enumeration follows `truncated`/`cursor`.** `listAllObjects(bucket, prefix)`
backs the backup reconcile sweep, the orphan archive sweep, `cleanupOldConfigVersions`
and `listConfigVersions`. Previously each site read a single page (1,000 backup
objects, 500 config versions) and silently treated it as the whole result. A
truncated page without a forward cursor logs and stops rather than looping; page
sizes are otherwise unchanged.

**Restore robustness.** `restoreIfNeeded`/`restoreBackupById` persist the
`restore-needed` marker before changing the mounted filesystem or applying an
archive. This includes an ordinary cold restore where no marker existed before
startup: if the restore or legacy materialization fails after writing partial
state, the next invocation retries instead of treating a leftover config file as
proof of success. The marker is cleared after `restoreBackup` and legacy
materialization both succeed, or when candidate scanning finds no restorable
archive and there is no stable recovery point (the explicit clean-start case). The
pre-restore `umount` is best-effort: `|| true`
guards the shell command, but the **exec RPC itself** can still throw (a
cold-start/platform interruption); that is caught and logged instead of aborting
the restore before the archive is applied. An empty catalog read also never
deletes a pending marker — it can be transient (a reconcile `list` failure, or
an archive not yet visible), and dropping it would silently lose a requested
retry.

`scheduleBackupIfNeeded` refuses to run while the durable `restore-needed`
marker is present (or its status cannot be read), rechecking under the shared
lease, so a scheduled wake that races a restore cannot snapshot a partial
filesystem and replace the stable recovery point. It deliberately does not gate
on a container-state probe: the legacy `exists` probe is unreliable in a
deployed container, and a false "no state" would silently disable scheduled
backups. The marker is normally absent while idle or after a successful restore;
cold restore selection uses the committed recovery point, not marker presence
alone.

When candidate scanning finds no restorable archive and there is no stable
recovery point, restore remains a permitted clean start; the temporary retry
marker is removed so onboarding and the next scheduled snapshot can proceed. If
a stable recovery point exists but cannot be restored, startup fails closed and
the marker remains.

**Fail-closed restore (no silent empty boot).** A committed `stable-backup.json`
is the tenant's recorded recovery point, so its presence means the tenant was
initialized. If `restoreIfNeeded` finds a recorded recovery point but **no
restorable archive** (an empty catalog with no stable pointer is only a
brand-new tenant; a catalog drained of its archives, or a stable pointer whose
archive is gone, is not), it now throws `TenantStateUnavailableError` instead of
reporting a clean start. Letting the container boot empty would run onboarding
against an empty filesystem and the next snapshot would overwrite the last good
recovery point — silent data loss. The cold start aborts (the gateway never
launches), `restore-needed` is preserved, and the instance stays visibly
unavailable for an operator to recover. When the catalog is empty but the stable
pointer _is_ present, the recorded entry is attempted directly first, so a
transient catalog-read failure still restores rather than falsely failing
closed.

**Restore skip is keyed on container state, not runtime liveness.** The
cross-isolate cold-boot dedupe previously skipped a restore when
`isRuntimeActive()` returned true ("Runtime already active; skipping duplicate
restore"). But a freshly provisioned container — started by the prelaunch port
probe, or replaced by an image rollout — is already "active" while its
filesystem is **empty**, so that check let a replaced container skip its
restore and come up empty (observed on `tenant-client-dev`: the skip logged 34×
alongside legacy-stage bridge timeouts). `restoreIfNeeded` now probes the
container's actual state (`containerHasState`, the restored/onboarded
`openclaw.json` under the legacy `/root` root or the v1 `/home` root) instead:
absent state means a fresh container that must restore. It is always safe to
attempt an idempotent restore; it is never safe to infer state from a running
process. A probe failure is treated as "no state" (restore), never as "already
restored". `isRuntimeActive()` remains the correct read for the non-waking
health/sleep paths, where the question really is "is the runtime up?", not "is
the tenant state present?".

**Pre-replacement checkpoint (`X-FourIA-Control: checkpoint`).** The `/control`
route gained a **synchronous, verified** `checkpoint` action. The control plane
(Lerma) calls it immediately before an image rollout / container replacement, so
the state about to be discarded is durably captured first. Unlike `backup`
(interval-gated, `202`-accepted, best-effort) it bypasses the interval gate, runs
the full snapshot orchestration, and **fails closed** (non-2xx) unless the
committed archive's `meta.json` is present and restorable. A sleeping/destroyed
instance has no live state and is reported as `skipped: not-running` rather than
woken. The Dispatch Worker forwards `checkpoint` with the same raised CPU budget
as `backup` (`BACKUP_CONTROL_CPU_MS`). In Lerma, both upgrade paths
(`TenantProvisioner`'s `@upgrade_pipeline` `:checkpoint` step and
`Instances.do_upgrade`) invoke `Cloudflare.Containers.checkpoint/1` before the
rollout. The step is **best-effort**: a control-plane failure is logged and the
upgrade proceeds, so a checkpoint problem can never brick a valid upgrade.

**Control-plane reachability (Cloudflare Access).** The Dispatch Worker's control
endpoints (`POST /_control/:slug/{freeze,unfreeze,backup,checkpoint}`,
`GET /_invalidate/:slug`) sit behind Cloudflare Access on the platform domain
(`dev.fouria.io` staging, `dash.fouria.io` production). Without a path-scoped
Access application those headless calls are redirected to the Access login and
return HTML, so the checkpoint is skipped. `apps/fouria/iac/access.tf` declares
`public_control_apps` with a **bypass** policy (same posture as the Lerma
`/api/*` CI bypass), so the path is reachable without an Access login; the
Worker still enforces its own internal bearer token on top. Lerma must send that
bearer, so its deploy sets `INTERNAL_DISPATCH_INVALIDATE_TOKEN` to the dispatch's
hardcoded key (`internal-fouria-dispatch-key`) — without it the internal calls
403 and the checkpoint cannot run.

**Checkpoint outcome telemetry.** A successful tenant checkpoint returns the
archive `backupId`, `quiesced` (whether `killGateway` completed), and
`maintenanceWindow` (whether the cross-isolate R2 guard was opened). Dispatch
forwards only those checkpoint fields, and Lerma logs them as
`backup_id`/`quiesced`/`maintenance_window`; `skipped` is reported separately
when the tenant is not running. Require both `quiesced: true` and
`maintenanceWindow: true` to claim the guarded quiesce path ran; a successful
snapshot by itself only proves archive creation.

**The checkpoint takes the R2 lease before quiescing.** A live `createBackup`
can race the gateway's own writes, so after acquiring the shared `backup-lock`
lease the action records a durable, **cross-isolate** maintenance window
(`src/maintenance.ts`, `maintenance/replacement.json` in `BACKUP_BUCKET`) and
then stops the gateway (`killGateway`) before snapshotting. A busy lock fails
before maintenance is opened or the gateway is stopped; it must never drain the
tenant while waiting for another snapshot. While the window is open,
`ensureGateway` refuses to start the gateway
(`GatewayMaintenanceError`) so a request served by another isolate cannot
resurrect it mid-snapshot; the window has an explicit expiry (15 min) so a
crashed controller can never wedge a tenant, and a malformed/expired record is
discarded (fail open). The `checkpoint` action always restores service and closes
the window in a `finally`, even when the snapshot fails, so a failed checkpoint
never leaves the tenant drained. Its recovery restarts the **existing live
filesystem** with `skipRestore: true`: the snapshot attempt did not replace the
container, and trying to restore while recovering from a lock timeout would
enqueue another waiter behind the same lock. The proxy maps maintenance refusal
to a retryable `503`.

`restoreIfNeeded` also coalesces concurrent calls within a Worker isolate by
R2 bucket. The startup WebSocket allows only **one in-flight**
`ensureGateway({waitForReady:true})` warm-up, retrying only after a failed attempt
settles and with bounded exponential backoff. Its recurring health snapshots are
**passive/observation-only**, and startup-progress shell reads run only when the
startup wrapper is actually visible; it does not start a fresh restore attempt
or shell RPC on every poll. Together these prevent a long-lived startup stream
from multiplying 120-second waiters behind one in-flight persistence operation
or waking an empty container ahead of the restore.

Checkpoint recovery restarts the existing container filesystem with
`ensureGateway({ bypassMaintenance: true, skipRestore: true })`. A checkpoint
does not replace or clear that filesystem; if its snapshot cannot acquire the
shared backup lock, recovery must not call `restoreIfNeeded` and enqueue another
waiter on the same lock it just timed out waiting for.

**Only the snapshot is fail-closed.** The maintenance window and the gateway stop
are _best-effort guards_: a failure to open the window or to stop the gateway
falls back to the previously-shipped **live snapshot** instead of failing the
replacement (a broken guard must never block an upgrade). The post-snapshot
`meta.json` check is retried briefly before it is treated as a real failure, so a
transient R2 visibility gap cannot fail an otherwise-successful snapshot. The
checkpoint returns `quiesced: true` only when the gateway was actually stopped.

This closes most of the planned-replacement loss window, but it is **not** yet a
full drain-to-replacement fence: the gateway is restarted after the checkpoint,
so writes between the checkpoint and the rollout are still outside the archive,
and the shared control-plane coordination record is not implemented. Durable
primary storage for **unexpected** container loss remains the gated next step in
`docs/plans/legacy-sandbox-zero-data-loss.md`.

The pre-snapshot **legacy-stage shell** runs the node:sqlite online backup plus a
symlink-preserving `tar` pipe of the legacy roots, with
`--exclude='*/node_modules/openclaw'`. It deliberately **preserves symlinks**
rather than dereferencing them (`cp -aL`) and **excludes the image-provided
shared OpenClaw package** (`extensions/<id>/node_modules/openclaw` → the image's
`/usr/local/lib/node_modules/openclaw`). `-L` expanded that package into every
archive (three full copies), which pushed the stage past its bound and — worse —
made the **restore** materialization a multi-minute copy (observed on
`staging-client-dev`: `cp -a … /root/openclaw/.openclaw/` running 8m45s, starving
the gateway launch, leaving `restore-needed` set, and re-running the restore
every boot). The restore bridge (`materializeLegacyRootsAfterRestore`) uses the
same `tar --exclude` pipe, so an old backup's dereferenced copy is never
materialized; a copy-then-`rm -rf` is equally slow because deleting a
squashfs-lower tree requires copying each file up first. `restoreIfNeeded` clears
the `restore-needed` marker immediately after a successful materialization
(before the analytics bookkeeping) so an isolate eviction cannot leave a stale
marker. On a large tenant the stage can exceed the
old 300s bound (observed killing the `tarjetero` stage at exactly 300s — issue
#344 follow-up). The bound is `LEGACY_STAGE_TIMEOUT_MS` (default `600000`), the
idempotent stage is retried **once** on a transient timeout/transport failure, and
a failure logs `exit`/`timedOut`/stderr plus a structured process event with the
Sandbox `processId` for cross-dataset correlation. The stage wait is
**transport-resilient** (`runShellResilient` → `waitForExitResilient`): a Sandbox
DO↔container drop mid-stage reattaches by process id and resumes within the stage
deadline, so a mid-stage DO reset no longer ends the request. The Dispatch
`backup` CPU budget (`BACKUP_CONTROL_CPU_MS`) is aligned to the same 10 minute
bound.

The scheduled `backup` control action acknowledges immediately (`202
{ ok: true, accepted: true, jobId }`) and enqueues the snapshot as a durable job
on the per-tenant `ActivityScheduler` DO, so neither the dispatch cron nor an
operator request is bounded by it. The DO alarm runs the job past the request
and retries under a bounded lease — the previous `waitUntil` approach was
cancelled ~30s after the response and could drop a snapshot that was still
cold-booting. Combined with the crash-consistent catalog commit and orphan
reconciliation (#344) the archive stays durable even if the isolate is replaced
mid-snapshot. `POST /debug/backup/async` and `POST /debug/backup/sync`
(DEBUG_ROUTES, ROOT) expose the two modes on demand.

**Config-version snapshots.** Alongside the archive, each snapshot captures
`/home/openclaw/.openclaw/openclaw.json` into R2 under `configs/<timestamp>.json`
(30-day retention) for the Backups page's Config Version History / diff. The
capture normalizes `sandbox.readFile()` across SDK generations
(`readSandboxFileText` in `persistence.ts`): the legacy SDK returns
`{ content, … }`, SDK 1.0 returns a `Response`, and older helpers return a bare
string/`Blob`. Calling `String()` on the structured result collapsed it to
`"[object Object]"`, which was stored verbatim and then rejected by the diff with
`"[object Object]" is not valid JSON`. A `JSON.parse` write-time guard now
refuses non-JSON config versions, and the diff viewer (`DiffTool`) shows a clear
"corrupted" message if a pre-existing bad snapshot is opened.

`POST /api/admin/storage/test-connectivity` (ADMIN/ROOT) is a read-only
diagnostic that verifies the R2 backup bucket binding is reachable by HEADing the
backup catalog handle — a 404 (no handle yet) still counts as connected. It is
whitelisted for ROOT through the diagnostics gate (`DIAGNOSTICS_PATHS` in
`src/routes/api.ts`) so platform operators can test backup connectivity from the
Diagnostic Dashboard without explicit access to the ADMIN-gated Backups page.

Hard restart (`POST /debug/restart-openclaw`) takes the shared lease, then opens a
`restart` maintenance window and kills the gateway **before** cutting the archive, so
the checkpoint is consistent. It sets `restore-needed` before closing the window, so no
request can relaunch the gateway on pre-restart state in between. If the checkpoint
fails after the stop, the gateway is restarted in place on the untouched filesystem and no restore is armed. It refuses with `409` while a restore is in flight
or a maintenance window is open, and while a restore is pending over the recovery point.
After a successful checkpoint, the next activation restores the committed snapshot.

**Restore outcome.** `sandbox.restoreBackup` is bounded by `RESTORE_BACKUP_TIMEOUT_MS`.
Because `withTimeout` only stops waiting, a timeout is recorded in
`restore-target.json` as `outcome: 'unknown'`, not as a failure. While that record is
younger than `GATEWAY_RESTORE_PHASE_BUDGET_MS`, `getRestoreInFlightReason` blocks every
restore (`RestoreInFlightError`, `409` on the admin route). A definitive failure is
recorded as `outcome: 'failed'` and does not block. The in-lease checks (restore
entry, `restoreIfNeeded`, `restoreBackupById`) re-read this under the lease, and
`restoreIfNeeded` also re-checks the maintenance window after it waits for the lease.

**Destructive guard.** `getPersistenceBusyReason` is the single busy check for freeze
(`/control`), `POST /debug/destroy-container`, and `POST /debug/reprovision`. It is
busy while any maintenance window is open or a restore is in flight. Reprovision also
re-checks after its checkpoint. A snapshot-only lease is **not** busy: an abandoned
snapshot leaves no half-applied filesystem behind, whereas an abandoned restore does.

```mermaid
flowchart TD
  DC["Dispatch Worker cron<br/>backup control action"] --> RC{"runtime active?<br/>isRuntimeActive()"}
  RC -- no --> SKIP["skip (never wake)"]
  RC -- yes --> S["scheduleBackupIfNeeded"]
  S --> G{"credentials + interval + lock"}
  G -- no --> N["no-op"]
  G -- yes --> L{"backup-lock free?<br/>(non-blocking)"}
  L -- busy --> SKIP["scheduler skips this tick"]
  L -- free --> HOLD
  MAN["POST /api/admin/storage/sync"] --> W["acquire lease<br/>wait ≤120s w/ backoff"]
  HR["POST /debug/restart-openclaw"] --> CP["checkpoint first"]
  CP --> W
  W -- busy > 120s --> ERR["error (no snapshot)"]
  W -- acquired --> HOLD["lease held over legacy stage<br/>+ sandbox.createBackup"]
  HOLD --> A["Sandbox snapshot<br/>/home/openclaw"]
  A --> E["optional F4E1 encryption"]
  E --> P["commit backup-handle.json"]
  P --> REL["release lease (CAS tombstone)"]
  CP --> K["graceful kill + restore marker"]
  K --> R["restore committed snapshot<br/>on next activation"]
  RP["POST /debug/reprovision"] --> RCP["checkpoint first<br/>(or require existing backup)"]
  RCP --> RD["destroy instance<br/>409 unless restorable backup or force"]
  SR["POST /api/admin/gateway/restart"] --> SOFT["SIGUSR2 in-process reload<br/>no snapshot"]
```

Do not perform repeated hard restarts without verifying a successful
checkpoint. The checkpoint protects only files inside `/home/openclaw` and only
after the backup catalog has been committed.

#### Durable async backup/restore jobs + stable recovery point

`POST /api/admin/storage/sync?async=1` and
`POST /api/admin/storage/restore/:id?async=1` no longer rely on request
`waitUntil()` (Cloudflare only guarantees ~30s of background time after a
response). Instead the route commits a job record to the per-tenant
`ActivityScheduler` Durable Object (`src/activity/backup-jobs.ts`) and returns
`202 { accepted: true, jobId }`. The DO alarm performs the work and retries with
a bounded lease (`BACKUP_JOB_LEASE_MS`) so a multi-minute snapshot survives
isolate eviction and Worker redeploys. `GET /api/admin/storage/jobs/:id` reads
the persisted status. The `ActivityScheduler` owns one alarm, so the DO merges
the next automation wake and the next backup job and arms the earliest
(`scheduleCombinedAlarm`); a queued job is never dropped when the activity
schedule clears the alarm.

**Every durable job runs one bounded step per alarm invocation.** A Durable
Object alarm handler has a hard 15-minute wall limit, while a snapshot is
`stage (≤10 min) + createBackup + config capture + prune` — and the alarm also
runs the boot, maintenance and activity queues. A single-invocation job could
therefore be killed by the platform, stay `running` until its lease expired, and
restart the whole operation from zero (the orphaned-archive class of incident).
`src/activity/job-steps.ts` persists a `cursor: { step, attempts, startedAt }`
on the job record; each alarm runs at most one step, commits the cursor, and
re-arms one second later so the other queues get a fair turn.

| Job                  | Steps                                                  |
| -------------------- | ------------------------------------------------------ |
| snapshot (manual)    | `repair` → `prepare` → `archive` → `finalize`          |
| snapshot (scheduled) | `repair` → `gate` → `prepare` → `archive` → `finalize` |
| deprovision          | `repair` → `prepare` → `archive` → `finalize` → `stop` |
| restore              | `restore` → `launch`                                   |
| boot                 | `restore` → `launch`                                   |
| maintenance          | `preflight` → `execute`                                |

- `repair` brings the catalog back to a consistent state before any snapshot
  work: it **sweeps** archive objects that no catalog entry can use (the SDK
  writes `backups/<id>/meta.json` only _after_ a successful upload, so an
  upload that lost its metadata write is invisible to both the reconciler and
  the retention prune — it would sit in the bucket forever, billed and
  unrestorable) and it **encrypts** the archives that are not encrypted at rest.
- The lease covers one **step**, not the whole operation (`STEP_LEASE_MS` =
  6 min). A Durable Object alarm has a hard 15-minute wall limit, so the
  previous 14-minute lease meant a step killed at minute 3 stalled the job for
  another 11 minutes before it was retried. Alarm invocations are serialized per
  object, so a healthy step is never duplicated.
- A step recovered after its lease expired had an **unknown** outcome: the job
  says so (`detail: 'step-interrupted'`, or `'step-deadline-exceeded'` past the
  12-minute soft deadline) and emits a `job-step.interrupted` event instead of
  silently replaying. Every step emits a `job-step.completed` /
  `job-step.failed` event with its duration — the measurement the step budget is
  derived from (`max(configured, 2 x the last snapshot duration)` for the
  cadence, 12 min for the per-step deadline).
- `prepare`/`archive`/`finalize` are the phased snapshot
  (`prepareSnapshot`/`archiveSnapshot`/`finalizeSnapshot` in
  `src/persistence.ts`). The catalog entry is still committed in `archive`,
  immediately after encryption, so a crash before `finalize` leaves a restorable
  archive (and `finalize` re-adopts it if the catalog write was lost).
- Each step takes the shared R2 lease on its own (`withSnapshotLease`); the stage
  is idempotent and `archive` re-checks the `restore-needed` latch, so a
  hand-off between steps cannot capture a half-restored container. A **scheduled**
  snapshot refuses to archive behind a pending restore; a manual/deprovision
  checkpoint deliberately still archives (without promoting), which is the
  documented recovery case.
- The retry budget is **per step** (`isStepExhausted`) as well as per job, so a
  permanently broken step fails the job instead of consuming attempts that
  belong to healthy steps. A completed step is never re-run: a boot that is
  interrupted during `launch` resumes at `launch`, not at the destructive
  `restore`.
- The legacy stage runs with `maxAttempts: 1` inside a durable step (the step
  cursor owns the retry), so its worst case stays at the 10-minute
  `LEGACY_STAGE_TIMEOUT_MS` instead of 20 minutes.
- The activity alarm now **enqueues** a scheduled snapshot instead of running
  `scheduleBackupIfNeeded` inline, so a 10-minute stage no longer shares an
  invocation with the automation wake.

**Archives that are not encrypted at rest are visible and repaired.** While a
master key is configured, an archive whose `meta.json` does not carry
`encrypted: true` (a legacy plaintext archive, or one whose upload finished
while the encryption step never ran) is catalogued with `unencrypted: true` and
queued in `pendingEncryption`. It is **never dropped** (it stays restorable) and
**never promoted** to the recovery point; the durable `repair` step encrypts it
in place and clears the flag. The dashboard shows a "not encrypted" badge for
such an archive.

**Scheduled cadence is a floor, not a promise.** `BACKUP_INTERVAL_MINUTES`
(default 5) is the minimum; `resolveEffectiveIntervalMs` stretches the effective
interval to at least **2× the last observed end-to-end snapshot duration**
(recorded as `lastSnapshotDurationMs` in the catalog). Without that, a tenant
whose snapshot takes 10 minutes would start a new one the moment the previous
committed: continuous R2 lease contention, continuous container work, and a
starved automation wake. A fast tenant is unaffected.

`createSnapshot` promotes a **stable recovery point** only after the SDK
archive, optional F4E1 encryption, and the catalog commit have all succeeded
(`promoteStableBackup` → `stable-backup.json`). A cold restore prefers that
pointer and never restores a newer archive whose promotion was interrupted;
retention (`cleanupOldBackups`) always keeps the stable entry, so a
just-uploaded-but-unpromoted archive cannot evict the last committed recovery
point. The pointer is also **monotonic** (an older or replayed archive never
moves it backwards) and **fails closed**: a malformed `stable-backup.json`
aborts restore and retention rather than falling forward to an unpromoted
archive. Unit-tested in `src/persistence.test.ts` (the “stable recovery point
promotion” block). The latest validated config is also mirrored to a fixed-key
hot copy (`stable/config.json`), kept out of the version-history listing and
30-day prune.

A **targeted restore** (`POST /api/admin/storage/restore/:id`) force-promotes the
restored archive (the exception to the forward-only pointer) so the next cold
boot restores the same archive. `?setCurrent=0` (sync or `?async=1`) threads
`{ setRecoveryPoint: false }` through `restoreBackupById` (and the durable
restore job) to apply an archive **without** moving the recovery point — the
next fresh container then reverts to the unchanged pointer. A restore also opens
a cross-isolate `maintenance/replacement.json` window (operation `restore`) and
stops the gateway right after taking the shared restore lease, closing the
window in a `finally`; `restoreIfNeeded` and `ensureGateway` both refuse to run
under it, so a refetch cannot restart the gateway and remount the latest
recovery point mid-restore.

This makes the async workflow durable and the recovery point monotonic. It does
**not** turn interval snapshots into zero-RPO replication: writes acknowledged
after the last committed snapshot are outside the guarantee. Closing that gap
requires a write-through storage path and is tracked in issue #377.

### Sandbox → R2 Connectivity (`outboundByHost`)

Sandbox code can reach Worker bindings through a **virtual hostname** without
any SDK/client library or credentials inside the container. `@cloudflare/sandbox`
(via `@cloudflare/containers`) exposes the canonical per-host outbound handler:

```ts
// src/index.ts
Sandbox.outboundByHost = {
  [RELEASES_R2_HOST]: (request, env, ctx) => handleReleasesR2Request(request, env),
};
// catch-all: internet passthrough
Sandbox.outbound = async (request) => fetch(request);
```

The handler runs **in the Workers runtime** with full `env` bindings, so the
sandbox never sees R2 credentials. Unmatched hosts fall through to the catch-all
`Sandbox.outbound`.

**Release bridge (`releases.r2.internal`).** `src/sandbox-r2-bridge.ts` maps
plain HTTP from the sandbox onto the tenant's `BACKUP_BUCKET` under a fixed
`releases/` prefix (the bucket is per-tenant, so the prefix is already
tenant-isolated):

| Method   | Path     | Effect                          |
| -------- | -------- | ------------------------------- |
| `GET`    | `/`      | JSON list of release objects    |
| `GET`    | `/<key>` | Stream an object (200) or 404   |
| `HEAD`   | `/<key>` | Size/etag metadata (200) or 404 |
| `PUT`    | `/<key>` | Upload an object (201)          |
| `DELETE` | `/<key>` | Remove an object (204) or 404   |

Keys are sanitized against traversal before touching R2; uploads use a
`FixedLengthStream` (bounded memory) when `content-length` is advertised.

**Docs bridge (`docs.r2.internal`).** `src/sandbox-docs-bridge.ts` exposes the
tenant's client documentation (`customers/<slug>/docs/`) for on-demand fetch by
the Fouria Builder agent / sandbox tooling, reusing `R2ClientDocsStore`
sanitization. The tenant slug is resolved via the shared `resolveTenantSlug`
(`src/sandbox-tenant.ts`): an explicit `customer-<slug>` container id wins,
otherwise it is derived from the per-tenant Worker env (`CUSTOMER_SLUG`, else
`WORKER_NAME` / the always-provisioned `WORKER_URL`, with the `tenant-` /
`staging-` prefixes and `-dev` suffix stripped), since production provisions the
sandbox under the fixed id `openclaw`. `GET /` lists,
`GET/HEAD /<name>` reads, `PUT /<name>` writes, `DELETE /<name>` removes. This
lets the sandbox read docs without R2 credentials and without waiting for
Worker-side materialization.

**Leads bridge (`leads.r2.internal`).** `src/sandbox-leads-bridge.ts` exposes the
**global** `paso4-leads` proposal bucket (the `fouria-leads` skill) over
`LEADS_BUCKET`. The bucket folder layer **is the tenant slug**, and the bridge
resolves the caller's slug via the shared `resolveTenantSlug` in
`src/sandbox-tenant.ts` (explicit `customer-<slug>` container id, else the
per-tenant Worker env `CUSTOMER_SLUG` / `WORKER_NAME` / `WORKER_URL`). Every path
the agent sends is **relative to its own `<slug>/` root** — the handler prepends
the slug,
so a sandboxed agent cannot list, read, or write another client's proposal. The
only bucket-root object a tenant may read is the shared, proposal-agnostic
`build-pdf.sh` (`SHARED_LEADS_READ_KEYS`); writes/deletes of it resolve under the
tenant's own folder. Routes: `GET /` (or `GET /?prefix=<p>`) lists **relative**
keys, `GET/HEAD /<key>` reads, `PUT /<key>` writes, `DELETE /<key>` removes. Keys
reuse the shared `sanitizeLeadsKey` / `sanitizeLeadsPrefix` rules from
`leads/operations.ts`. This is how the sandbox reaches proposal docs **without a
public route** — the legacy `wss://<worker>/api/leads/ws` proxy
(`src/routes/leads.ts`) is intercepted by Cloudflare Access on tenant hostnames
(a per-tenant Access app shadows the `*.fouria.io/<path>` apps) and is now used
only by the host-side E2E fixture (`test/e2e/o_leads_proxy.txt`, which
authenticates with the `CDP_SECRET` persisted by `server/start`).

> **Binding requirement.** `LEADS_BUCKET` must be provisioned as an
> `r2_bucket` binding (lerma `instances.ex` `instance_bindings/2`, config
> `LEADS_R2_BUCKET`, default `paso4-leads`). A plain-text var holding the bucket
> name makes `env.LEADS_BUCKET` a `string`, so `bucket.list()` throws inside the
> `outboundByHost` handler and the container sees an opaque `ECONNRESET`. All
> three bridges validate the binding with `isR2Bucket` (`src/r2-binding.ts`) and
> wrap their handler body in a try/catch, returning a structured JSON `503`/`500`
> instead of resetting the socket.

**R2 S3 credentials are deprecated for the container.** `buildEnvVars` no longer
forwards `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` into the sandbox: the
bridges run in the Workers runtime with the `BACKUP_BUCKET` / `LEADS_BUCKET`
bindings, so the container never sees credentials. The S3 credential remains a
**Worker binding** for the Sandbox SDK presigned-URL backup path only.

> **SDK 1.0 (preview).** The canonical `static outboundByHost` API is unchanged
> in the 1.0 preview, so these bridges run as-is on
> `@cloudflare/sandbox@0.13.0-next`; see the Sandbox SDK 1.0 section below.

### Client Docs (dev-only admin surface)

Client documentation is managed from the Admin **Client Docs** page
(`/_admin/docs`, ADMIN-only, dev instances only) backed by
`src/routes/api/fouria-docs.ts` (dev-gated by `isDevAgentsEnabled`, so it 404s in
production). It manages two tenant-scoped namespaces **directly from the Worker**
— no sandbox/CLI round-trip:

- **Documentation** — `customers/<slug>/docs/` in `BACKUP_BUCKET` (markdown,
  text, and binary files like PDFs/images; content type from R2 metadata or the
  extension). Store: `R2ClientDocsStore` (`listDocs` / `getDoc` / `getDocBinary` /
  `putDoc` / `putDocBinary`).
- **Proposals (leads)** — `<slug>/` in the global `LEADS_BUCKET` (PDFs and
  assets), scoped to the caller's tenant.

The SPA renders markdown client-side (`marked` + `DOMPurify`, sanitized) and opens
PDFs with the browser's native viewer through the `/raw` endpoints. Raw responses
set `X-Content-Type-Options: nosniff`; HTML/SVG are forced to `attachment` with a
sandbox CSP so client-authored files can never execute in the admin origin.

**Versioning (`DocsGitStore`).** Every doc write is committed to a Worker-side,
git-like object store in `BACKUP_BUCKET`:

```
customers/<slug>/docs-git/objects/<blobSha>    content-addressed file bytes
customers/<slug>/docs-git/commits/<commitSha>  commit JSON ({ tree, parents, … })
customers/<slug>/docs-git/refs/heads/dev       mutable HEAD
customers/<slug>/docs-git/refs/promoted        promoted ref
```

The working namespace always mirrors HEAD. History is exposed in the dashboard's
commit-tree shape (`GET /commits`); `POST /rollback/:sha` restores an older commit
as a **new, non-destructive revert commit**.

**Backups (`DocsBackupStore`).** `POST /backup` snapshots the working namespace to
`customers/<slug>/docs-backups/<id>/` (files + `manifest.json`), independent of the
git store; `GET /backups` lists (retention keeps the newest 10), and
`POST /backups/:id/restore` re-applies a snapshot as a new commit.

### Sandbox SDK 1.0 side-by-side foundation (`SandboxV1`)

Alongside the default legacy `Sandbox` path, the Worker carries an **additive,
not-routed** Sandbox SDK 1.0 foundation so the side-by-side move can be built
without a big-bang cutover. It is a foundation only — **no tenant is on it and
it carries no durability claim.**

- **Class** — `SandboxV1` (`src/sandbox-v1.ts`), a user-owned `DurableObject`
  that drives `this.ctx.container` (the 1.0 model). Exported from `src/index.ts`
  together with `DirectoryBackupGateway` (needed for the DO to pass
  `ctx.exports.DirectoryBackupGateway` to `DirectoryBackup`).
- **Registration** — `wrangler.jsonc` migration `v7`
  (`new_sqlite_classes: ["SandboxV1"]`), a `SandboxV1` DO binding, and a separate
  `scheduling_policy: "durable_object"` container application per environment
  (`fouria-sandbox-v1`, `fouria-sandbox-v1-staging`) that builds
  `images.sandbox` from `apps/fouria/Dockerfile.v1`.
- **Default routing unchanged** — the legacy `Sandbox` class + `getSandbox()`
  still serve every tenant. No route selects `SandboxV1`; there is no customer
  routing change and no cutover.
- **Package / image pairing** — root catalog installs
  `@cloudflare/sandbox@1.0.0` (this DO) and
  `@cloudflare/sandbox-legacy@0.13.0-next.776.1` (the still-default legacy class,
  imported via the `-legacy` alias). `Dockerfile.v1` uses `node:24-trixie-slim`
  and COPYs the `sandbox-shim` helper from the matching `cloudflare/sandbox:1.0.0`
  image (the shim tag must track the package version); the legacy base image no
  longer applies because 1.0 removes the in-image SDK server.
- **TTL (derived, capped at 6h)** — `resolveInactivityPolicy()` reuses
  `resolveSleepAfter()`: non-billable instances get a finite idle timeout
  (staging 15m, dev 2h) clamped to the 6h platform maximum accepted by
  `setInactivityTimeout()`. Billable production gets **no inactivity timeout**
  (`timeoutMs: null`): the container stays awake until an explicit lerma freeze.
  There is no `SANDBOX_SLEEP_AFTER` binding.
- **Real `DirectoryBackup` with F4E1 at-rest wrapping** — the additive DO
  instantiates the real `DirectoryBackup` (from `@cloudflare/sandbox@1.x`)
  against `ctx.container` + `ctx.exports.DirectoryBackupGateway` with a
  tenant-scoped `sandbox-v1/<encoded-do-name>/` R2 prefix in `BACKUP_BUCKET`.
  The SDK writes one flat plaintext object per record
  (`<prefix><id>.tar.zst`); the Worker then post-encrypts it in place to
  `<prefix><id>.tar.zst.enc` (F4E1/AES-GCM) plus a readable
  `<prefix><id>.meta.json` sidecar, and deletes the plaintext canonical object.
  Restore transiently decrypts `.enc` back to the canonical key for the SDK,
  then always deletes that plaintext in a `finally` block; delete removes all
  three objects (strict for explicit user deletion, best-effort on cleanup).
  Plaintext therefore exists only in the short post-upload window and the
  restore/decrypt window. The master key stays Worker-only
  (`env.BACKUP_ENCRYPTION_KEY`, never passed to the container); the record's
  `sha256`/`size` remain plaintext and the SDK `DirectoryBackup.restore()`
  verifies them on restore. With `BACKUP_ENCRYPTION_REQUIRED=true` a missing
  key/meta fails closed before any upload or restore.
- **Still no scheduling, conversion, or cutover** — Worker-owned periodic
  snapshot scheduling + retention is **not implemented** (no approved defaults
  wired; the intended targets are a 5m snapshot interval, retaining the latest
  interval snapshots plus a 24h rolling window and daily 30d history, once a
  scheduler exists). There is **no** 0.13 `.sqsh.enc` → DirectoryBackup
  conversion, **no** customer cutover, and **no** default-routing change: the
  legacy `Sandbox` class + `getSandbox()` still serve every tenant, no route
  selects `SandboxV1`, and the complete restore-before-Gateway path is not wired
  for the v1 line. This is not a final durability/no-loss guarantee. See
  `docs/plans/sandbox-v1-scaffold.md`.

### Sandbox SDK runtime lines

The default tenant runtime is the **legacy** `Sandbox` class, installed via the
`@cloudflare/sandbox-legacy` catalog alias of `npm:@cloudflare/sandbox`
`0.13.0-next.776.1`, paired with the `docker.io/cloudflare/sandbox:0.13.0-next.776.1`
image. The Worker package and the container image **must stay on the same line** —
the stable and 1.0 control protocols are mutually incompatible.

The side-by-side `SandboxV1` Durable Object uses the stable `@cloudflare/sandbox`
`1.0.0` package (see the Sandbox SDK 1.0 section above). The intended runtime is
selected by the `SANDBOX_RUNTIME` binding (`legacy` default | `v1`); until V1
implements the full `Sandbox` surface the legacy class keeps serving and
`resolveSandboxRuntime()` reports the requested V1 as `pending` (see
`src/sandbox-runtime.ts`).

Every tenant-facing site that needs a `Sandbox` obtains it through the single
runtime acquisition boundary `acquireSandbox()` in `src/sandbox-contract.ts` —
the request middleware (`src/index.ts`), the task buffer (`src/tasks/index.ts`),
the `ActivityScheduler` Durable Object (`src/activity/scheduler.ts`), and the
per-customer gateway launcher (`src/gateway/process.ts`). The boundary resolves
the runtime but always returns the legacy stub for now (a `v1` request is still
`pending`), preserves the container identity (`openclaw` for the single-tenant
instance, `customer-<slug>` for a per-customer gateway) and the legacy idle
policy (defaults to `buildSandboxOptions(env)`), and never starts a container —
`getSandbox()` returns a Durable Object stub, so acquiring a runtime never wakes
a sleeping tenant or touches the v1 DO namespace. This is the single flip point
for the cutover.

Key differences from the 0.12 stable line, all funneled through
`src/gateway/sandbox-exec.ts`:

- **No `startProcess` / `execStream` / `getLogs()`.** `sandbox.exec(argv)`
  starts a process and resolves once it is **launched**, not once it finishes;
  it never implies a shell. `startShell(sandbox, script)` wraps a shell command
  line in `['/bin/bash', '-lc', script]` and returns the handle;
  `runShell(sandbox, script)` waits and buffers `{stdout, stderr, exitCode}`;
  `outputOf(proc)` / `collectLogs(proc)` read an existing handle.
- **`listProcesses()` returns `ProcessStatus[]`** (`.state`, argv `.command`,
  `.startedAt`) with no methods. `findExistingGatewayProcess` fetches a live
  handle via `getProcess(id)` and returns a `GatewayProcess` wrapper
  (`.state`, `.command` string, `.refresh()`, `.waitForPort()`, `.kill()`,
  `.getLogs()`).
- **`waitForProcess`** delegates to `waitForExit({ timeout })` and kills the
  process on a local wait timeout (the SDK timeout cancels the wait only).
- **Busy-DO hardening for gateway reads.** A contended container-control channel
  can hold an `exec` launch for the SDK default (120s) and stall the Sandbox DO.
  Read routes therefore bound their sandbox work and degrade instead of hanging
  past the client timeout:
  - `ensureGateway(waitForReady:false)` (every API read route) is bounded by
    `ENSURE_GATEWAY_READ_TIMEOUT_MS` (30s) and returns `null` on expiry; the
    caller serves its normal "not ready" payload. Cold-start (`waitForReady:true`)
    is exempt.
  - A short-TTL positive readiness result (`GATEWAY_READINESS_CACHE_TTL_MS`, 5s)
    collapses concurrent read-path probes onto one; it is cleared on
    kill/restart/recovery.
  - The gateway port probe (`isGatewayPortOpen`) is bounded by
    `PORT_PROBE_TIMEOUT_MS` (10s) with a container-side `curl -m`, and marks the
    expected closed-port exit (`curl` 7 = connection refused) with
    `reportNonZeroExit: false` (operation `gateway.port-probe`) so the normal
    "gateway down, start it" outcome is not logged as a `sandbox.process.failed`
    error on every cold start — matching the `gateway.health-probe` behavior.
  - Expected CLI non-zero exits are not logged as failures either: `runShell`
    (via `reportNonZeroExit`) and `waitForProcess` (same option) let a call site
    mark an exit it handles as normal. The knowledge `wiki status` probe uses it
    (`operation: 'knowledge.wiki-status'`) because an uninitialized vault exits
    non-zero and the handler reports `initialized:false`. Timeouts always log.
  - Route flows that poll the container wrap their whole flow in a budget
    (`/api/admin/devices` 25s, `/debug/container-config` 20s).
- **No transport selection.** `SANDBOX_TRANSPORT` was removed; the preview uses
  RPC automatically.
- **`SandboxOptions.labels` was removed** upstream (worker identity is no longer
  attached at container start).

**Deploy cutover:** because the protocols are incompatible, a production deploy
must roll containers in one step — `wrangler deploy --containers-rollout=immediate`
— with `rollout_active_grace_period` at `0`. In-flight processes/terminals stop
and pre-deploy process IDs become invalid.

### Container Image Build Profiles (dev vs production)

The Dockerfile takes a `BUILD_PROFILE` build arg (default `production`):

- **production** (default) installs only `xz-utils` — the slim image
  (~1.36 GB) used by production and by E2E.
- **dev** additionally installs the optional PDF/document tooling
  (`pandoc` + `weasyprint` — a large Python/cairo/pango stack, ~760 MB) that the
  dev-only Fouria Builder `fouria-leads` skill needs
  (`apps/fouria/skills/agent-only/fouria-leads/SKILL.md`; the Builder is injected
  only when `OPENCLAW_DEV_MODE=true`).

The profile is selected per wrangler environment via `image_vars` (Cloudflare's
build-time variables, equivalent to `docker build --build-arg`):

| Environment                            | `BUILD_PROFILE`        | Image                  |
| -------------------------------------- | ---------------------- | ---------------------- |
| `env.staging` (dev/test)               | `dev`                  | with pandoc/weasyprint |
| `env.production`                       | default (`production`) | slim                   |
| default / local Vite dev (local + E2E) | default (`production`) | slim                   |

There is a **single image definition** (`apps/fouria/Dockerfile`); CI
(`test.yml`) builds it with no build args when it pushes `unstable-<sha>`, and
the release CD (`deploy-fouria.yml`) promotes that same image by retagging it, so
released images are the slim production profile. The E2E `basic` (PR) and
`production` (develop) profiles both provision the slim production image (only
instance type/env differ). To build a dev image locally:

```bash
docker build --build-arg BUILD_PROFILE=dev -f apps/fouria/Dockerfile .
```

### Sandbox Recovery (stale + orphaned)

When the platform terminates a container behind a live Sandbox DO (idle
recycle, maintenance, SIGTERM), every RPC into it fails with the SDK's
`HTTP error! status: 500` wrapper — and retrying against the same instance
fails forever, bricking the tenant. `gateway/sandbox-recovery.ts` classifies
these dead-container errors (`isStaleContainerError`) and recovers
(`recoverStaleSandbox`): destroy the instance, reset the startup-probe and
persistence caches. The next RPC auto-provisions a fresh container and the
normal cold-start flow runs (R2 restore → `start-openclaw.sh` → gateway).

A container destroyed **out-of-band** (Cloudflare Containers dashboard "Destroy",
failed operator force-stop) is a second failure class: the DO is alive but points
at a placement that no longer exists, so RPCs fail with the SDK's _retryable_
admission errors (`no container instance available`, `maximum number of running
container instances exceeded`) instead of the 500 signature. Those are legitimate
during a cold start, so they are not treated as stale on their own.
`isContainerUnavailableError` + `trackProvisioningFailure` escalate them to a
recovery once the streak reaches `ORPHANED_INSTANCE_MIN_FAILURES` **and** spans
`ORPHANED_INSTANCE_GRACE_MS` (120s — longer than the SDK's ≈110s provisioning
budget, so a slow-but-healthy cold start is never cancelled mid-provision). The
streak is cleared whenever a sandbox RPC succeeds. `maybeRecoverSandbox` unifies
both classes for callers.

Recovery is wired into three failure points in `gateway/process.ts`:
`findExistingGatewayProcess` (listProcesses failures), `checkGatewayHealth`
(exec failures), and the `ensureGateway` `startShell` path (recover + retry
once). It is cooldown-guarded (`STALE_RECOVERY_COOLDOWN_MS`, 60s) and never
fires for deploy-window `OperationInterruptedError`. Permanent image errors still
require the rescue tool (`rescueContainer`, see OPERATOR-GUIDE §5.1). Logs use the
`[SandboxRecovery]` prefix.

#### Transient transport drops (retry, don't reprovision)

A third failure signature is not a dead container: `Network connection lost.`
(and similar socket/WebSocket transport messages). It appears when the DO↔container
connection drops _while the container is alive_ — most visibly while the Gateway is
applying a config/plugin reload (e.g. adding a model provider in the Control UI),
which restarts the Gateway and briefly tears down the container RPC/WS transports.

`isContainerTransportError` (`gateway/sandbox-recovery.ts`) classifies these, and
`isStaleContainerError` explicitly **excludes** them (even when the SDK wraps the
message in its `sandbox.<op> error` prefix) so a transient drop never destroys a
live instance. `ensureGateway` instead retries with a short backoff: the port
probe (`isGatewayPortOpenResilient`, up to `TRANSPORT_RETRY_ATTEMPTS`) and the
`startShell` spawn (once). Before this, a reload that dropped the transport mid
probe fell straight through to `start-openclaw.sh` → `Failed to start process` →
a 502 to the user, even though the Gateway came back seconds later. The retry
keeps the request alive across the reload window.

A Durable Object **code update** (a normal Worker/DO redeploy, or a Lerma
redeploy of the tenant script) resets the control session between the DO and its
container even though the container itself is not replaced. The cold start then
loses the transport on the **supervised port wait**: the atomic
`start-openclaw.sh start` process (config setup + gateway under one lock) keeps
running, but `waitForPort()` on the stale handle rejected and previously aborted
the cold start — killing a gateway that would have completed (observed as
`Network connection lost.`). The atomic `start` path has **no separate
setup-process wait**, so only the gateway port wait reattaches via
`getProcess(id)` and resumes (`waitForPortResilient`,
`gateway/sandbox-exec.ts`), bounded by the shared `STARTUP_TIMEOUT_MS` deadline.
The generic `waitForExitResilient` helper remains in `sandbox-exec.ts` for the
legacy/boot-smoke supervised waits. Only a non-transport error, an
un-reattachable process, or an exhausted deadline still fails the cold start.

The `openclaw doctor` diagnostics run through the same transport-resilient path:
`runShellResilient`, which retries the `exec` launch (`startShellResilient`) and
reattaches the wait (`waitForExitResilient`). Previously a transient
`The operation was aborted` drop on the doctor `startShell` surfaced as a hard
`500 { ok: false, error: "The operation was aborted" }`; it now reattaches to the
process and returns the structured diagnostic instead. `GET /api/doctor` still
runs inline this way; `POST /api/admin/config/fix` and `POST /debug/sqlite-integrity`
now **enqueue a durable maintenance job** on the ActivityScheduler DO
(`activity/maintenance-jobs.ts` + `activity/maintenance-executor.ts`) and return
`202 { accepted, jobId }`, because a large integrity check or doctor repair can
exceed a Worker request's lifetime (the old 30s/90s shell timeouts surfaced as
"timeout in under 2 minutes"). The DO alarm runs the command under a 14-minute
lease with bounded retries (`runMaintenanceOperation`, still `runShellResilient`
internally) and stores the bounded result on the job record, polled via
`GET /debug/sqlite-integrity/jobs/:id` / `GET /api/admin/config/fix/jobs/:id`.
Both endpoints fall back to an inline run when the `ACTIVITY` binding is absent
(local dev / tests). The doctor repair runs `openclaw doctor --fix --non-interactive`
(the supported command for index-only `integrity_check` corruption — missing or
wrong-count index entries — which verifies the table data, preserves a backup,
and `REINDEX`es) and **quiesces the gateway first**: `runMaintenanceOperation`
opens the cross-isolate maintenance window (`beginMaintenance(..., 'doctor-repair')`),
kills the gateway so the repair can take exclusive state ownership, then closes
the window so the next request re-launches the gateway.

**Restore → repair ordering (`restore-config-pending.json`).** A restore replaces
the container's writable layer but launches nothing, so a config repair applied
around a restore is silently overwritten by the next re-apply of the recovery
point (observed on `tenant-tarjetero-dev`: a config fix was lost on the next
gateway restore). Every successful restore (`restoreIfNeeded` and
`restoreBackupById`) now writes a durable `restore-config-pending.json` marker.
The durable boot job **and** the durable restore job launch the gateway on the
restored state through the shared `launchRestoredState` helper
(`src/post-restore.ts`); the admin sync/async-fallback restore route ends there
too, so every restore path has the same ordering guarantee. The helper calls
`ensureGateway({ waitForReady: true })` and then
`captureRestoredConfigVersion` — writing `configs/<ts>.json` and
`stable/config.json`, so the restored config becomes the **newest and stable**
config version and the dashboard's diff/history reflects what is actually live.
The marker is cleared only after that capture. A gateway-launch failure never
re-runs the destructive restore: restore jobs stay `succeeded` with
`detail: "gateway-launch-pending"` and the sync route returns
`gatewayReady: false`; the pending marker drives the next boot / maintenance
tick. A `doctor-repair` job checks,
before running: an active restore job, an open maintenance window / in-flight
restore (`getPersistenceBusyReason`), or a pending restored-config marker. In any
of those cases it **defers** (`MaintenanceJobDeferredError`): the job keeps its
attempts and queue position (60s re-check) instead of failing its three attempts
while the restore runs, and executes only after the gateway is up on the restored
state.

Job status is pushed over a **WebSocket** — `GET /api/admin/config/fix/jobs/:id/ws`
(and the `/debug/sqlite-integrity/jobs/:id/ws` variant) — instead of the
dashboard polling the JSON endpoints above every 2s (one request + Worker log
line per tick for the whole multi-minute repair). The streamer
(`routes/api/maintenance-ws.ts`) reads the durable record from the
`ActivityScheduler` DO, sends a frame only when it changes, and closes `1000`
once the job is terminal; the dashboard's `subscribeMaintenanceJob` falls back to
a slow HTTP poll only while the socket is down. While a job is active the
streamer also calls the DO's `rearm()` **self-heal**, so a job whose alarm was
lost is scheduled again immediately.

The shared alarm is written through an **optimistic-concurrency guard**
(`armWithGenerationGuard`): the `ActivityScheduler` owns one alarm, so three job
enqueues, the warm-path schedule refresh, and the alarm handler's own trailing
re-arm can interleave at `await` points. A stale arm that read "no pending job"
could execute `setAlarm(later)`/`deleteAlarm()` _after_ a just-enqueued job's
`setAlarm(now)` and strand the job until the later time — observed as a
`doctor-repair` job enqueued at 12:57:30 that only began at 13:02:11 (exactly the
previous boot job's 14-minute lease). The guard bumps a stored generation and
abandons a superseded arm's write, so the newest arm (fresh read) owns the
alarm.

While the control session reconnects, the loading page's `GET /api/startup/ws`
stream treats a transport-drop snapshot as transient: it re-sends the last known
state (or a synthetic `starting` view) and backs off exponentially
(`STARTUP_WS_POLL_INTERVAL_MS` → `STARTUP_WS_MAX_BACKOFF_MS`, 30s) instead of
re-probing — and re-logging the same error — every 5s. A successful probe resets
the cadence.

The manual lever is `POST /debug/reprovision` (DEBUG_ROUTES, ROOT): it
checkpoints the live container first and **fails closed** — unless it just
committed a snapshot or a restorable backup already exists it refuses with
`409` rather than silently destroying tenant state; `{"force":true}` is the
explicit override for an already-unrecoverable container. The checkpoint is only
attempted when `isRuntimeActive()` reports a live runtime (so an idle container
is never woken) and is time-bounded (`REPROVISION_CHECKPOINT_TIMEOUT_MS`); the
restorability check HEADs the newest catalog entries' `meta.json` rather than
trusting the entry count. On success it destroys the instance, clears the
startup/persistence caches, and resets the orphan streak without ever contacting
the gateway — so it works when the container is entirely gone. Compare
`POST /debug/restart-openclaw` (gateway-aware, checkpointed) and
`POST /debug/destroy-container` (teardown only, used by E2E).

### Activity-Driven Wake (`ActivityScheduler`)

Automations ("jobs") run **inside the Gateway**, so the Gateway must be running
for a schedule to fire. Historically a Worker cron trigger woke every tenant on
every tick (expensive and since removed). `src/activity/` replaces it with a
per-tenant activity clock:

- **`ActivityScheduler` Durable Object** (SQLite-backed, binding `ACTIVITY`) —
  stores a mirror of the tenant's automation schedule and owns a single alarm.
  Deliberately a **separate** DO from the SDK `Sandbox` DO, which already uses its
  own alarm for `sleepAfter`.
- **Mirror** — warm paths (`GET /api/health`, only when `readiness.ok`) read
  `openclaw automations list --json` and push the earliest `nextRunAtMs` into the
  DO. The DO throttles this to once per 5 min, and it never wakes a sleeping
  container.
- **Alarm** (`activity/alarm.ts`) — fires `CRON_WAKE_AHEAD_MINUTES` (default 10)
  before the next job, wakes + heals via `ensureGateway` (which recovers an
  orphaned instance — see above), then re-reads the schedule and re-arms. No
  enabled jobs → alarm cleared → the container sleeps. A due-but-pending job is
  woken once per occurrence and re-checked on a bounded retry, so it can never
  spin.

Net: healing/wakeup is independent of user traffic, gated on real job activity,
and free when idle. Traffic may cold-start the container; jobs get a pre-warmed,
healed one. Requires the `ACTIVITY` binding in `wrangler.jsonc` and in the lerma
provisioner (`tenant_provisioner.ex` `resource_bindings`) + the
`new_sqlite_classes` migration (`dispatch_namespace.ex` `@do_sqlite_classes`).

### Change-Aware Backups (feature-flagged)

Scheduled snapshots can be gated on whether tenant state changed. Two independent flags, both **off by default**:

| Flag                                                | Effect                                                                                                                                                                                                                                                                                |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BACKUP_CHANGE_DETECTION=true`                      | **Phase 1 (cost).** The scheduled snapshot path (`scheduleBackupIfNeeded`) runs a container-side scan first. Unchanged state ⇒ no squashfs, no legacy staging, no lease.                                                                                                              |
| `BACKUP_CHANGE_WATCH=true` + `BACKUP_WATCH` binding | **Phase 2 (RPO).** The per-tenant `BackupWatch` DO checks every `BACKUP_CHANGE_CHECK_MINUTES` (2) and requests a durable snapshot from `ActivityScheduler` once a change has been quiet for `BACKUP_CHANGE_DEBOUNCE_MS` (60 s) and the floor (`BACKUP_INTERVAL_MINUTES`) has elapsed. |

**Scan (`src/change-detection.ts`).** Runs `find` over the real state roots (`/home/openclaw`, `/root/openclaw/.openclaw`, `/root/clawd`; the home directory is symlink-only for state, and `find` does not follow symlinks). It prunes the transient legacy stage and `.npm`. It compares against the baseline marker `/tmp/.fouria-last-backup` and prints `unchanged`, `changed`, or `active` (changed within the debounce window). Only entries _inside_ the roots count (`-mindepth 1`), so a root's own mtime, which moves when the legacy stage is created, cannot read as a change. Known limitation: a pure deletion with no accompanying write is not detected until the next write in the same tree, because `find -newer` sees mtimes, not removals. Every ambiguous outcome is `changed`: a missing baseline, no roots, a non-zero exit, a timeout, or a thrown error.

**Baseline.** `/tmp/.fouria-pending-backup` is stamped before a snapshot captures state and promoted to the baseline only after the catalog commits. A failed snapshot leaves the baseline untouched. This runs on every snapshot, whether or not a flag is set, so enabling a flag later has a valid baseline.

**Watch (`src/backup-watch/`).** `watch.ts` is the pure state machine. `durable-object.ts` is the adapter, and `arm.ts` holds gating and policy with no DO imports, so warm paths can call it without an import cycle.

- Never wakes a container: liveness is the non-waking `isRuntimeActive()`. A not-running tenant is disarmed and not scanned.
- **Keepalive guard.** Every scan is an `exec` that renews the container's sleep timer. A tenant unchanged for `BACKUP_CHANGE_IDLE_DISARM_MS` (10 min, below the staging 15 min sleep window) is disarmed, and re-arming never resets the idle clock. The watch can therefore delay a quiet container's sleep by at most that window.
- **No starvation.** An `active` tenant defers snapshots for at most `BACKUP_CHANGE_MAX_DEFER_MS` (10 min), so constant writes still get snapshotted.
- **No hot-path work.** Snapshots are never taken from the watch directly. It enqueues the existing `ActivityScheduler` durable `snapshot` job, which keeps that job's gateway start, lease, retention, and restore-pending guards.
- Arming happens on a running tenant only: the dispatch `backup` control action (`/control`) and the warm `refreshActivitySchedule` path (gateway healthy).

**Wiring.** `wrangler.jsonc` declares the `BackupWatch` class (migration `v8`) and the `BACKUP_WATCH` binding in every environment. In production the tenant's bindings come from lerma, which binds `BACKUP_WATCH` only when the `backup_change_watch` provisioning flag is on (see `apps/lerma`). Both flags are plain-text vars that lerma always sets to `true` or `false`, so turning a flag off explicitly reverts it on redeploy.

### Backup before deprovisioning

A container is never deprovisioned without a committed snapshot. The durable
backup-job queue (`src/activity/backup-jobs.ts`) gains a `deprovision` operation
(`mode: 'sleep' | 'freeze'`); the `ActivityScheduler` alarm runs it as
`snapshot → stopIfIdle()` (sleep) or `snapshot → destroy()` (freeze), and a
failed snapshot retries rather than tearing the container down (fail closed).

**Reuse a snapshot from inside the idle window.** `hasRecentSnapshot()` (a pure
catalog read in `src/persistence.ts`) answers whether the newest committed
archive is newer than the instance's idle timeout (`resolveSleepAfterMs()`:
staging 15m, dev 2h, production 6h) plus `IDLE_SNAPSHOT_REUSE_SLACK_MS` (5m, so
the archive finalized just before the previous stop attempt still counts).
`onActivityExpired()` uses it to stop
inline instead of enqueueing a snapshot when the container has already been idle
for a whole window — the filesystem cannot have changed, so a second archive is
pure cost. A read failure is treated as "no recent snapshot" (fail closed). This
also breaks the loop where the deprovision snapshot's own RPCs renewed the
activity window, `stopIfIdle()` refused, and the next expiry snapshotted again
(observed on `tenant-tarjetero-dev`: three 422 MB archives per idle expiry while
the container never stopped).

**Deprovision archives stamp the schedule.** `finalizeSnapshot` receives
`stampScheduled: true` for `deprovision` jobs as well as `trigger: 'scheduled'`
ones, so a committed deprovision archive resets `lastScheduledBackup` and the
next cron tick does not archive the same state again.

**Stop only on inactivity (stop-time re-check).** The snapshot can take minutes
on a large tenant, so the container may be used again between enqueue and stop.
The sleep branch therefore does **not** call `sandbox.stop()` directly: it calls
`TenantSandbox.stopIfIdle()`, an RPC on the Sandbox DO (the activity authority).
That re-reads the DO's own idle window — renewed by every request, including a
proxy request for an in-flight chat — and the container process list, and stops
only when the window has genuinely elapsed and no transient work is running. If
the tenant became active during the snapshot the job returns `skipped`
(`detail: active:<reason>`) and leaves the container running; the next idle
expiry enqueues a new job. Refusals: `activity` (window renewed),
`transient-activity`, `probe-failed` (unknown is **not** idle),
`billable-production` (never auto-sleeps), `not-running`.

- **Idle sleep.** `TenantSandbox.onActivityExpired()` stops inline when a recent
  snapshot covers the idle window, and otherwise enqueues a `deprovision`
  (sleep) job, so the Sandbox DO leaving memory never loses an in-flight
  multi-minute snapshot. With no `ACTIVITY` binding it falls back to the
  previous stop-now path.
- **Freeze.** `POST /control` freeze enqueues a `deprovision` (freeze) job and
  acknowledges immediately (`202 { ok, accepted: true, frozen: false, jobId }`),
  failing closed at enqueue (`409` when a durable job is already active). The
  caller does not hold a multi-minute request open: it polls the tenant's
  **`GET /api/health/backup/:jobId`** until the job reaches a terminal status.
  A snapshot failure surfaces as a `failed` job and the container stays running,
  so lerma never marks a still-running instance frozen.
- **Platform stops** (an inactivity timeout firing after the DO left memory, a
  container crash, or main-process exit) have no Worker hook; the periodic +
  change-watch snapshots bound that residual RPO.

The pollable endpoint (`GET /api/health/backup/:jobId`) carries no in-Worker
authentication — the same posture as `/api/health` — and is reachable by the
lerma control plane's health-monitor service token through the **health Access
tunnel** (the `*.fouria.io/api/health` path application in
`apps/fouria/iac/access.tf`). Cloudflare Access path inheritance covers the
`/backup/*` sub-path, and it is also listed explicitly so the tunnel allowlist
documents the endpoint.

### Health & Status Endpoints

`GET /api/health` is the **unified** health endpoint: it returns the startup/readiness/liveness
probes plus the canonical `status` (`running|starting|stopped`), `processId` and `processType`.
It requires no authentication inside the Worker, but at the Cloudflare edge it is
Access-protected by path-scoped applications (`*.fouria.io/api/health`) that carry
no role requirement: they admit any authenticated user (browser) and the lerma
health-monitor service token (headless).
`GET /api/status`, `GET /api/admin/gateway/status/polling` and `GET /api/admin/gateway/status/ws` are thin
aliases over the same snapshot. Browser clients use the sibling paths outside `/api/health/*` —
`GET /api/health-ws` (WebSocket stream every ~2s), `GET /api/health-snapshot` (HTTP snapshot), and
`GET /api/health-components` (Cloudflare component states) — while `/api/health`, `/api/health/ws`,
and `/api/health/components` remain mounted for headless service-token consumers (lerma). The
browser-facing sibling paths exist because `/api/health` and `/api/health/*` are covered by the
path-scoped `FourIA Health - <slug>` Access app whose immutable AUD differs from the host app's: the
browser's host-app session is invalid there, so Access answers with a 302 login redirect to
`*.cloudflareaccess.com` — which WebSocket handshakes cannot follow and browser `fetch()` calls
reject on cross-origin CORS.

By default `/api/health` **warms** the container (lerma keep-alive). `?passive=1` probes **without** warming
— the ROOT `/_admin/diagnostics` surface uses it so an operator can debug an instance whose container is
down. The admin dashboard subscribes to the health WebSocket (with an HTTP polling fallback) and never
blocks the UI on gateway status.

`GET /healthz` is a Kubernetes-style **liveness** probe: one non-waking
`isRuntimeActive()` read, always `200 { ok, ts, status }` (`status` = `active` |
`idle`), never warming the container or probing the gateway. It exists because
`/healthz` otherwise fell through to the gateway proxy (`app.all('*')`), which
warmed the container and waited for the port — ~108 s on a cold instance, so the
platform killed the request before it answered. Readiness stays on `/api/health`.

A single `/api/health` request issues **one** gateway `/health` `curl` exec, shared by the
readiness and liveness probes. They previously each ran their own, so every poll spawned a
**pair** of identical `curl` processes (visible in `/debug/processes`) — a duplicate on the
frequent lerma probe. The process scan and the `/health` probe are now both fetched once, in
parallel, and reused across the snapshot.

#### Gateway memory (`gateway: { pid, rssBytes, threads }`)

The gateway is one Node process in a 4 GiB / no-swap container whose RSS is dominated by
per-`worker_thread` V8 heaps that an in-process restart never reclaims
(`docs/plans/sandbox-memory-usage.md`). A dev tenant "rebooted" once and the only way to answer
"was it memory?" was to SSH in and read `/proc`, `dmesg` and the gateway log by hand.

`src/gateway/memory.ts` makes that a first-class signal:

| Piece                            | Behaviour                                                                                                                                                                                                                                                                                   |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `buildGatewayResourceScript()`   | One shell exec: `pgrep -x 'openclaw-gatewa'` → `gw:<pid>:<rssKb>:<threads>` per live gateway server (same exact `comm` as the soft reload, so it never measures the `openclaw gateway run` launcher; zombies skipped).                                                                      |
| `readGatewayResources()`         | Runs it (`gateway.memory-read`, bounded) and returns the highest-RSS process. Never throws: a failed read is `null`, not a broken health probe.                                                                                                                                             |
| `sampleGatewayResources()`       | Throttles the read to `GATEWAY_RESOURCE_SAMPLE_INTERVAL_MS` (60s), so the frequent health poll issues at most **one** extra exec per interval and only a `fresh` sample writes a metrics row. A cached `null` is served inside the window too, so a broken probe is not retried every poll. |
| `maybeRestartGatewayForMemory()` | The dev-gated watchdog (below).                                                                                                                                                                                                                                                             |

`attachGatewayResources()` in `routes/public.ts` attaches the sample to the snapshot as
`gateway` and writes a `gateway-resources` Analytics Engine row (`double1` = RSS bytes,
`double2` = threads, blob1 = `gateway-resources`, blob2 = customer, trailing
`platform`/`environment` blobs like every other cost row). It is applied by **both** `/api/health`
(and `/api/health-snapshot`) and the `/api/health-ws` stream, and **only when the snapshot is
`running`** — a sleeping/stopped container is never woken for a diagnostic.

**Dev-gated memory watchdog.** When the sample reaches `GATEWAY_RSS_RESTART_BYTES` (3 GiB of the
4 GiB container) on a **dev** instance (`isDevAgentsEnabled`: `-dev` hostname, `OPENCLAW_DEV_MODE`,
or local `DEV_MODE`), the gateway is replaced — `killGateway` + `resetStartupCache` +
`ensureGateway({ skipRestore: true, bypassMaintenance: true })`. That is the only mechanism that
actually returns RSS to baseline.

Design constraints worth keeping:

- **The threshold guards the container limit, not OpenClaw's warning.** OpenClaw's own
  `rss_threshold` is 1.5 GiB and fires permanently on a healthy-but-large dev gateway
  (~2.1 GiB observed, ~2.6 GiB peak). Using it as the action threshold would restart the gateway
  in a loop.
- **No soft-reload step.** A measured in-process `SIGUSR2` restart does **not** reclaim RSS
  (1.14 → 1.38 → 1.18 GiB), so reloading first would only add a failed attempt.
- **`skipRestore`.** A kill leaves the filesystem untouched; re-entering the R2 restore barrier
  would deadlock on the backup lock for no benefit.
- **Cooldown claimed before acting.** `GATEWAY_MEMORY_WATCHDOG_COOLDOWN_MS` (30 min) is recorded
  _before_ the replacement, so two concurrent health polls cannot start two restarts.
- **Detached.** The watchdog is dispatched through `executionCtx.waitUntil` (or left detached when
  there is no context), so a multi-minute replacement never delays or fails a health response.

Every decision returns a stable machine reason (`not-dev`, `no-resources`, `below-limit`,
`cooldown`, `rss-above-limit`), and an actual restart logs a structured
`gateway.memory.pressure` event with `pid`, `rssBytes`, `rssMib`, `threads` and the limit.

#### Cloudflare component states

The dedicated `GET /api/health/components` endpoint returns a quick view of the
Worker-bound platform pieces rendered by the ROOT diagnostics horizontal
**Cloudflare Components** widget. `GET /api/health` also carries the same
(redacted) `components` entry — but **only on the non-passive path**. Lerma's
HealthMonitor polls `/api/health?passive=1` frequently and never reads
`components`, so the passive probe skips the component probe entirely (no
`isRuntimeActive` read, no R2 HEAD) to keep the frequent poll slim:

| Component      | Probe                                                                                  |
| -------------- | -------------------------------------------------------------------------------------- |
| Container      | Non-waking `sandbox.isRuntimeActive()` (`up` / `sleeping` / `unknown`)                 |
| Durable Object | `Sandbox` + `ACTIVITY` bindings present and the control channel answered               |
| R2             | Bindings-shaped `BACKUP_BUCKET` plus a HEAD of `backup-handle.json`                    |
| AI Gateway     | Present **only** when all three gateway credentials are configured (`up` = configured) |

Every probe is **non-waking** and individually bounded (3s), so the endpoint is
safe on a sleeping instance and can never hang the health response; on failure
the `components` entry is omitted instead of blocking. The AI Gateway is
inclusion-gated on credentials and reported as `up` ("configured") — the probe
deliberately does not issue an authenticated Cloudflare API call on every poll,
and a missing account-scoped token must not read as a false outage.

Each component also carries binding-derived `facts` (a `{ label, value }[]`
shown in the widget's hover panel) and a Cloudflare dashboard `logsUrl` (a
per-product logs/metrics deep link opened on click). Both are derived from
Worker bindings/env only — **never** an authenticated Cloudflare API call — so
the surface stays non-waking and free (`src/gateway/cloudflare-links.ts` builds
the links: Container → Workers Logs/observability, Durable Object → DO
metrics/logs, R2 → the `BACKUP_BUCKET_NAME` bucket page, AI Gateway → the
gateway page). When the account id is not bound the links fall back to the
dashboard's documented `?to=/:account/…` form; a component without a resolvable
link renders as a plain (non-clickable) card. The unified `GET /api/health`
snapshot **redacts** `facts`/`logsUrl` (`redactComponentDetails`) — it is a
broadly-consumed public probe — and serves only the state summary; the enriched
view is exclusive to `GET /api/health/components`.

On phones the widget collapses into a compact **status widget** that shares a
row with the gateway and persistence-lock status widgets at the top of the
diagnostics page (not floated or overlaid): a single pill carrying the Cloudflare
icon, a worst-case rollup status dot and an `up/total` summary. Tapping it
expands the same component strip as a compact panel. To build that parallel row,
`AdminLayout` omits the topbar gateway + persistence indicators on the
diagnostics page on phones (`App.tsx`) and `DiagnosticDashboard` renders
`GatewayStatusLed` + `PersistenceLockIndicator` beside the Cloudflare widget. The
gateway pill gains an identifying icon and the lock an
`Idle`/`Backing up`/`Restoring` label (its color still encodes the state); both
use compact phone sizing so the three widgets stay on one line on normal phones
and wrap gracefully on very narrow ones. The desktop layout is unchanged — the
topbar keeps its indicators and the Cloudflare card renders full-width.

The ROOT `/_admin/diagnostics` page renders its operational tools as separate
clickable **Fouria DevOps** widget cards (backup connectivity + configuration
editor + paired devices). Each card's title row toggles its body while the
action buttons stay live, and the cards load their data asynchronously and
render their chrome immediately — there is no blocking loading animation for the
data that populates them. On phones each option collapses into a vertical
**icon + name** menu row (collapsed by default; tap to open) with the option's
actions revealed inside the opened body. The Configuration Editor reflows to the
viewport: action buttons stack full-width, the section tabs scroll horizontally,
long JSON wraps, and the raw textarea shrinks. The Debug Tools grid drops to a
single column so nothing forces sideways scrolling, and the admin bottom nav
uses flexible items (with safe-area insets) so every nav entry fits. The
floating dev role switcher (`DevUserSelectorCompact`) is lifted above the bottom
nav on phones so it neither covers the last nav item nor overflows the viewport.

The gateway status indicator lives in the admin **topbar** beside the
persistence-lock indicator and the notification bell (`GatewayStatusLed`, mounted
in `App.tsx`'s `AdminLayout`). Its color-coded LED reflects the gateway status
(`running`/`starting`/`sleeping`/`stopped`/`unknown`); clicking it reveals the
complete status value (status, process, process type, the startup/readiness/
liveness probes, restore state and last update) and a **reboot icon** that
performs the same soft-reload/restart as the removed Gateway Controls banner. The
reboot action is hidden behind an `Admin` badge when `/debug/*` is disabled,
matching the previous gating.

#### Live gateway detection

On the Worker-driven cold start the gateway is a **supervised `sandbox.exec`
process** (`/bin/bash -lc 'exec openclaw gateway run …'`), so it _is_ visible to
`listProcesses()` for the life of the container. The legacy `full` mode (manual
runs, `test:gateway:boot`) still launches it as a detached background child of
`start-openclaw.sh` (`… &`), and a container provisioned by an older image may
still be serving a detached gateway during a rolling deploy — for those, after
the script exits the gateway binary is **invisible to `listProcesses()`**.
To avoid false `no_process` reports on log/status surfaces, those endpoints
(`GET /api/admin/logs`, `GET /debug/logs`, `GET /debug/stop-gateway`,
`POST /api/admin/gateway/restart`, `GET /api/admin/audit-logs`) call
`findExistingGatewayProcess(sandbox, { probeLive: true })`, which falls back to
a bounded `/health` probe and — when the gateway responds — returns a synthetic
`gateway-live` process whose logs are read from `/tmp/gateway-stdout.log`. The
passive health snapshot path deliberately does **not** probe so a down container
fails fast.

### Log Levels

| Level   | When to use                   |
| ------- | ----------------------------- |
| `debug` | Local development             |
| `info`  | Production default            |
| `warn`  | High-traffic production       |
| `error` | Security-sensitive production |

---

## TypeScript & Tooling

### Type Generation

Worker types are auto-generated:

```bash
bun run types   # wrangler types → worker-configuration.d.ts
```

**Never manually edit `worker-configuration.d.ts`** — it's regenerated from `wrangler.jsonc`.

### Testing

```bash
bun run test             # Unit tests (Vitest)
bun run test:coverage    # With coverage report
bun run test:client      # SPA component/page tests (jsdom)
bun run test:ui          # Playwright UI suite (Vite dev server + mocked API)
bun run test:ui:headed   # Same, with a visible browser
bun run test:e2e         # E2E tests (cctr + Playwright)
bun run test:e2e:cli     # CLI-only E2E (cctr)
```

The Playwright UI suite (`test/ui/`, config `playwright.config.ts`) drives the
real SPA served by `bun run dev` with per-test `/api/*` mocks — no Worker,
container, or Cloudflare credentials. It covers the app shell, role-based
navigation and redirects, every admin page's render, and the notification bell.
See [docs/TESTING.md](/latest/llms/platform/reference/testing/index.md#ui-tests-playwright-local) for the
full description and the Nix `PLAYWRIGHT_BROWSERS_PATH` notes.

### Formatting & Linting

```bash
bun run format          # oxfmt (auto-format)
bun run format:check    # Check formatting
bun run lint            # oxlint
bun run typecheck       # tsc --noEmit
```

---

## Common Tasks

### Adding a New API Endpoint

1. Create route handler in `src/routes/api/<domain>.ts`
2. Register in `src/routes/api.ts`
3. Add client method in `src/client/api.ts`
4. Write tests (unit + integration)
5. If new dependencies: update `package.json`

### Adding a New Environment Variable

1. Add to `OpenClawEnv` interface in `src/types.ts`
2. If passed to container: add to `buildEnvVars()` in `src/gateway/env.ts`
3. Update `.dev.vars.example`
4. Document in [../../docs/ENV-VARIABLES.md](../../docs/ENV-VARIABLES.md)
