FourIA — Worker Dashboard Architecture
On this page
Deep-dive into the apps/fouria component, the core Tenant Worker of FourIA.
See ../docs/DEVELOPMENT.md for project-level overview and ../docs/ENV-VARIABLES.md for environment variable reference.
Architecture Overview
Browser
│
▼
┌─────────────────────────────────────────┐
│ Cloudflare Worker (index.ts) │
│ │
│ Middleware: Logging → Auth → RBAC │
│ │
│ Routes: │
│ /api/* → Hono API routes │
│ /_admin/* → React SPA (Vite build) │
│ /debug/* → Debug endpoints (ROOT) │
│ /* → Proxy to OpenClaw GW │
│ │
│ Activation: Worker restore → onboard │
│ → patch → launch gateway │
└──────────────┬──────────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Cloudflare Sandbox Container │
│ │
│ start-openclaw.sh: │
│ 1. openclaw onboard (auth setup) │
│ 2. Enable plugins (unified billing) │
│ 3. moltlazy patch (config injection) │
│ 4. Start Cloudflare Tunnel (optional) │
│ 5. background gateway + health poll │
│ │
│ ┌────────────────────────────────────┐ │
│ │ OpenClaw Gateway │ │
│ │ - Control UI on port 18789 │ │
│ │ - WebSocket RPC protocol │ │
│ │ - Agent runtime │ │
│ └────────────────────────────────────┘ │
└─────────────────────────────────────────┘
Frontend Architecture (Vite + React SPA)
The dashboard is a React 19 SPA built with Vite 6 and served from /_admin/.
Build & Dev Flow
bun run dev → Vite dev server on :8787 (vite.ui.config.ts)
│ Hot-reload for frontend changes
│ Proxies /api/*, /debug/* to :8788 (worker)
│
bun run dev:worker → Vite dev server on :8788 (vite.config.ts)
│ Runs the actual Worker in workerd via the
│ Cloudflare Vite plugin; Worker changes hot-reload
│
bun start → bun run build:docker && vite dev
Full local stack (build + Worker dev server)
Key insight: both dev servers are Vite.
bun run devonly handles the admin SPA; the Worker (routes, auth, gateway, DOs, containers) runs under the Cloudflare Vite plugin viabun run dev:worker, the same plugin used byvite build. The plugin has nowrangler devflags: local E2E vars are injected through itsconfighook and.dev.varsis loaded from this directory.
Vite Configuration
// vite.config.ts — Worker dev server (bun run dev:worker)
base: '/_admin/' // All assets served under /_admin/
server: {
port: 8788,
allowedHosts, // DEV_ALLOWED_HOSTS (tunnel hostnames)
}
// vite.ui.config.ts — admin SPA dev server (bun run dev, :8787)
server: {
port: 8787,
proxy: {
'/api': 'http://localhost:8788',
'/debug': 'http://localhost:8788',
'/cdp': 'http://localhost:8788',
},
}
Exposing the Worker through a tunnel (e.g. Telegram webhooks): Vite blocks unknown
Hostheaders, so pass the public hostname explicitly —DEV_ALLOWED_HOSTS=foo.trycloudflare.com bun run dev:worker(comma-separated; a leading dot allows a whole domain). Point the tunnel at:8788, not the SPA at:8787(it does not proxy/telegram-webhook).
Client Module Structure
src/client/
├── App.tsx # Root component with React Router
├── main.tsx # Entry point — mounts React app
├── api.ts # Typed API client → /api/admin/*
├── gateway-gate.ts # Health circuit breaker (park/resolve gateway queries)
├── GatewayHealthContext.tsx # App-wide passive health subscription + gate wiring
├── pages/ # Route-level pages
│ ├── HomePage.tsx # Dashboard overview
│ ├── AgentsPage.tsx # Agent management
│ ├── ChannelsPage.tsx # Connections/integrations
│ ├── KnowledgePage.tsx # Knowledge base/wiki
│ ├── AccountPage.tsx # User/team management
│ ├── IntegrationsPage.tsx # Integration config
│ ├── SecretsPage.tsx # Vault management
│ ├── BackupsPage.tsx # Backup management
│ ├── DiagnosticDashboard.tsx # Costs + diagnostics
│ ├── AuditLogsPage.tsx # Gateway ledgers: audit activity + background tasks
│ └── AutomationEditorPage.tsx # Automation workflows
├── components/ # Shared UI components
└── hooks/ # Custom React hooksFrontend ↔ Backend Communication
The frontend (src/client/api.ts) calls /api/admin/* endpoints. These are served by the Worker’s Hono routes (src/routes/api.ts), which:
- Verify RBAC (
requireWriteRole()for mutations) - Execute OpenClaw CLI commands inside the Sandbox container
- Parse CLI output and return JSON
Client circuit breakers (health-gated gateway queries)
Two client-side circuit breakers keep the dashboard responsive when the OpenClaw gateway is not ready (issue #371):
- #1 — gateway queries are asynchronous. Pages paint immediately and never block first paint on a gateway response; mutations are fire-and-notify through
NotificationProvider.api.tsroutes every gateway-touching call throughgatewayApiRequest. - #2 — gateway queries require the passive health stream to be UP. The single app-wide
GatewayHealthProvider(src/client/GatewayHealthContext.tsx) subscribes once to/api/health-ws?passive=1and drives a module-singletonGatewayGate(src/client/gateway-gate.ts). UP is defined assnapshot.readiness.ok === true;starting/stopped/sleepingare not UP (sleepingis healthy-idle but the gateway is not running). Before dispatching,apiRequestawaitswhenGatewayUp(signal); while down the query is parked and theGatewayGateNoticeindicator shows, then the queue flushes automatically on the next UP snapshot. A transient (503/transport) failure is retried once after the next UP.
The gateway-touching boundary is the GATEWAY_TOUCHING_PREFIXES manifest in api.ts (/agents, /models, /channels, /config, /devices, /integrations, /secrets, /knowledge, /audit-logs, /dreaming, /gateway/restart). Diagnostics (/debug/*, /logs, /gateway-api, /gateway/status/polling, /gateway-token, /version) and worker-local surfaces (/storage, /costs, /billing, /users, /vault, /releases, /fouria-docs, /whoami) are deliberately not gated, so an operator can still inspect a down gateway and non-gateway features keep working.
Diagnostics (DiagnosticDashboard). The ROOT dashboard is explicitly skip-not-park: useGatewayRunning() (GatewayHealthContext.tsx) reports snapshot.readiness.ok ignoring the ROOT bypass, and the page only dispatches its container reads (/config/raw, /config) while it is true; they re-run automatically when the gateway transitions back to running. The debug-routes flag is Worker-local and still loads. Every call is dispatched after paint (scheduleAfterPaint → requestIdleCallback, setTimeout(0) fallback) and the widgets render their waiting state first, so nothing is blocked on a container response: the Configuration Editor renders its tabs/editor with a non-blocking “Loading configuration…” / “Waiting for the gateway to start…” notice (gatewayReady prop on ConfigSection) and the device list shows a waiting state until Refresh. Gateway-only actions (device list/approve, config save, channel probe) are disabled/guarded while down; doctor --fix repair is NOT one of them: it is the recovery action for a broken/down gateway (the backend stops the gateway itself), so a dedicated Repair Environment control sits outside the Configuration Editor (always rendered for ADMIN/ROOT, disabled only while a repair is in flight) and the in-editor repair button stays enabled while the gateway is down. The Debug Tools that call the container (logs, processes, version, sqlite integrity) and the recovery actions (restart/rescue/destroy) stay available on purpose — they are how an operator inspects and revives a down gateway — while the ones that require a running gateway (gateway-api, stop-gateway) are skipped with an inline explanation.
ROOT bypass. ROOT operators are never gated: App.tsx passes bypass={whoami?.role === 'ROOT'} to the provider, which flips the gate’s bypass (GatewayGate.setBypass). This preserves the “ROOT can always reach a tenant” rule and keeps the operator’s non-passive device-pair request (GET /api/admin/devices/ws → listDevices → ensureGateway({ waitForReady: false })) able to wake a sleeping container. The notification bell is passive for every non-ROOT role (subscribeDeviceNotifications(..., { passive: true }) → ?passive=1), so a regular dashboard load never wakes a sleeping tenant; only ROOT’s stream is non-passive. Server-side, a passive /api/admin/devices/ws first does the non-waking readRuntimeActive() read and, when the runtime is not live, emits an empty snapshot and skips the gateway event stream. Once the ROOT wake brings the gateway UP, the shared health stream flips to up and every parked query (for all roles) flushes. The GatewayGateNotice is suppressed while bypassed, since ROOT actions are never queued.
The gate is fail-open when no provider is mounted (non-SPA callers, unit tests): the module default is up. The provider installs the shared gate during its first render, before any child page effect runs. It is a module singleton rather than a useState initializer because React StrictMode double-invokes initializers in dev — two gates would leave api.ts parked on a different instance than the provider drives.
Backend Architecture (Hono Worker)
Request Pipeline
Incoming Request
│
├─► Logging middleware (all routes)
├─► Sandbox init (getSandbox() into context)
│
├─► Public routes (no auth):
│ /sandbox-health, /api/otel/*
│
├─► CF Access Auth middleware:
│ Extract JWT → validate → set accessUser
│
├─► RBAC resolution:
│ resolveCustomerRole → set userRole, customerSlug
│
├─► Route dispatch:
│ /api/* → API routes (auth + RBAC)
│ /_admin/* → Admin UI (SPA assets)
│ /debug/* → Debug routes (ROOT only)
│ /* → Catch-all proxy to gateway
│
└─► Gateway proxy:
HTTP proxy → containerFetch(port 18789)
WS proxy → WebSocket relay with token injectionAPI Route Structure
// src/routes/api.ts — route aggregation
api.route('/admin', adminApi)
├── /whoami, /version, /logs, /gateway-api, /debug-status, /gateway-token (misc)
├── /devices/* → devicesRoutes
├── /storage/* → storageRoutes
├── /releases/* → releasesRoutes
├── /gateway/* → gatewayRoutes
├── /config/* → configRoutes
├── /cron/* → cronRoutes
├── /dreaming/* → dreamingRoutes
├── /users/* → usersRoutes
├── /integrations/* → integrationsRoutes
├── /knowledge/* → knowledgeRoutes
├── /secrets/* → secretsRoutes
├── /agents/* → agentsRoutes
├── /models/* → modelsRoutes
├── /channels/* → channelsRoutes
├── /costs/* → costsRoutes
├── /vault/* → vaultRoutes
├── /billing/* → billingRoutes (plan, checkout, portal, invoices, freeze-status, freeze, unfreeze)
└── /audit-logs/* → auditLogsRoutes (audit activity ledger + task ledger)
// Write gate: POST/PUT/DELETE/PATCH require ADMIN or ROOT
// Read gate: GET/HEAD available to all authenticated roles
Build identity.
GET /api/admin/versionandGET /debug/versionreport the FourIA Worker’s ownfouria_version(component CalVer fromapps/fouria/package.json) andfouria_commit(short git SHA) alongside the container’s OpenClaw/Node versions. Both are injected into the bundle byvite.config.ts(__FOURIA_VERSION__/__FOURIA_COMMIT__, consumed insrc/build-info.ts;GITHUB_SHAwins in CI,git rev-parse --short HEADlocally). The DiagnosticDashboard Version Info tool surfaces them for operators.
Cloudflare error and trace correlation
The Worker emits structured worker.request.started and
worker.request.completed records with the method, path, status, duration, and
Cloudflare Ray ID (cfRay). Unhandled exceptions are logged as
worker.request.failed with the exception name/message/stack and the same
request fields. Cloudflare also attaches its invocation $metadata.requestId
and $metadata.traceId to each Worker log, so the invocation’s request, logs,
and trace can be opened together in Workers Observability. Error records pass an
Error object to console.error as well, so Cloudflare populates its exception
metadata (including $metadata.error) alongside the indexed event fields.
Sandbox command failures are emitted as sandbox.process.failed. These records
include a stable operation label and phase, the Sandbox processId when a
process was launched, and available exit-code/timeout/duration fields. The
processId is the join key to the processId field on the Containers dataset’s
process.complete event; inspect exitCode and timedOut there rather than
trusting the generic event message, which can say success when the process
itself exited non-zero. Raw shell commands, stdout, and stderr are deliberately
omitted because they can contain tenant config or credentials.
Persistence failures use stable events such as persistence.backup.failed,
persistence.config-version.failed, persistence.restore.failed, and
storage.restore.failed. Config-version and restore errors include backupId
when an archive exists; use it to join to the backup.create / backup.restore
container events. A config-version capture failure is best-effort: it identifies
the archive whose supplemental config snapshot was not captured, but does not
mean the archive itself failed to be created.
Backup and restore work also emits paired persistence.phase records. Each
record has an operationId, operation (backup or restore), phase, and
status (started, completed, or failed); terminal records include
durationMs, and phases after an archive handle exists include backupId.
source names the route or job, cfRayId correlates request-triggered work with
worker.request.*, and jobId correlates work running later in the
ActivityScheduler. Use operationId to group the phases; if a started phase has
no terminal record, the Worker may have been interrupted during that phase.
Useful phases include sandbox.createBackup, archive.encrypt,
catalog.commit, sandbox.restoreBackup, and legacy-state.materialize.
For a failure investigation, filter the cloudflare-workers dataset to
event = "sandbox.process.failed", record processId and the Worker
$metadata.requestId/$metadata.traceId, then search the containers dataset
for the same processId (and the tenant’s $container.applicationId). For a
request-level failure, filter to event = "worker.request.failed" or use
cfRay to pivot to the matching edge request. For backup/restore CPU-limit
investigations, filter to event = "persistence.phase", find the last
started event for the relevant cfRayId or jobId, then group by its
operationId to see which phases completed before it.
Worker observability export destinations
Account-level Cloudflare Observability → Destinations export this Worker’s
logs and traces to Lerma over OTLP/HTTP JSON (issue #337). The destination names
are referenced in the observability block:
// staging
"observability": {
"logs": { "destinations": ["lerma-worker-logs"], "head_sampling_rate": 0.1, "persist": false },
"traces": { "destinations": ["lerma-worker-traces"], "head_sampling_rate": 0.1, "persist": false }
}
// production
"observability": {
"logs": { "destinations": ["lerma-worker-logs-prod"], "head_sampling_rate": 0.1, "persist": false },
"traces": { "destinations": ["lerma-worker-traces-prod"], "head_sampling_rate": 0.1, "persist": false }
}
The -prod destinations point at https://lerma.paso4.io/api/ingest/otel/v1/*;
the un-suffixed pair points at https://lerma-dev.paso4.io/....
apps/fouria/wrangler.jsonccarries the matching names in each env block (stagingandproduction, direct deploys).- WFP-deployed tenant workers get the names from Lerma’s
Lerma.Cloudflare.Observability.worker_metadata/0(WORKER_OBSERVABILITY_LOGS_DESTINATIONS/WORKER_OBSERVABILITY_TRACES_DESTINATIONS). The CD sets the staging names for staging and the-prodnames for production, so a redeploy inherits them. - Lerma is the destination of record:
persist: falsestops the duplicate Cloudflare-side copy, and the head sampling rate (10% at rollout) bounds export volume. Lerma drops message text, stacks, URLs/queries, user agents, request/response bodies and DB query text at ingestion; only allowlisted metadata is stored. - Cloudflare gzips each OTLP/HTTP JSON batch (
Content-Encoding: gzipwithContent-Type: application/json).LermaWeb.Plugs.OtlpBodyReaderruns beforePlug.Parserson the ingest routes and decompresses it; without that step the normal JSON parser returns400and the destination retries forever. - The export only feeds diagnostics + the Lerma instance Worker telemetry
widget. It never replaces
HealthMonitor(still the source ofinstances.status) or billing, and it is not a probing path — no container is woken to produce it.
Ops: create the two destinations under Observability → Destinations with the
custom header Authorization: Bearer <OTEL_INGEST_TOKEN>, set the token and the
destination names on Lerma (staging first), then redeploy the tenants. Delivery
status shows Last: n minutes ago; the uptime numbers come from
GET /api/analytics/v1/uptime/<script_name>?window=24h.
Gateway start lifecycle + stuck-start watchdog
A cold start is one shared promise (ensureGateway) spanning the restore
barrier → the supervised start-openclaw.sh start launch → the port wait. A
wrapper that never execs the gateway, or a container control-channel drop, can
leave that promise “in progress” for many minutes while every request logs
Gateway start already in progress … waiting... and waits on it.
The start is now tracked per sandbox key with a phase (probe → restore →
launch → port-wait) and a per-phase timer. post-start runs detached
after readiness (see below), so it is deliberately not part of the awaited start
promise:
gateway.start.begin/gateway.start.phasestructured events record the attempt, phase, elapsed time, and (after launch) the process id; the “already in progress” log carriesphase,age, andattempt.isGatewayStartStuckflags a phase that overruns its budget +GATEWAY_START_STUCK_GRACE_MS(therestorephase getsGATEWAY_RESTORE_PHASE_BUDGET_MSbecause the legacy stage may take 10 min; all other phases useSTARTUP_TIMEOUT_MS).- When the watchdog trips,
recoverStuckGatewayStartemits agateway.start.stuckstructured event, kills the stuck supervised process (killGateway), clears the startup/readiness caches, and returns so the next request starts clean. AfterGATEWAY_START_STUCK_MAX_ATTEMPTSconsecutive resets it reprovisions the container (recoverStaleSandbox). - Separately,
findExistingGatewayProcessusesclassifyGatewayStartto detect astart-openclaw.sh startwrapper older than the cold-start budget with the port closed and kills it immediately (sourcestartup-wrapper-age) instead of waiting out another full readiness budget. - A non-owner isolate that merely observes a running wrapper waits to the
same deadline the owner’s watchdog uses —
STARTUP_WRAPPER_STUCK_BUDGET_MS(STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS) measured from the wrapper’s start — and re-probes/healthbefore any kill. A fixedSTARTUP_TIMEOUT_MSmeasured from observation expired while the wrapper had alreadyexeced the gateway and was still binding, killing it mid-boot, releasing the startup lock, and making the waiting second wrapper re-run the entire setup (observed onstaging-client-dev: a ~14-minute cold boot). The kill now only fires once the wrapper has genuinely overrun the owner’s own budget with the port still closed.
The port-wait also completes on a positive /health probe. The SDK’s
waitForPort drives a container-side port-watch + log-readiness subscription
that can hang indefinitely on some boots even after the gateway is serving.
Waiting on it alone let the shared start promise overrun its budget, so the
watchdog force-killed a healthy gateway and reprovisioned the container
(observed on tenant-client-dev: gateway.start.stuck phase=port-wait after
368 s while HTTP proxying returned 200). The wait races waitForPortResilient
against the same positive /health probe the public proxy uses. If that waiter
reaches its deadline and parks, the watchdog makes one fresh bounded health
check before resetting: a late-positive response emits
gateway.start.late-ready and is adopted without killing the gateway. A failed
health RPC is treated as unknown, not closed, and defers the reset until a
later request can probe again; only a confirmed closed port follows the normal
stuck-start reset path.
Process count is not a health signal. listProcesses() returns every
tracked process for the container’s lifetime — including exited/error
entries the Worker never garbage-collects — so the raw length only grows. The
log line now reads listProcesses returned <total> processes (<running> running)
and every health decision (classifyGatewayStart, the gateway matcher) considers
only running entries. The SDK exposes no completed-process cleanup, so the
list is append-only; never gate health on the total.
Gateway protocol ledgers (/api/admin/audit-logs)
The AuditLogsPage reads the two append-only ledgers from the Gateway protocol
(docs.openclaw.ai/gateway/protocol/ledgers):
GET /api/admin/audit-logs— audit activity ledger (audit.activity.list,operator.read), newest first. SupportsagentId,sessionKey,runId,kind,status,direction,channel,after,before,limit, andcursor; returns{ events, nextCursor? }. The bundledadmin-http-rpcplugin allowliststasks.*but notaudit.*, so this read goes over the Gateway WebSocket RPC using OpenClaw’s reference client (@openclaw/gateway-client/browser+@openclaw/gateway-protocol) bridged throughsandbox.wsConnect, authenticating with the shared gateway token as anoperator(seesrc/gateway/gateway-rpc.ts). It falls back toopenclaw audit --jsonwhen the WS path is unavailable (older Gateway); the CLI fallback gets the resolvedOPENCLAW_GATEWAY_TOKENinjected because a freshsandbox.execshell does not inherit the gateway process env.GET /api/admin/audit-logs/tasks— task ledger (tasks.list,operator.read), newest first. Supportsstatus(repeatable),agentId,sessionKey,limit,cursor, andsortBy; returns{ tasks, nextCursor? }. RPC-first over the admin-http-rpc plugin, with anopenclaw tasks list --jsonCLI fallback (same token injection).
Both surfaces are ADMIN-only (absent from DIAGNOSTICS_PATHS) and return an
empty page with an error string when the ledger is unavailable, so the
dashboard degrades gracefully instead of failing the request.
Why a WebSocket client.
admin-http-rpconly exposes an allowlisted method subset. Methods outside it (notablyaudit.*,plugins.*,secrets.*,agents.files.*) require the wire-v4 Gateway WebSocket handshake (connect.challenge→connect→hello-ok).src/gateway/gateway-rpc.tsadaptssandbox.wsConnectto the reference client’sGatewayProtocolSocket; the client version sent in the connect frame is injected at build time fromcatalog.openclaw(__OPENCLAW_VERSION__).
RPC-first agent board + integrations
Eligible dashboard routes try Admin HTTP RPC first, the official Gateway
WebSocket RPC second, and keep the CLI as the final fallback for older
Gateways (helpers return null/false on transport or method failure):
- Agent board (
/api/admin/agents) — list readsagents.listand channel bindings inconfig.getover Admin HTTP RPC. If either result is unavailable, the Gateway WebSocket client fetches the missing read(s) over one connection; the CLI remains the final fallback. Create →agents.create; delete →agents.delete; identity (name/emoji/avatar) →agents.update; model (+ fallbacks) →config.patchonagents.entries.<id>.model. Explicit bind/unbind operations have no matching RPC method and stay on the CLI; persona read/write are filesystem operations. - Devices (
/api/admin/devices) — list, approve, reject, and approve-all retain Admin HTTP RPC as the first path, use Gateway WebSocket methods (device.pair.list,.approve,.reject) if HTTP RPC is unavailable, and retain the CLI as the compatibility fallback. - Models (
/api/admin/models) and channel probe (/api/admin/channels/probe) —models.listandchannels.statususe Admin HTTP RPC first, then Gateway WebSocket RPC, then the existing CLI path. - Integrations (
/api/admin/integrations) — the whole endpoint is WebSocket-free.GET /reads the allowlistedchannels.status+agents.listover the admin HTTP RPC, and resolves plugin enabled-state fromconfig.get(plugins.entries.<id>.enabled) merged with the staticPLUGIN_FIELDScatalog — noplugins.listsocket. When no channel accounts are configured the supported-channel catalog is synthesized locally fromCHANNEL_FIELDSinstead of shelling out toopenclaw channels list --all. Plugin/channel toggles and non-secret configure fields →config.patch; secrets reload → theopenclaw secrets reloadCLI.secrets.applyand agent bindings have no 1:1 RPC method and stay on the CLI. CLI fallbacks remain for a degraded HTTP RPC transport. The admin RPC fetcher also cancels a latecontainerFetchbody when its timeout wins, preventing the Sandbox DO’s “ReadableStream received over RPC disconnected prematurely” warning and the churn that dropped later sockets. The same cleanup is shared by the public proxy and thegateway-api/diagnostics probes viasrc/utils/container-rpc.ts(containerFetchWithTimeoutcancels a late body,wsConnectWithTimeoutcloses a late socket). Reading a running gateway’s logs must not open its live SDK stream (it never settles and is later disconnected):GatewayProcess.getLogs()tails/tmp/gateway-stdout.log, and/debug/processes?logs=true//api/admin/logs?id=refuse a live stream for a still-running process. The catch-all proxy logs only an allowlisted header subset (safeHeaderLog) — never the forwarded map, which carries CF Access credentials and cookies. - Config / secrets / agent resolution — config set/unset and dreaming
settings use
config.patchover Admin HTTP RPC, then Gateway WebSocket RPC, then the CLI fallback.POST /api/admin/secrets/reloadusessecrets.reload(WebSocket);resolveAgentFlag(shared by themodels/knowledgeCLI calls) resolves the roster withagents.listover Admin HTTP RPC, then WebSocket RPC, then CLI.
secrets applyhas no--yes. The current CLI accepts only--from <plan.json>,--dry-run,--allow-exec, and--json; the plan itself is the confirmation. Bothintegrationsanddreamingapply without--yes.
Realtime device-pairing notifications (/api/admin/devices/ws)
The dashboard NotificationBell subscribes to GET /api/admin/devices/ws:
- On connect it pushes a
{ type: "device.pair.snapshot", pending, paired }frame (best-effort, non-blocking), then relays Gatewaydevice.pair.requested/device.pair.resolved/device.pair.changedevents as{ type: "device.pair.event", event, payload }. - The Gateway subscription uses
openGatewayEventStream(the official@openclaw/gateway-clientbrowser client oversandbox.wsConnect), which reconnects with backoff until the browser socket closes. - Worker-hosted notification sockets rotate after 10 minutes with close code
1012; this is intentionally below the observed Cloudflare load-shed lifetime for long-lived Worker invocations. The dashboard reconnects and receives a fresh snapshot, releasing the old Worker invocation and Gateway subscription. - Admins approve/reject directly from the bell:
POST /api/admin/devices/:requestId/approve/.../reject(RPC-firstdevice.pair.approve/device.pair.reject, CLI fallback). The client helper issubscribeDeviceNotificationsinsrc/client/api.ts.
Dashboard notification centre. NotificationBell also renders application
notifications (backup created/failed, backup restored/failed) from the
NotificationProvider context (src/client/NotificationContext.tsx) that wraps
the admin layout. Long-running admin actions push their result there instead of
blocking on a modal, so the outcome still reaches the admin if they navigate away
before the request settles. The bell badge counts unread app notifications plus
pending device-pairing requests; opening the panel marks app notifications read
and each can be dismissed.
Persistence-lock indicator. PersistenceLockIndicator (rendered beside the
bell) shows a closed red lock while a dashboard-initiated backup or restore
is in flight, and an open green lock otherwise. The state comes from
PersistenceLockContext (src/client/PersistenceLockContext.tsx): the
BackupHistoryPanel raises it (begin('backup' | 'restore')) when it fires an
async snapshot/restore and drops it (end()) when its completion check settles
(success, failure, or timeout), so the indicator is synchronous with the
operation lifecycle — there is no polling and no lease read. A counter keeps
the lock held if two operations overlap. The widget is ADMIN/ROOT only (a
BASE_USER cannot trigger a backup).
OTel Ingest (/api/otel/*, shared-secret auth via OTEL_SHARED_SECRET)
POST /api/otel/v1/metrics— OTLP/HTTP JSON metrics (cost/tokens). AI cost rows keepopenclaw.provider+openclaw.modelso lerma can attribute the spend (unified-billing gateway-routed vs. direct provider / OTel-only).POST /api/otel/v1/traces— OTLP/HTTP JSON traces ({resourceSpans:[...]}). Onlyopenclaw.message.processed,openclaw.harness.run,openclaw.tool.executionspans are approved; others are counted asignored. Each approved span is classified (src/billing-classification.ts) into a semantic billing action (telegram_query,signal_analysis,email_draft, …) — an explicitopenclaw.billing.action_typemarker wins, span attributes are the fallback. Approved-but-unmapped spans are accepted and counted asnonBillable. Stableaction_id = lowercase(traceId) + lowercase(spanId)(no separator);action_keydedupes a business unit andtrace_iddrives precedence (email_draft/review_analysissuppresstelegram_query). Tenant is required per span (customer.sluget al.; one batch = one tenant).application/x-protobufis rejected with415(usehttp/json). Seedocs/contracts/otel-trace-action-id.md. Normalized traces are held in-memory for downstream reconcile; Analytics Engine stays source of truth. Emission guard:isBillableEventsEnabled(env)(src/dev-mode.ts) decides whether an instance may mint billable actions. Billable production always emits and cannot be opted out, and the staging platform always emits in every environment so the billing pipeline runs with production fidelity before release (its rows land in the staging Analytics Engine dataset). A dev environment is non-emitting by default — it accepts spans (200) and the cron drain drops buffered traces — and emits only whenBILLABLE_EVENTS=trueopts it in (e.g. to exercise billing end-to-end from a dev instance). The override is emission-only —getCostContextstill labels rows with the realplatform/environment(staging/devstay non-billable cost axes), andshouldAutoSleepis unchanged, so an opted-in dev instance keeps its idle timeout and staging keeps its 15m one.
Cost-tracking dimensions & development expense
Every Analytics Engine cost-tracking row (ai-query, container-active,
r2-operation, cron-trigger, ws-session, action-completed, action-step,
health-check) carries two trailing blobs — platform (production |
staging) and environment (production | dev) — resolved by
getCostContext(env) (src/metrics.ts → resolveInstanceDimensions in
src/dev-mode.ts). The canonical customer key is the base client slug
(CUSTOMER_SLUG, else tenant-{slug}[-dev] with the prefix/suffix stripped),
so worker-side rows join against the client.slug lerma reconciles. health-check
rows (emitted from non-passive GET /api/health) feed lerma’s uptime/downtime
statistics. Dev/staging rows are non-billable development expense — infra +
AI cost of building for a client, tracked in AE and pulled asynchronously by
lerma, never charged to the client’s credit pool.
Software Provisioning (declared, re-applied every reprovision)
Software that agent skills install at runtime (a plugin, an apt/npm
package, a downloaded binary) lives only in the container filesystem, so it
is lost when the image is upgraded or the container is destroyed on idle-sleep.
The softwareProvisioning feature flag makes that software declarative:
| Step | Where |
|---|---|
| Declaration | customers/<slug>/software.json in BACKUP_BUCKET (PUT /api/admin/software/manifest), { id, check, install, verify?, timeoutSec? } per package |
| Materialization | materializeSoftwareManifest() → /home/openclaw/.openclaw/software.json (inside the backup root, so a warm restore also carries it) |
| Reconcile | moltlazy software provision (packages/moltlazy/src/software/): check → skip, else install → optional verify; per-package timeout, non-fatal, exit 0 |
| Boot trigger | ensureGatewayImpl dispatches the CLI detached (void spawnSoftwareProvision(...)) just before start-openclaw.sh start, so it runs in parallel and never blocks readiness |
| On-demand trigger | POST /api/admin/software/provision (ADMIN/ROOT) runs the same CLI to completion, bounded by SOFTWARE_PROVISION_RUN_TIMEOUT_MS (5 min) |
It applies on every provisioning cycle — cold start, restored boot, idle
wake, image upgrade, and /debug/restart-openclaw — because it lives in the
Worker’s ensureGatewayImpl rather than in the (skippable) setup body of
start-openclaw.sh. A soft reload (SIGUSR2) does not re-run it; nothing is
lost there.
Outputs: /tmp/software-report.json (structured; read by
GET /api/admin/software/report) and /tmp/software-progress.log (ticks; read
by /api/startup-progress as softwareLog). The CLI truncates the progress log
at the start of every run because /tmp survives an in-place restart, so
otherwise ticks from earlier boots would be interleaved.
Operational notes:
- Both
GETendpoints returnenabledso the dashboard can distinguish “flag off”, “instance asleep”, “not run yet” and “nothing declared”; the homepage widget’s Run provisioning button (ADMIN/ROOT) calls the on-demand endpoint and renders the fresh report. - A failed package stays
failedin the report and does not abort the rest of the manifest or the boot. A report is only rewritten when the CLI actually runs, so a stale report can exist inside one container after the flag is switched off — theenabled: falsemarker is what the UI keys on. - The manifest is arbitrary shell executed in the tenant container. The upload and on-demand routes are ADMIN/ROOT-only; there is no command allowlist by design (the container is single-tenant).
Container Startup Flow (start-openclaw.sh)
start-openclaw.sh takes a phase argument ($1):
| Mode | Behaviour |
|---|---|
start | Production cold-start (single entry point): acquire the container-wide startup lock first, then run the full setup body, then become the single OpenClaw owner — execing the gateway while still holding the lock. |
setup | Standalone/smoke only: run everything up to (not including) the gateway launch, then exit. |
launch | Standalone/smoke only: acquire the container-wide launch lock and become the single OpenClaw owner (or wait for the current owner to become healthy), then exec the gateway. |
post-start | After the gateway is ready — dev automation + boot-log reset, then exit. Dispatched detached (fire-and-forget) by the Worker so it never extends the shared start promise. |
full (default) | Manual/back-compat + boot smoke: run setup, then launch + poll the gateway in the background, then exit. |
The Worker’s ensureGateway drives production cold-start as a single
/usr/local/bin/start-openclaw.sh start call followed by post-start. Unlike
the standalone phases, start acquires the container-wide startup lock before
running setup and holds it through the Gateway exec, so setup and launch are
serialized under the same owner — the Worker no longer invokes setup and
launch separately. The gateway is not a shell background job: the start
owner execs openclaw gateway run while holding the lock. The lock is held for
the Gateway process lifetime via the inherited fd (released exactly when the
gateway exits), so the supervised process is the gateway — tracked by
listProcesses/getProcess for the life of the container and killable via its
handle. A concurrent invocation that loses the lock polls /health and exits 0
without starting a duplicate owner. The serial
test/e2e/ac_gateway_startup_concurrency.txt suite checks that exactly one owner
exists and that neither the migration-input-change error nor the SQLite
offline-maintenance error appears. The standalone setup/launch modes (used
only by smoke/manual runs) keep the older split behaviour — a setup flock
that serializes the slow setup body and is released before a separate launch —
which is why ownership there is enforced solely by the dedicated launch lock.
full keeps the legacy background-launch behaviour for manual runs and
test:gateway:boot.
Restored containers skip the setup body (zero-data-loss). The R2 restore
happens in the Worker (restoreIfNeeded → sandbox.restoreBackup()), before
start-openclaw.sh runs — the script never performs the restore. When that
restore actually restored a backup, ensureGateway passes
OPENCLAW_RESTORED=true in the launch env (set only when the restore reports
hadBackup && restored, never for a brand-new tenant), and the script skips the
entire setup body — onboard → plugin installs → moltlazy patch → config
validate — emitting those phases as skipped and launching the gateway directly
from the restored filesystem. The restored state is therefore authoritative: the
cold-start budget is spent launching it, not re-mutating it (which could diverge
from or clobber the snapshot, and which wastes startup time). A brand-new tenant
(no backup) still runs the full setup body and onboards normally. Each boot
rewrites a durable receipt at /tmp/openclaw-restored-setup-skipped (written on
the skip path, removed on the setup path) so an operator can confirm which path
ran; the local backup E2E
(test/e2e-backup/zzz_backup_restore_skip_setup.txt, opt-in via
E2E_BACKUP_RESTORE=1) asserts the receipt and that the restored setup state is
left byte-for-byte intact.
OPENCLAW_RESTORED is keyed on the in-memory restore status
(getRestoreStatus() — hadBackup && restored, set by restoreIfNeeded in the
same isolate that launches the gateway), not on the durable R2
restore-needed latch. The two are deliberately separate signals:
restore-needed is a cross-isolate retry/fence latch (written before
sandbox.restoreBackup(), cleared only after restore + legacy-state
materialization succeed), while the skip-setup decision is simply the result of
the restore that just ran. A normal cold restore does not require
restore-needed to be present — its absence is normal while idle and after a
successful restore; the restore is triggered by the container-state probe
(containerHasState) plus the committed recovery point (stable-backup.json).
The loading page’s “Restoring…” spinner observes the in-memory restore status the
same way, so it appears the moment a cold restore is in flight regardless of
whether the marker was present in R2.
One targeted exception: the restored boot re-applies only the gateway
section. The Worker-critical gateway config (trustedProxies, auth,
controlUi.allowedOrigins) must self-heal even on a restored boot: a
warm-restored openclaw.json can carry a stale gateway.trustedProxies (e.g.
the pre-2026-09 default ['127.0.0.1','::1']) that omits the Cloudflare Sandbox
bridge range (10.1.0.x) the Worker proxy connects from, so OpenClaw rejects
every proxied request with proxy_attribution_required (symptom: the tenant
serves the gateway’s JSON error {"error":{"type":"proxy_attribution_required"}}
on the root HTML/WS proxy). The skip path therefore runs
moltlazy patch --only gateway --file "$CONFIG_FILE" (bounded, non-fatal): it
heals trustedProxies (drop catch-alls, union the required ranges — see
healTrustedProxies) and re-asserts auth/origins, but never regenerates the
user-editable agents/session/tools/logging sections nor re-seeds
workspaces, so the snapshot’s user data is untouched (the backup E2E fingerprints
moltlazy-session.json + a sentinel, both unaffected).
The skip is guarded on the restored config actually existing. A “restored”
boot can carry partial container state but no materialized openclaw.json — an
interrupted restore (a container-image rollout changes the runtime mid-restore),
or restoreIfNeeded’s “container already holds tenant state; skipping duplicate
restore” short-circuit, which skips the legacy-state materialization yet still
reports restored=true and therefore sets OPENCLAW_RESTORED=true. Skipping the
setup body then launches OpenClaw with no config at all (observed on
tenant-tarjetero-dev after the v2026.10.13 rollout). start-openclaw.sh
therefore only honours OPENCLAW_RESTORED=true when $CONFIG_FILE exists;
otherwise it logs a warning and falls through to the setup body, which
onboards/repairs the config instead of launching a configless gateway.
The cold boot is a durable ActivityScheduler DO job, not a request
waitUntil(). Cloudflare cancels a request’s background (waitUntil) work a
short time after the response, so a cold boot driven from a request was killed
mid-restore and re-attempted forever: [Gateway] Gateway start already in progress … (phase=restore, age=686365ms) with start-openclaw.sh never run,
141 waitUntil() tasks did not complete warnings, 14× re-decrypt of the same
archive, and 64 probe processes hammering the Sandbox DO (observed on
tenant-client-dev). The boot is now a durable job (src/activity/boot-job.ts)
on the ActivityScheduler DO: POST /api/warmup, non-passive GET /api/health
and /api/startup/ws only request it (requestBoot, deduped) and return;
the alarm performs restoreIfNeeded → ensureGateway and retries on a bounded
lease independent of any request. When the ACTIVITY binding is absent (local
dev / unit tests) the routes fall back to the request-scoped waitUntil boot.
A request never owns a destructive restore. Even with the durable boot job,
every ensureGateway call on a cold path re-asserts the restore barrier. A
request that would actually need an archive restore now defers it:
restoreIfNeeded(..., { restoreOwner: 'request' }) (the default for
ensureGateway) performs only the cheap non-destructive checks — the
restore-needed latch and containerHasState() — returns immediately when the
container already holds tenant state, and otherwise throws
RestoreDeferredToBootJobError. ensureGateway catches it, calls
requestDurableBoot(), and throws GatewayBootPendingError, so the proxy/route
answers 503 (or the loading page) while the DO alarm performs the restore +
launch. Durable callers (the ActivityScheduler boot/wake jobs and the queue
consumer) pass restoreOwner: 'durable' and run it inline. The proxy no longer
calls restoreIfNeeded itself. Motivation: on tenant-tarjetero-dev a 350 MB
encrypted archive was re-decrypted and sandbox.restoreBackup aborted at
~30.1 s (backup.restore error … The operation was aborted) on every cold
request, because the restore ran under a request execution context that
Cloudflare cancels shortly after the response — a reboot/restore loop while the
tenant was actively used.
Three supporting bounds keep a stalled boot from hiding behind the long restore
phase: the SDK sandbox.restoreBackup RPC is bounded by
RESTORE_BACKUP_TIMEOUT_MS (fail fast on a hung control channel); the restore
reports sub-phase progress (onProgress) so isGatewayStartStuck judges the
restore phase on GATEWAY_RESTORE_NO_PROGRESS_MS of no progress rather
than the full 11-min budget; and the persistence withLease releases the shared
R2 lease when its enclosing request aborts (PersistenceDiagnosticContext.signal),
so a cancelled operation cannot leave restore-needed + a held backup-lock
re-triggering the loop.
Snapshot lease renewal and admin snapshot mode. The shared R2
backup-lock TTL is STEP_DEADLINE_MS (12 minutes), with a heartbeat every
quarter-TTL (3 minutes). Snapshot work explicitly renews before/after legacy
staging, sandbox.createBackup, and encryption, and before final config capture;
restore renews around decrypt and archive application. Renewals are serialized on
one promise chain, awaited before the release tombstone, and best-effort (a failed
renewal is logged but does not abort the snapshot). Structured events report
renewed, late (at least half a TTL), and failed renewals. A 12-minute TTL trades a
longer stale-holder recovery window for avoiding false lease expiry during a
legitimate long step; lock waiters can still hit their bounded acquisition timeout.
POST /api/admin/storage/sync uses the durable backup job by default when the
ACTIVITY binding is configured (202 with jobId, 409 if another backup/restore
owns the slot). ?inline=1 explicitly requests synchronous completion; ?async=1
keeps the accepted-then-background behavior. Environments without ACTIVITY retain
the inline fallback. E2E scenarios that need an immediate backup id must use
?inline=1.
Never call output()/runShell on the live gateway handle. The gateway never
exits, so the SDK’s buffered output() only opens an RPC log stream, waits for
the local timeout, and disconnects it, surfacing workerd’s ReadableStream received over RPC disconnected prematurely. Read diagnostics from the
/tmp/gateway-stdout.log / /tmp/startup.log tails (readLogTail) instead.
The setup/post-start phases must stay child-free. The Worker waits on
them with waitForExit, which — despite its name — only settles once the
supervised terminal exits, which never happens while the script leaves a live
background child. setup is awaited on the critical path; post-start is
dispatched detached but still waited with a bounded waitForExit (a lingering
child would keep the post-start process alive until the
GATEWAY_POST_START_TIMEOUT_MS kill). So neither phase may start a tail -f log
mirror or background cloudflared: the mirror is started by the native launch
(buildGatewayRunScript), and cloudflared is detached with setsid. A lingering
child in setup stalls cold start for the full STARTUP_TIMEOUT_MS and makes the
Worker re-run setup on retry (observed: ~11 min to first gateway launch).
post-start never blocks readiness. Once the port wait succeeds and
markGatewayReady runs, the gateway is serving; ensureGatewayImpl dispatches
start-openclaw.sh post-start with void runGatewayPostStart(...) instead of
awaiting it. Dev instances run openclaw automations list/create there (30–60s),
and awaiting it previously held the shared start promise so every request logged
Gateway start already in progress … waiting... while /health already returned 200. startShell dispatches the sandbox.exec synchronously before its first
await, so the work still runs server-side even if the isolate is evicted;
runGatewayPostStart swallows its own errors (best-effort).
The entrypoint runs sequentially:
1. Fast-path check: Is gateway already running? → exit 0 (skipped for `post-start`)
2. Serialize concurrent startup/setup/full invocations (exclusive `flock` on
/tmp/openclaw-setup.lock, ~900s wait). The Worker's in-process
`gatewayStartPromises` guard is per-isolate, so bursty cold-start polling can
land several runs in different isolates at once; without the lock they
all run the slow setup body concurrently, contend on OpenClaw's native
config/state locks, and stall cold start for many minutes (observed: ~18 min
to gateway-ready in run 36692775633). Only one body runs; a waiter re-checks
the gateway fast-path after acquiring and exits 0 once the winner is up.
In production, `start` takes this lock **before** the setup body and holds it
through the Gateway `exec`; `post-start` is outside the lock (it runs after
the gateway is reachable). The standalone `launch` mode has its own dedicated
launch lock and must not double-lock. `full` releases fd 9 before
backgrounding the gateway so the child does not inherit the setup lock.
3. Worker-side lifecycle coordination
- Restore, snapshot, restart, and destroy are coordinated by the Worker.
- OpenClaw's native lock protects gateway ownership; the startup `flock`
above serializes onboarding/config/plugin filesystem mutations **and** the
launch across concurrent Worker isolates — `start` holds it from before
setup until the gateway is `exec`'d, so a single owner covers both. The
standalone `setup`/`launch` smoke modes still split the lock (setup lock
released before the separate launch lock), so within that path single-owner
launch is enforced by the dedicated launch lock alone.
4. Pre-backup optimization:
- Move logs to /tmp (excluded from squashfs snapshots)
- Clean stale backup files
5. Onboard (if needed):
- Detect Cloudflare AI Gateway auth from env vars
- Ensure the `cloudflare-ai-gateway` provider plugin is present before
onboard. It is **pre-installed into the image at build time** (see
Dockerfile), so this is normally a no-op detection; the install is a
resilience fallback. Non-interactive onboarding cannot approve plugin
capabilities, so without the plugin its plugin phase fails and burns the
60s onboard cap on every cold start. With it present, the phase is a no-op
and onboarding completes cleanly.
- Run: openclaw onboard --non-interactive --accept-risk
6. Enable plugins (single working `config patch`):
- All three plugins are **pre-installed into the image at build time**
(Dockerfile, unpinned → latest ClawHub release). The installs below run
only as a fallback when the extension directory is missing (e.g. a warm R2
restore replaced state). Detection (`plugin_extension_present`) checks that
directory — the artifact the gateway actually loads — and deliberately does
**not** read `openclaw plugins list --json`: the registry refresh hangs for
its full `CLI_TIMEOUT` when ClawHub/npm are slow, and a hung/partial list
reads as "not installed" and triggers a needless network install (~57s
ai-gateway + ~74s otel on a full cold boot on `staging-client-dev`). See
`docs/plans/cold-start-latency.md`.
- **Build/boot guard:** every ClawHub install is best-effort. A ClawHub
outage, a transient network error, or a plugin that requires a newer plugin
API than the pinned OpenClaw runtime (observed: `@openclaw/cloudflare-ai-gateway-provider`
resolving to 2026.9.9 against the 2026.9.8 runtime) logs a `WARNING` and
continues — the Docker build no longer fails, and the boot proceeds. The
enablement patch below only enables a plugin whose extension directory is
present, so a failed install cannot invalidate the whole patch.
- cloudflare-unified-billing (when installed) — the Paso4 plugin, served from
https://clawhub.ai/paso4/plugins/cloudflare-unified-billing. Entry opts
into `hooks.allowConversationAccess=true` so the
`agent_end`/`message_received` hooks run (OpenClaw 2026.8.2 blocks
non-bundled conversation hooks without it)
- cloudflare-ai-gateway (when installed)
- admin-http-rpc (always) — the Worker's RPC-first admin read path
- diagnostics-otel (enabled only when `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`
is set and not in E2E mode; otherwise explicitly disabled). Exporter
protocol is always `http/json`. When `WORKER_URL` is set (and
not in E2E mode) `buildEnvVars()` also derives
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT={WORKER_URL}/api/otel/v1/traces`.
`diagnostics.otel.traces` is opt-in via `OTEL_TRACES_ENABLED=true`
(default: disabled); the config shape is unit-tested in
`src/gateway/diagnostics-otel.ts` (`buildDiagnosticsOtelConfig`).
- The patch must NOT carry `plugins.bundledDiscovery` (removed in OpenClaw
2026.9.x — including it fails schema validation and drops the whole patch)
nor `plugins.allow` (an allowlist disables every bundled provider not
listed). Memory search is disabled via `memory.search.enabled=false`.
- **memory-core dreaming cadence (dev):** the same patch slows the dreaming
sweep to weekly (`plugins.entries.memory-core.config.dreaming.frequency =
"0 3 * * 1"`, default `0 3 * * *`) when `OPENCLAW_DEV_MODE=true`. The full
sweep is a heavy isolated agent turn (~6 min, `toolsAllow: ["*"]`) that the
cron replays on every wake (`cron: running missed jobs after restart`) — a
major RSS spike source on the 1-vCPU dev sandbox (observed 1.6 → 2.08 GiB
during a replayed sweep). Production keeps the OpenClaw default cadence.
- **Version drift:** the plugin-API mismatch above means a build on the
pinned runtime ships **without** the Cloudflare provider plugins until
OpenClaw is bumped (see the openclaw-migration skill) or the specs are
pinned. The guard keeps the image and boot healthy in the meantime; it does
not restore the missing provider by itself.
7. Patch config via moltlazy:
- Run: node /app/moltlazy/dist/cli.js patch
- Generates per-section includes (moltlazy-session/tools/logging.json) injected
via per-section $include; gateway/agents written inline (no root $include)
- References the gateway token as an env SecretRef (kept in the container environment)
- Run: node /app/moltlazy/dist/cli.js validate
- **Validation is FATAL**: an invalid config would prevent the gateway from
booting, so the script aborts (exit 1) instead of proceeding.
8. Harden file permissions:
- chmod 600 moltlazy-*.json, chmod 700 .openclaw
9. Cloudflare Tunnel (optional):
- Start cloudflared for obsidian-sync (port 18790), detached with `setsid`
(stdout/stderr → `/tmp/cloudflared.log`) so `setup` stays child-free
10. Start gateway:
- `start` mode (production): after setup, `exec` the gateway under the same
startup lock the phase acquired at step 2 — the Worker no longer calls
`launch` separately (its stdout log mirror / `post-start` run as usual)
- `setup` mode: smoke/manual-only standalone path — `mark_step gateway`,
then exit 0 without launching the Gateway. The production Worker uses the
atomic `start-openclaw.sh start`, which holds the startup lock and `exec`s
the Gateway, followed by `post-start`
- `full` mode: background `openclaw gateway run --port 18789 --verbose
--allow-unconfigured --bind lan`, poll `/health`, then exit while the
gateway child remains detachedStartup Progress (/api/startup-progress)
The loading page shows a transparent boot stepper + live log while the gateway
starts. start-openclaw.sh declares its boot phases in a single STEPS array
(single source of truth) and emits progress to two /tmp files on each phase
(kept in /tmp so they stay out of squashfs backups and R2):
| File | Content |
|---|---|
/tmp/startup-progress.json | { steps:[{id,label}], currentStep, currentStepId, totalSteps, skipped, ts } |
/tmp/startup.log | Clean, secret-free, chronological boot detail (verbosity-gated) |
mark_step <id> [skipped] resolves the label/index from STEPS, writes the
progress JSON atomically (mktemp + mv), and echoes the transition to the
container stdout. log_boot <level> <msg> writes clean boot detail to
/tmp/startup.log. Step transitions are intentionally not written to the
log — the loading page’s stepper shows step progress, so the log stays clean
(no duplicated step/checked info). Adding a phase = one STEPS row + one
mark_step call — the loading page renders the list dynamically, so no other
code changes.
GET /api/startup-progress (public, no warm-up) reads both files and returns
{ available, steps, currentStep, currentStepId, totalSteps, skipped, ts, log }.
It is bounded by STARTUP_PROGRESS_TIMEOUT_MS and never calls
restoreIfNeeded/ensureGateway, so it adds no load to the health path.
The loading page consumes this data over the GET /api/startup/ws WebSocket
(not by polling). Each push is a combined { ...HealthSnapshot, progress }
payload — the health snapshot (which drives readiness/redirect) plus the boot
steps + log. On socket open the endpoint registers one ensureGateway warm-up
with waitUntil; every subsequent health snapshot is passive/read-only. It
does not re-enter restoreIfNeeded on every push. This distinction matters when
restore is blocked on an R2 backup-lock: the former per-push warm-up queued a
new 120-second restore-lock waiter every 5–30 seconds, which could starve gateway
startup and the pre-replacement checkpoint while the UI kept its socket open.
The loading page shows the boot log panel while the gateway is starting up —
both while start-openclaw.sh is running its setup body (processType === 'script') and while the gateway binary is binding its port (processType === 'gateway'). During the earlier restore phase (no process yet) the log is empty,
so the page shows the FourIA logo instead. The processType === 'gateway' state
no longer hides the log: start-openclaw.sh start execs the gateway and a
start_boot_log_stream mirror tails the gateway’s own stdout
(/tmp/gateway-stdout.log) into the curated boot log, so the log keeps filling
with real launch progress (SQLite migration, plugin load, port bind) instead of
going blank for the minutes the gateway takes to bind — the exact “Launching the
gateway — no progress” window that previously looked stuck. The mirror is killed
(stop_boot_log_stream) and the buffer cleared once the gateway is ready.
While a cold restore is in flight, the page also shows a restoring spinner
beside the status line (title “Restoring FourIA” / status “Restoring data from
backup…”). It is driven by the health snapshot’s in-memory restoreStatus
(hadBackup && !restored) — the same signal the startup.ok probe uses to
report “Backup found but not restored” — or the durable recovery.restoring
marker (below), and is cleared the moment the restore commits (or the start
fails/redirects). It is intentionally not keyed on the R2 restore-needed
latch, so the spinner shows for a restore that would have run anyway (a fresh
container with a committed recovery point).
The page also names the backup + date being loaded, so a cold restore is
legible instead of a generic spinner. /api/health (non-passive, not-yet-ready)
and /api/startup/ws attach a durable recovery object read from R2
(readRecoveryState): { restoring: { id, created } | null, committed: { id, created } | null }. restoring is the in-flight restore-target.json written
by restoreIfNeeded/restoreBackupById just before sandbox.restoreBackup and
cleared on success; committed is the stable-backup.json recovery point a cold
boot restores. It is read from R2 (not the per-isolate restoreStatus) because
the durable boot/restore runs in the ActivityScheduler DO, whose in-memory
state is invisible to the request isolate serving the page. Lerma’s frequent
passive poll never pays the two R2 reads; they are bounded by
RECOVERY_STATE_TIMEOUT_MS and fail soft to null.
During a restore the loading UI is a progress bar synchronized with the DO’s
own phase events (not a client-side timer). restoreIfNeeded /
restoreBackupById call their onProgress(phase) hook on every
persistence.phase; the DO wires that to recordRestoreProgress, which maps the
phase to a label + a monotonic fraction (describeRestoreProgress, in
activity/restore-progress.ts) and exposes it via getRestoreProgress().
/api/startup/ws pushes it as restoreProgress: { phase, label, fraction };
loading.html renders the bar and hides the stepper while a restore is in
flight, then switches back to the container’s startup-progress.json stepper
once the gateway launch begins. A request-scoped restore with no DO feed shows an
indeterminate bar. /api/admin/boot-state also returns restoreProgress for
operators.
The label stays honest when a restore is interrupted. A failed attempt is marked
before the durable job rethrows (recordRestoreFailure), so the bar reads
Restore attempt failed during "<phase>" — retrying instead of the last phase.
A kill that skips that catch (e.g. a Worker CPU-limit kill) leaves the last
durable phase in place, so getRestoreProgress() appends
— no progress for N min once the record is idle for 2 minutes. The fraction
never moves, so a stale bar is labelled rather than presented as live.
Archive decryption (decryptStream / encryptStream in
crypto/backup-encryption.ts) assembles each 4 MiB chunk from R2 reads with a
byte queue that copies each byte once. The earlier concat-per-read buffer
recopied the pending chunk on every ~64 KB read, which exhausted the Worker CPU
budget on large recovery archives (Worker exceeded CPU time limit during
Decrypting backup …). Measured in isolation on a 96 MiB archive: decrypt
1507 ms → 140 ms at 64 KB reads, and 5614 ms → 137 ms at 16 KB reads.
The R2 binding delivers the archive in 4 KiB reads (local workerd: 64 MiB
arrives as 16,385 reads), so the old per-read concat was the dominant cost.
Each restore logs [crypto] Decrypt read profile with the read count, bytes,
largest and average read, and chunk count, so the read pattern can be checked
on a live tenant.
The loading page uses a single push channel — GET /api/startup/ws — and
does not poll while the WebSocket is open. Flow:
- Handshake — the first WS push reports whether the gateway is already running; if so the page redirects immediately, otherwise the stream keeps warming and pushing step updates.
- Boot phase — while boot steps are in progress, the sandbox pushes the
health snapshot +
progress(steps + log) everySTARTUP_WS_POLL_INTERVAL_MS(~5s). No client polling. If a transient DO↔container transport drop makes the probe fail, the stream re-sends the last known state and backs off exponentially up toSTARTUP_WS_MAX_BACKOFF_MS(30s) until the control session reconnects, so the page keeps waiting and the same error is not surfaced/re-logged every 5s. - Fallback — if the WebSocket drops (e.g. local dev server, a network
blip), the page falls back to slow HTTP polling (
/api/healthevery 5s) indefinitely. The page has no wall-clock budget or retry cap of its own: the cold boot is driven out-of-band by theActivitySchedulerDO, which owns retries and terminal failure. A run of failed polls does not hard-fail the page (“Unable to reach server”): a container-image / Worker deploy briefly resets the DO↔container control session, so the page keeps retrying and shows “Reconnecting — the instance may be updating…”.
The page only declares failure on a permanent container error
(degraded: true in the /api/health snapshot — deleted image, missing app,
where a rescue is required), surfaced immediately via the container error, or on
the durable boot job’s terminal boot.status === 'failed' state. Every other
state — still booting, gateway reachable but warming up, restore in progress —
keeps waiting, because a cold start legitimately takes 3–5+ minutes (onboard +
gateway warmup; the provider plugins are pre-installed in the image, so plugin
installs only run on the fallback path). Mixed probe states (e.g. startup.ok
lagging behind readiness.ok while a backup restore is pending) are not
treated as failures, and there is no client-side timeout that interrupts a
long-but-progressing boot.
The loading page treats the canonical gateway status (status === 'running',
driven by the readiness probe) as “ready” rather than trusting startup.ok
alone, because startup.ok is gated on backup-restore state that can lag a
healthy, already-serving gateway.
A slow boot step is surfaced as an alert. Once the current boot step has run
longer than SLOW_STEP_MS (120s), the loading page shows a non-fatal notice
naming the step and its elapsed time (e.g. “Still working on “Installing
plugins” (185s)…“), driven by the script’s own progress.ts. It is not a
failure — cold starts legitimately take minutes — and it is hidden on ready/error.
It exists so an operator can tell a slow-but-progressing boot from a stuck one
without opening logs or the Workers Observability console. (See
docs/plans/cold-start-latency.md.)
The Worker’s root HTML route (app.all('*') in src/index.ts) uses the same
ready predicate (snapshot.readiness.ok, i.e. status === 'running') — it must
not additionally require startup.ok. Requiring both made the two disagree
whenever readiness.ok was true but the startup probe lagged (a restore still
settling, a stale cached startup result, or a start-openclaw.sh start wrapper
not yet replaced by the gateway binary): the loading page redirected to /
while the root route re-served the loading page — an infinite reload loop
where OpenClaw never appeared to boot (observed on tenant-client-dev). Keep the
two predicates in lockstep.
The gateway WebSocket proxy gates on the same readiness predicate. The
catch-all WS path in src/index.ts must call isGatewayReadyForProxy() before
sandbox.wsConnect(). wsConnect (and containerFetch) wait for their target
port inside the Sandbox DO for containerTimeouts.portReadyTimeoutMS
(CONTAINER_PORT_TIMEOUT_MS, 90s). During a container-image rollout the
container is healthy — Cloudflare reports it active and the container logs are
clean — but the gateway has not bound 18789 yet, so that one RPC parks for the
full 90s and every other RPC (health, components, other proxies) queues behind
it, surfacing as “Unable to reach server” on the loading page. When the gateway
is not ready the WS path returns a fast, retriable 503 + Retry-After: 5
(and fires ensureGateway({ waitForReady: true }) via waitUntil so the cold
start still progresses) instead of calling wsConnect. The HTML and non-HTML
HTTP proxy paths already gate the same way.
A "Timed out waiting for runtime port" error — the DO’s port wait expiring
because the container is up but the service has not begun listening — is
classified by isRuntimePortTimeoutError() (src/gateway/sandbox-recovery.ts)
as transient: the WS path returns a retriable 503, never a 502, and it
must not trigger killGateway/stale-container recovery, which would discard
a healthy container mid-rollout. It is deliberately excluded from
isStaleContainerError.
Likewise, Network connection lost, WebSocket connection failed, and the
Worker’s bounded Operation timed out after … from sandbox.wsConnect() are
transient Sandbox DO WebSocket-transport failures, not proof that the gateway
process died. The proxy returns 503 with Retry-After: 5 and leaves the
container and R2 state untouched. This matters when the admin HTTP RPC
health/status path still works: it uses containerFetch, a separate transport
from the browser’s WebSocket connection.
Security invariant: the boot log contains curated log_boot lines — the
script’s raw stdout (which can contain the AI Gateway API key as an openclaw onboard --cloudflare-ai-gateway-api-key … argument) must never be written to
/tmp/startup.log. In addition, once the gateway launches, a
start_boot_log_stream mirror tails the gateway’s own --verbose stdout
(/tmp/gateway-stdout.log) into the boot log so the loading page can follow
launch progress; this is the same output already surfaced to ADMIN/ROOT via
/debug/gateway-logs and /api/admin/logs, and it is stopped (stop_boot_log_stream)
and the buffer truncated at readiness. The gateway’s startup output is a distinct
stream from the script’s onboard stdout (which is where the API key appears as a
CLI argument and is still never captured).
Verbosity is controlled by STARTUP_LOG_VERBOSITY (steps | detailed (default) | verbose),
passed through from the Worker via buildEnvVars().
moltlazy patch — Config Injection
moltlazy patch is the mandatory config system. It runs every startup and generates:
| Config Section | Type | Content |
|---|---|---|
gateway.* | Mandatory | Port, mode, bind, auth |
session.* | Mandatory | Session scope, DM settings |
tools.toolSearch | Mandatory | Tool search configuration |
logging.* | Mandatory | Log levels, format, destinations |
agents.defaults | Always-on | Default model, system prompt |
agents.entries | Always-on | Premade agents keyed by id (dev-only fouria-builder, fouria-qa) |
skills.entries | Always-on | Built-in skills |
memory.* | Feature-gated | Memory backends (knowledge graph) |
channels.* | Env-gated | Channel tokens (Telegram, Discord, etc.) |
plugins.* | Feature-gated | Plugin management config |
The moltlazy-owned sections (session/tools/logging) are generated into per-section include files and injected into openclaw.json via per-section $include directives — not a root $include (root includes fail closed for Control UI saves / config.apply). They are regenerated from scratch every startup — never edit manually. gateway.* and agents.* are written inline into openclaw.json; the gateway token is an env SecretRef (resolved from the container environment, which start-openclaw.sh keeps set).
Admin RPC (admin-http-rpc) — RPC-first read path
The Worker reaches the gateway’s admin control plane over the gateway’s loopback
HTTP listener using OpenClaw’s bundled admin-http-rpc plugin
(POST /api/v1/admin/rpc). The typed client lives in packages/moltlazy/sdk
(createClient, HTTP RpcClient); apps/fouria/src/gateway/admin-rpc.ts
supplies the transport — a fetch-shaped adapter over sandbox.containerFetch
(global fetch cannot resolve the container’s localhost).
This replaced the CLI-first read path for the hot dashboard routes. CLI
invocations (startShell(sandbox, 'openclaw ...')) reap a fresh process and
re-establish a WebSocket on every call (~10–15s); the HTTP RPC path is a single
container request. Routes attempt RPC first and fall back to the CLI
(runCliFallback / the existing parsers) whenever the transport is unavailable
or returns an unexpected shape:
GET /api/admin/devices→device.pair.listPOST /api/admin/devices/:id/approve→device.pair.approveGET /api/admin/models→models.listGET /api/admin/channels/probe→channels.status { probe: true }
Enablement: start-openclaw.sh enables the plugin with a dedicated
plugins.entries["admin-http-rpc"].enabled = true config patch. It is kept
separate from the batched plugin patch because OpenClaw 2026.9.x rejects that
batch (plugins.bundledDiscovery was removed upstream), and setting
plugins.allow would disable every bundled provider that is not listed. Enablement
applies to a running gateway automatically; no restart.
Security: the plugin exposes a full operator control-plane surface on the
gateway listener. It is only reached in-process via sandbox.containerFetch;
src/index.ts shadows /api/v1/admin/rpc with a 404 before the public
catch-all gateway proxy, so it is never reachable through a tenant hostname.
Never remove that guard.
Cold-start guard (health probe). sandbox.containerFetch auto-starts the
container and blocks until the requested port is ready. The RPC-first health
probe (getRpcHealthSnapshot, src/gateway/health-probes.ts) therefore only
issues the RPC once a gateway process is actually detected; while the container
is cold, or the production start-openclaw.sh start wrapper is still running
setup, it returns null — RPC health is skipped until the local Gateway
health/port is open — so /api/health falls back to the exec-based
getHealthSnapshot (which never waits on the gateway port). Callers that already hold a process record pass it as
{ proc } to skip the extra scan — buildHealthSnapshot does this with the
findExistingGatewayProcess result it already fetched.
Cold-start boot lifetime. #318 split the boot so the Worker launched the
gateway after start-openclaw.sh setup exited. /api/health is bounded to a
5s snapshot, so the request returns long before the (multi-second) setup
finishes; workerd then cancels the unregistered ensureGateway continuation and
the container-side setup completes with no gateway ever launched (the
observed cold-start stall: 68 processes, setup exit 0, every curl /health exit
7). The fix was to keep the boot alive past the response, and the current runtime
contract collapses the production path into a single
/usr/local/bin/start-openclaw.sh start call (startup lock acquired before
setup and held through the Gateway exec) followed by post-start — the Worker
no longer invokes setup and launch separately. Any route that triggers a cold
boot must still keep it alive past the response with
c.executionCtx.waitUntil(ensureGateway(sandbox, env, { waitForReady: true }))
— waitForReady: true returns the shared start promise unwrapped (no read
budget) and dedupes onto the promise the bounded snapshot already started.
/api/health and /api/warmup do this; do not replace it with a bare
await ensureGateway(..., { waitForReady: false }).
The SDK’s WebSocket transport (createWsClient) is not used: the current
OpenClaw wire protocol (v4) requires a connect handshake and a paired
operator.admin device identity, which the shared-token loopback path does not
provide. HTTP RPC over the shared token is the least-effort RPC-first transport
and is what admin-rpc.ts implements.
RBAC & Access Control
Roles
| Role | Permissions | Access Scope |
|---|---|---|
| ROOT | admin:platform, admin:customer, api:all | All tenants, all debug endpoints |
| ADMIN | admin:customer, api:all | Own tenant only |
| BASE_USER | api:own | Own tenant, read-only |
Dashboard Pages by Role
| Page | ROOT | ADMIN | BASE_USER |
|---|---|---|---|
| Dashboard Home | Yes | Yes | Yes |
| Agents | Yes | Yes | Read-only |
| Connections | Yes | Yes | Read-only |
| Knowledge | Yes | Yes | Read-only |
| Account/Team | Yes | Yes | Read-only |
| Integrations | Yes | Yes | No |
| Secrets Vault | Yes | Yes | No |
| Backups | Yes | Yes | No |
| Diagnostics | Yes | No | No |
Debug (/debug/*) | Yes | No | No |
| Audit Logs | Yes | No | No |
ROOT is the maintainers’ intended role. It has full access to debugging tools, container inspection, and platform-level diagnostics at
/_admin/diagnosticand/debug/*. ROOT is restricted to diagnostics-only API paths (DIAGNOSTICS_PATHSinsrc/routes/api.ts) — management pages (Agents, Connections, Backups, etc.) are ADMIN-gated. As a diagnostic exception, ROOT can still verify backup connectivity (Test Backup Connectivityon the Diagnostic Dashboard →POST /api/admin/storage/test-connectivity) without the Backups page.
Workflow commit tree. The tenant’s workflow git history (commit tree, branches, tags, releases) is client data. It is served by
GET /api/admin/releases/commits, which is deliberately not inDIAGNOSTICS_PATHS: ROOT is blocked at the diagnostics gate and BASE_USER byrequireAdminOrRoot, leaving ADMIN-only access. It is rendered by theCommitTreecomponent on the ADMIN-gated Backups page. The tree is produced from the sandbox git repository by the release tooling; until that substrate ships the endpoint returns an empty tree.
In a harness/dev mode (
DEV_MODEorE2E_TEST_MODE),/debug/*additionally permitsADMINso the automated E2E suite can drive CLI/operator routes from a bareworkers.devworker that has no Cloudflare Access or Dispatch Worker in front of it. Real production (DEV_MODE=false,E2E_TEST_MODE=false) stays ROOT-only per the table above.
Agent Push (dev → production promotion)
A tenant owns two instances: its dev environment builds and validates agents
(the dev-only fouria-builder / fouria-qa personas), and its production
environment serves them. Agent push promotes the built content from dev to
production.
The promoted artifact is the full promotable manifest, owned by
@moltlazy (packages/moltlazy/src/promotion/index.ts): openclaw.json, the
moltlazy per-section $include files, and the agent workspace/. The dev-only
agents and all runtime state — state/, credentials/, agents/, sessions/,
secrets/ — are never promoted. Provider/channel credentials carried inside
openclaw.json do travel; only the runtime secret stores are excluded.
The transport is HTTP, Worker → Worker, and the production instance applies
the release as a custom backup: it keeps its own runtime state (sessions,
transcripts, the control-plane SQLite) while overwriting config / agents / tools
and re-applying the gateway section for its own WORKER_URL, then soft-reloads
the gateway. The reload is off the response path, so neither dashboard is
blocked.
Routes (src/routes/api/releases.ts, src/services/promotion/):
| Route | Instance | Behavior |
|---|---|---|
GET /api/admin/releases/commits | both | Workflow commit tree (client data; ADMIN-only, not a ROOT diagnostic). |
GET /api/admin/releases/status | production | The current release pointer (staged/applied + agents). |
POST /api/admin/releases/promote | dev | Export a sanitized bundle of the promotable manifest and POST it to <targetUrl>/api/admin/releases/import. Refuses a non-http target and a self-target. |
POST /api/admin/releases/import | production | Receive + stage a bundle, acknowledge 202, then apply + checkpoint in executionCtx.waitUntil. Requires the PROMOTION_PUSH_TOKEN bearer when that binding is set. |
POST /api/admin/releases/apply | production | Apply the staged release now (idempotent), checkpoint it as a recovery point, and reload the gateway. |
Export builds a gzip bundle: the config files + workspace are staged in the
container (dev-agent workspaces skipped), openclaw.json is sanitized (dev-only
agent entries removed, ambient-owner pointers repaired), then archived and
streamed out with sandbox.readFileStream. Import writes the stream with
sandbox.writeFile, extracts over /home/openclaw (state untouched — it is not
in the archive), re-sanitizes, runs
moltlazy patch --only gateway (restored-boot self-heal), and soft-reloads.
src/services/promotion/index.ts keeps its own copy of the exclusion list so the
Worker bundle never depends on the moltlazy config module; a unit test pins both
lists together.
E2E: apps/fouria/test/e2e-push/ (local-only;
boots a dev + production Worker pair and pushes over HTTP).
Auth Flow
- Cloudflare Access intercepts request → user authenticates (email OTP, Google, GitHub)
- Access injects
Cf-Access-Jwt-Assertionheader - Worker verifies JWT against Cloudflare Access JWKS endpoint
resolveCustomerRole()resolves the RBAC role — Paso4 ROOT users (platform Access AUD) are alwaysROOT; client users get their lerma-assigned role (ADMIN/BASE_USER) forwarded by the Dispatch Worker viaX-FourIA-RoleuserRoleandcustomerSlugare set in Hono context- Route handlers check role before serving content
Dev-instance role preview (ROOT → ADMIN)
On a dev instance — a -dev hostname, or an explicit OPENCLAW_DEV_MODE
override (see isViewAsEnabled in src/dev-mode.ts) — a Paso4 ROOT user may
view and perform as a lower role from the dashboard’s “View as” switcher:
the X-Mock-Role cookie (POST /api/admin/dev/set-mock) downgrades a resolved
ROOT role to ADMIN (or BASE_USER) for that session. This lets support/ops
inspect and drive a tenant’s dev dashboard as the client’s ADMIN without the
production diagnostics-only restriction.
The preview is a strict downgrade and dev-only:
resolveCustomerRole()only honours the cookie when the role it already resolved isROOT(src/auth/middleware.ts). A non-ROOT user setting the cookie cannot escalate — e.g.ADMIN→ROOTis ignored.POST /api/admin/dev/set-mockreturns404outsideDEV_MODE/E2E_TEST_MODE/ dev instances, and403for non-ROOT callers on a dev instance.- The tenant slug is pinned to the instance’s own tenant;
X-Mock-Customeris ignored in preview mode, so the preview can never cross tenants. - Production instances are unaffected —
canViewAsstays false and ROOT keeps its diagnostics-only scope.
GET /api/admin/whoami reports canViewAs: true for a ROOT user on a dev
instance so the SPA renders the switcher (DevUserSelectorCompact); local
DEV_MODE keeps the existing full mock-role switcher (devMode).
Billing Freeze Gate (F4)
Durable per-customer billing_frozen flag (src/billing-freeze.ts). Frozen blocks
new billable actions only — in-flight actions complete, basic non-billable service
continues. This is distinct from tenant suspension (Dispatch/Lerma-level), which
denies everything.
- Durability: persisted per-customer as
billing-freeze/<slug>.jsoninBACKUP_BUCKET(R2 — survives cold starts; never in-memory only). No CustomerRegistry DO exists. - Lerma → Fouria: Dispatch Worker propagates
billing_frozeninsideX-FourIA-Metadata(shared Postgres is the source of truth; Fouria never calls Lerma directly). Effective freeze = stored flag OR header flag (fail closed). - Fouria → Lerma:
GET /api/admin/billing/freeze-status(ROOT-readable diagnostics path) plus the R2 object itself. - Writes:
POST /api/admin/billing/freeze|/unfreeze(ADMIN-only; ROOT is blocked by the diagnostics-only gate). - freezeGate(env, customer) is implemented in src/billable-actions.ts; it reads the durable per-customer R2 freeze record (no request headers available, fails open on read error); enforcement happens at emit time inside emitBillableActions, which filters billing-frozen customers’ events before writing Analytics Engine and enqueueing to BILLING_QUEUE, so in-flight work completes and basic service continues.
Development Tips
Debugging the Container
# Get container ID from Worker logs
docker ps | grep sandbox
# Enter container shell
docker exec -it <container_id> bash
# Inside container: check openclaw status
openclaw status
openclaw devices list --json
openclaw config get gateway.port
# Check logs
tail -f /tmp/openclaw-logs/*.logUnderstanding Boot Logs
During cold start, these log patterns are normal:
[Gateway] listProcesses returned 0 processes
[Gateway] Starting new OpenClaw gateway...
[ERROR] Uncaught ProcessReadyTimeoutError: Waiting for: port 18789 (TCP)
The gateway takes up to 3 minutes to become ready. The Worker retries automatically. If errors persist beyond 3 minutes:
- Check
docker logs <container_id>for OpenClaw errors - Verify CF AI Gateway config (
CLOUDFLARE_AI_GATEWAY_API_KEY+CLOUDFLARE_API_TOKEN+CF_AI_GATEWAY_ACCOUNT_ID+CF_AI_GATEWAY_GATEWAY_ID) - Run
bun run dev:workerto see full Worker output
Service Integrations (Cloudflare Tunnel)
The container can run Cloudflare Tunnels for exposing internal services:
# Obsidian sync on port 18790
# Requires CLOUDFLARE_TUNNEL_TOKEN secret
# start-openclaw.sh auto-starts cloudflared if token is setCRON and Scheduled Tasks
The Worker runs one scheduled handler against two crons (see
wrangler.jsonc → triggers.crons). The handler branches on controller.cron
(CRON_BILLABLE_DRAIN / CRON_OPENCODE_CREDITS in src/config.ts):
| Cron | Job |
|---|---|
*/2 * * * * | Drain buffered billable OTel traces to BILLING_QUEUE (retry safety net for the sync emit). |
0 */8 * * 1-5 | Enqueue the OpenCode credits check onto TASKS (the queue consumer runs it in the sandbox). |
The credits enqueue is deliberately not on the frequent tick — otherwise the
OpenCode credits notification would fire every couple of minutes. Staging and
production keep only the weekday 8-hourly cron; the frequent drain is a
local/E2E aid because the synchronous /api/otel/v1/traces emit already covers
the normal path.
Containers sleep per a derived idle timeout (staging 15m, dev 2h — the Sandbox SDK 1.0
setInactivityTimeout() platform cap). The billable production environment does not
auto-sleep; it is stopped only by an explicit lerma freeze (POST /control →
sandbox.destroy()), so a warmed gateway never incurs a cold-start/downtime window after
an idle period. There is no SANDBOX_SLEEP_AFTER binding; resolveSleepAfter() derives
the value from the instance’s own WORKER_URL (src/dev-mode.ts), the same source the
dev/billable axes use. Waking is driven by the per-tenant ActivityScheduler Durable Object
alarm, which fires ahead of the next automation and then runs an interval-gated
R2 checkpoint.
The container’s idle-sleep decision is owned by the Worker’s Sandbox Durable
Object class, TenantSandbox (src/sandbox-do.ts), exported under the DO
binding name Sandbox (wrangler.jsonc → class_name: "Sandbox"). It extends
the SDK’s legacy Sandbox and overrides onActivityExpired(): on a non-billable
instance the hook ignores the persistent gateway (its lock wrapper, the gateway
binary, and the tail -f stdout mirror) and stops the container once only that
persistent work remains — matching Sandbox SDK 1.0, where code inside the instance
does not keep it alive. Any other running process (an in-flight CLI/backup
command) still counts as activity and defers to the SDK gate. On the billable
production environment the hook defers to the SDK’s keepAlive gate instead,
so the container never stops itself.
When a non-billable instance is due to idle-sleep, the hook checks the R2
catalog first: a committed snapshot inside the idle window already captures
the unchanged filesystem (the container has had no activity for the whole
window), so the hook stops inline through stopIfIdle() without re-archiving.
That shortcut exists because the previous unconditional snapshot was itself
renewing the DO activity window, so the stop-time re-check refused and the next
idle expiry snapshotted again — a loop of 422 MB archives that kept
tenant-tarjetero-dev awake. Otherwise (no recent snapshot, no BACKUP_BUCKET,
or a catalog read failure) it enqueues a durable deprovision job (mode
sleep) on the ActivityScheduler DO, which snapshots the tenant and only then
calls sandbox.stopIfIdle() — fail closed, so a container is never slept without
a committed archive. Without the ACTIVITY binding (local/dev) it falls back to
stopping inline. See Backup before deprovisioning below.
Lerma’s HealthMonitor probes the passive GET /api/health?passive=1 every
few minutes (authenticated at the Cloudflare edge with a service token — see the
path-scoped public-health Access applications in apps/fouria/iac/access.tf).
Passive probes never run the exec RPCs that would cold-start a stopped
container: getHealthSnapshot first does the non-waking isRuntimeActive()
read and reports status: "sleeping" (HTTP 200, healthy-idle) for an idle
container. Health probes are observability-only and never schedule a backup.
The container runs scheduled tasks via OpenClaw’s cron system.
Task Buffer (Cloudflare Queues)
The Worker is a durable buffer for OpenClaw-side jobs (src/tasks/): it never
implements the job logic itself.
- Producer —
scheduledenqueues the recurring OpenCode credits check ontoTASKS;POST /api/admin/tasks/opencode-credits-check/enqueueis the manual trigger (ADMIN/ROOT). - Consumer —
default.queuewakes/heals the sandbox viaensureGateway, runsopenclaw automations run opencode-credits-checkinside it, then acks (Cloudflare Queues own retries/backoff;max_retries: 5). POST /api/admin/tasks/opencode-credits-checkis the manual consumer trigger (runs the automation directly, bypassing the queue). It clamps its total wall-clock toTASK_TOTAL_TIMEOUT_MS(90s) — the gateway wake, exec launch, CLI wait, and log recovery are each clamped to the remaining budget — so a cold/contended boot always answers within the endpoint’s ceiling instead of outliving the caller (the e2e suite’scurl --max-timehit 120s exactly on cold runs). The shared gateway start promise keeps booting in the background.- Automation definition —
start-openclaw.sh’sensure_opencode_credits_automation(dev-only) creates the recurringopencode-credits-checkcron job; the Worker only runs it. The guard queriesopenclaw cron list --all --jsonin addition to the legacyautomations list --json, which missed a prior boot’s job and produced two duplicateopencode-credits-checkjobs on a dev instance. - Heartbeat cadence —
start-openclaw.sh’sensure_dev_heartbeat_cadence(dev-only) reconciles theheartbeat-fouria-builderjob on every post-start. OpenClaw 2026.10.5 retiredagents.entries.*.heartbeat: a heartbeat is now an ordinary automation job, and config reloads/restarts never rewrite an existing job, so the instance created by the migration kept the default 30-minute cadence even though moltlazy declaresheartbeat.every: 12h(observed ontenant-tarjetero-dev: 30-minute heartbeats for 9 days, ~317k tokens each on the main session, until the OpenCode subscription hit 0%). The reconciler readsopenclaw automations get <id> --json, comparesschedule.everyMs/sessionTarget/payload.lightContext, and only then runsautomations edit --every 12h --session isolated --light-context(split so a rejected--light-contextcannot lose the cadence fix). It never resets the schedule anchor when the job already matches.
Telegram Webhook Ingress (#364)
Managed tenants switch Telegram to webhook mode so an inbound update wakes a
sleeping container (long polling runs inside OpenClaw and cannot wake it). The
Worker is the always-on ingress (src/telegram-ingress/):
- Route —
POST /telegram-webhook/:accountis mounted before Cloudflare Access (secret-auth, not JWT). It resolves the account’s config from a Worker-readable F4E1 registry, constant-time validatesX-Telegram-Bot-Api-Secret-Token, and rate-limits — all before any container wake, so a rejected request never starts or renews the container. - Registry —
telegram-ingress/<slug>/<account>inBACKUP_BUCKET(F4E1-encrypted,BACKUP_ENCRYPTION_KEY). Holds the per-account webhook secret- bot token, readable by the Worker without the container running. Populated
by
POST /api/admin/telegram-ingress/:account(multi-bot).
- bot token, readable by the Worker without the container running. Populated
by
- Hot path — when the gateway is running and ready, the update is forwarded
straight to OpenClaw’s webhook route (port 18789) and its
x-openclaw-delivery-acceptedresponse is passed back to Telegram. - Cold path (“instance is not in hot mode”) — the Worker notifies the sender
via
sendMessage(“waking up…”), enqueues the update on theTELEGRAM_INGRESS_QUEUE(Cloudflare Queue), acks Telegram (the message is now durable), and kicks the cold start. The queue consumer wakes/heals the gateway, waits for readiness, and forwards — acking only after OpenClaw confirms receipt, so a message is never “popped” until it has arrived (at-least-once, mirroring Telegram’s own delivery).max_retries: 10, retry_delay: 60scovers a multi-minute cold boot. - Freeze policy — enforced upstream by the Dispatch Worker: a
frozen/operator-stopped instance is never woken by a Telegram update (no user
role → always non-ROOT); it gets an explicit
503and the message stays in Telegram’s retry queue. - Dispatch ingress — a dedicated host (
hooks.fouria.io/hooks.dev.fouria.io,TELEGRAM_INGRESS_HOST) excluded from the wildcard*.fouria.ioAccess app. It resolves/v1/<slug>/<account>, applies the freeze gate, and forwards (rewritten toPOST /telegram-webhook/:account) with no Access JWT. - moltlazy —
start-openclaw.shpasses the Worker-injectedTELEGRAM_WEBHOOK_MANIFESTto moltlazy, which writeschannels.telegram.accounts.<account>.webhookUrl/webhookSecret/webhookPath/legacyWebhook(fill-only, preserving the user’sbotToken/dmPolicy). - Queue observability — the per-tenant
TelegramIngressDO records every ingress event (bounded counters + the last 20):queued(a cold update was enqueued and woke the container),forwarded(hot path),delivered(queue consumer confirmed receipt),failed(a delivery attempt failed and will retry).GET /api/admin/telegram-ingress/statusreturns the snapshot, andGET /api/admin/telegram-ingress/wsstreams it (a frame per change) for the dashboard. The admin shell’sTelegramIngressProvideropens that socket only while the feature is in use, raises a notification-bell entry (“Telegram message queued — waking your instance”, or a delivery-failure alert), and feeds the Telegram Queue widget on the home page (state + queued/delivered/failed- last event), which is itself hidden until the ingress is in use.
Backup and Hard-Restart Policy
Backups are snapshots of /home/openclaw, not a live filesystem journal. Agent
workspaces are therefore kept under /home/openclaw/workspace; files outside
that tree are not protected by the Sandbox snapshot.
The Sandbox SDK’s createBackup returns a handle marked with localBucket: true
when local-bucket mode is enabled (persistence.ts). That mode is an explicit
BACKUP_LOCAL_BUCKET=true opt-in (falling back to DEV_MODE only when unset, for
local development) — it must not be keyed on DEV_MODE alone, because the
cloud E2E deploy sets DEV_MODE=true merely to bypass Access and its backups are
real remote-R2 snapshots (see ENV-VARIABLES.md). The flag is persisted in
backup-handle.json and forwarded to restoreBackup(), so a local backup is
always restored through the local binding — never through the remote
presigned-URL path, which 404s for archives that exist only in the local bucket.
Legacy catalog entries without the flag default to the current local-bucket mode,
and a presigned-URL 404 during restore is treated as a gone backup and pruned from
the catalog.
Health probes are observability-only: GET /api/health must never write the
R2 backup catalog — no scheduleBackupIfNeeded, passive or active. A snapshot is
only meaningful for a live instance (a sleeping container has no workload, so
its filesystem has no changes to store), and waking one purely to snapshot would
restart billing. Snapshots therefore run only against a running runtime:
- the Dispatch Worker cron sends a
backupcontrol action; the tenant assertssandbox.isRuntimeActive()(a non-waking bridge read) and returnsskipped: 'not-running'for a sleeping container instead of waking it. A running instance is acknowledged immediately (202) and the snapshot is enqueued as a durable job on the per-tenantActivitySchedulerDO — awaitUntiltask is cancelled ~30s after the response (platform cap) and was observed dropping the snapshot mid-cold-boot, so the cron never blocks on it and the alarm runs it to completion; POST /debug/restart-openclawquiesces the gateway and checkpoints inside arestartmaintenance window (see the hard-restart paragraph below);POST /api/admin/storage/syncis the explicit admin checkpoint.?async=1acknowledges with202and snapshots inwaitUntil; the default (no flag) blocks and returns the result, which the E2E checkpoint relies on;POST /debug/backup/syncandPOST /debug/backup/async(ROOT) trigger the blocking and non-blocking checkpoint modes on demand;POST /api/admin/storage/reconcile(ADMIN/ROOT) runs the catalog heal (reconcileBackupCatalog) on demand: adopt orphanedbackups/*/meta.jsonarchives, sweep metadata-less uploads, re-encrypt plaintext archives, and enforceBACKUP_MAX_VERSIONS. The durablerepairstep runs the same work before every scheduled snapshot, andGET /api/admin/storage/backupsis a pure read that never heals the catalog.
Scheduled snapshots additionally refuse to run while a restore-needed marker
is present, rechecked under the shared lease, so a wake that lands mid-restore
cannot snapshot and promote a partial filesystem over the committed recovery
point. The marker is the durable, R2-backed restore latch — the scheduled gate
does not use a container-state probe, because the legacy exists probe can
fail in a deployed container and failing the gate closed would silently disable
every scheduled backup.
Soft reload (POST /api/admin/gateway/restart) does not snapshot. When no live
gateway is found it starts the gateway through ensureGateway (never arming
restore-needed, so live changes survive) and returns 409 during a maintenance window.
Snapshot promotion is further gated. A snapshot taken while restore-needed is set
over a committed recovery point is archived but does not move the pointer, and a
pre-replacement checkpoint is refused with 409 (RestorePendingError) before the
gateway is touched. A tenant with no recovery point yet may promote its first snapshot.
A restore-only restore (setRecoveryPoint: false) writes recovery-hold.json: scheduled
snapshots archive without promoting until an explicit snapshot clears the hold.
All three snapshot paths — scheduled, manual sync, and the hard-restart
checkpoint — share one tenant-scoped R2 backup-lock lease, so only one
isolate ever stages the fixed legacy stage directory or calls
sandbox.createBackup at a time. The lease has a 60s TTL, is renewed every 20s
while a snapshot is in flight, and is released by compare-and-swap to a
tombstone (never by deleting the object, which would race a concurrent
takeover). The two entry paths differ only in how they acquire it:
- Scheduled acquisition is non-blocking —
scheduleBackupIfNeededtakes the lease withtryAcquireLockand, if another isolate holds it, skips this tick (returns without waiting). - Manual (
POST /api/admin/storage/sync) and the hard-restart checkpoint wait for the lease with bounded exponential backoff (up to 120s), then fail with an error instead of silently reporting success.
The lease is held across the legacy stage + sandbox.createBackup consumption
and released via the CAS tombstone when the snapshot finishes. The dashboard’s
PersistenceLockIndicator mirrors the lock in the UI from the operation
lifecycle (PersistenceLockContext) rather than reading the R2 lease: the
BackupHistoryPanel raises it when it fires an async backup/restore and drops it
when the completion check settles, so there is no polling.
Async admin operations + non-blocking Backups page. A manual snapshot or a
restore can wait up to the 120s lease and then transfer a large archive, so
holding the admin HTTP request open is a poor UX and a needless failure mode (a
restore that lost the lease race surfaced as a bare 500 after ~132s, with the
underlying error dropped). Both admin routes therefore accept ?async=1:
POST /api/admin/storage/sync?async=1 and POST /api/admin/storage/restore/:id?async=1
return 202 { accepted: true } and run the work in executionCtx.waitUntil. The
BackupsPage reads the fast raw config (GET /api/admin/config/raw, a single
cat) instead of the slow structured /api/admin/config (which awaits
ensureGateway + up to two 30s shell waits) and paints every section
independently, so no single request blocks the page. BackupHistoryPanel fires
the async endpoints, immediately pushes a “started” notification to the bell,
then polls the durable job (GET /api/admin/storage/jobs/:id, getBackupJobStatus)
while a snapshot is in flight (falling back to the history diff when the server
returns no job id) or /api/health (restore) and pushes success/failure when the
operation completes — the outcome reaches the admin via NotificationProvider
even after navigating away. Restore failures are now console.error’d for
Observability.
Crash-consistency (issue #344). If a Worker isolate is killed by its CPU
budget between sandbox.createBackup() uploading the archive and the catalog
write, the archive would otherwise be orphaned (present in R2 but absent from
backup-handle.json, hence invisible to listBackups/restore). Two guards
prevent/repair this: the catalog entry is committed immediately after the
archive is created (after encryption, before the config-version capture and
retention prune), and the catalog is reconciled explicitly from
backups/*/meta.json by reconcileBackupCatalog (the durable repair step,
POST /api/admin/storage/reconcile, restoreIfNeeded,
scheduleBackupIfNeeded, and the reprovision guard) whenever an archive is
missing from the handle. getBackupHistory is a pure read: a dashboard poll
never adopts, prunes, sweeps, or writes. The scheduled backup control action
also runs under a dedicated CPU budget (BACKUP_CONTROL_CPU_MS in
workers/fouria-dispatch) so the orchestration can finish; a killed isolate’s
lease is reclaimed via the 60s TTL.
Retention is enforced by the reconcile, not by a catalog read. The two are
coupled: an isolate killed after createBackup() leaves an orphan that
reconciliation adopts, and if later snapshots then fail before their own prune
the catalog and R2 grow past BACKUP_MAX_VERSIONS indefinitely (observed on
tarjetero-dev: 7 archives under maxVersions: 3, the catalog written entirely
by reconcile, with the scheduled snapshots failing at the legacy stage).
reconcileCatalogState (behind reconcileBackupCatalog and
reconcileOrphanedBackups) runs cleanupOldBackups after merging, and the
durable repair step runs it before the gate step of every scheduled snapshot,
so the cap holds however the snapshot path ends. It still performs no
write/delete when the catalog is already within the cap, and no read path can
trigger it.
The catalog is a compare-and-swap, not a last-writer-wins file. Every
backup-handle.json write goes through mutatePersistenceStore(bucket, mutate)
(persistence.ts): it reads the handle and its etag, computes the caller’s
mutation, and writes with onlyIf: { etagMatches } (etagDoesNotMatch: '*' for a
create-only handle). On a conflict it re-reads and re-applies the mutation
against the fresh catalog — never a blind merge, because retention and restore
mutations legitimately remove entries — up to five times before raising
PersistenceStoreConflictError. A mutation returns null to skip the write.
Reconciliation, snapshot finalization (catalog.record-outcome), the retention
prune and the schedule-timestamp commit all run inside this CAS, so a dashboard
poll overlapping a snapshot can no longer drop the newly committed archive (or
resurrect a pruned one).
R2 enumeration follows truncated/cursor. listAllObjects(bucket, prefix)
backs the backup reconcile sweep, the orphan archive sweep, cleanupOldConfigVersions
and listConfigVersions. Previously each site read a single page (1,000 backup
objects, 500 config versions) and silently treated it as the whole result. A
truncated page without a forward cursor logs and stops rather than looping; page
sizes are otherwise unchanged.
Restore robustness. restoreIfNeeded/restoreBackupById persist the
restore-needed marker before changing the mounted filesystem or applying an
archive. This includes an ordinary cold restore where no marker existed before
startup: if the restore or legacy materialization fails after writing partial
state, the next invocation retries instead of treating a leftover config file as
proof of success. The marker is cleared after restoreBackup and legacy
materialization both succeed, or when candidate scanning finds no restorable
archive and there is no stable recovery point (the explicit clean-start case). The
pre-restore umount is best-effort: || true
guards the shell command, but the exec RPC itself can still throw (a
cold-start/platform interruption); that is caught and logged instead of aborting
the restore before the archive is applied. An empty catalog read also never
deletes a pending marker — it can be transient (a reconcile list failure, or
an archive not yet visible), and dropping it would silently lose a requested
retry.
scheduleBackupIfNeeded refuses to run while the durable restore-needed
marker is present (or its status cannot be read), rechecking under the shared
lease, so a scheduled wake that races a restore cannot snapshot a partial
filesystem and replace the stable recovery point. It deliberately does not gate
on a container-state probe: the legacy exists probe is unreliable in a
deployed container, and a false “no state” would silently disable scheduled
backups. The marker is normally absent while idle or after a successful restore;
cold restore selection uses the committed recovery point, not marker presence
alone.
When candidate scanning finds no restorable archive and there is no stable recovery point, restore remains a permitted clean start; the temporary retry marker is removed so onboarding and the next scheduled snapshot can proceed. If a stable recovery point exists but cannot be restored, startup fails closed and the marker remains.
Fail-closed restore (no silent empty boot). A committed stable-backup.json
is the tenant’s recorded recovery point, so its presence means the tenant was
initialized. If restoreIfNeeded finds a recorded recovery point but no
restorable archive (an empty catalog with no stable pointer is only a
brand-new tenant; a catalog drained of its archives, or a stable pointer whose
archive is gone, is not), it now throws TenantStateUnavailableError instead of
reporting a clean start. Letting the container boot empty would run onboarding
against an empty filesystem and the next snapshot would overwrite the last good
recovery point — silent data loss. The cold start aborts (the gateway never
launches), restore-needed is preserved, and the instance stays visibly
unavailable for an operator to recover. When the catalog is empty but the stable
pointer is present, the recorded entry is attempted directly first, so a
transient catalog-read failure still restores rather than falsely failing
closed.
Restore skip is keyed on container state, not runtime liveness. The
cross-isolate cold-boot dedupe previously skipped a restore when
isRuntimeActive() returned true (“Runtime already active; skipping duplicate
restore”). But a freshly provisioned container — started by the prelaunch port
probe, or replaced by an image rollout — is already “active” while its
filesystem is empty, so that check let a replaced container skip its
restore and come up empty (observed on tenant-client-dev: the skip logged 34×
alongside legacy-stage bridge timeouts). restoreIfNeeded now probes the
container’s actual state (containerHasState, the restored/onboarded
openclaw.json under the legacy /root root or the v1 /home root) instead:
absent state means a fresh container that must restore. It is always safe to
attempt an idempotent restore; it is never safe to infer state from a running
process. A probe failure is treated as “no state” (restore), never as “already
restored”. isRuntimeActive() remains the correct read for the non-waking
health/sleep paths, where the question really is “is the runtime up?”, not “is
the tenant state present?”.
Pre-replacement checkpoint (X-FourIA-Control: checkpoint). The /control
route gained a synchronous, verified checkpoint action. The control plane
(Lerma) calls it immediately before an image rollout / container replacement, so
the state about to be discarded is durably captured first. Unlike backup
(interval-gated, 202-accepted, best-effort) it bypasses the interval gate, runs
the full snapshot orchestration, and fails closed (non-2xx) unless the
committed archive’s meta.json is present and restorable. A sleeping/destroyed
instance has no live state and is reported as skipped: not-running rather than
woken. The Dispatch Worker forwards checkpoint with the same raised CPU budget
as backup (BACKUP_CONTROL_CPU_MS). In Lerma, both upgrade paths
(TenantProvisioner’s @upgrade_pipeline :checkpoint step and
Instances.do_upgrade) invoke Cloudflare.Containers.checkpoint/1 before the
rollout. The step is best-effort: a control-plane failure is logged and the
upgrade proceeds, so a checkpoint problem can never brick a valid upgrade.
Control-plane reachability (Cloudflare Access). The Dispatch Worker’s control
endpoints (POST /_control/:slug/{freeze,unfreeze,backup,checkpoint},
GET /_invalidate/:slug) sit behind Cloudflare Access on the platform domain
(dev.fouria.io staging, dash.fouria.io production). Without a path-scoped
Access application those headless calls are redirected to the Access login and
return HTML, so the checkpoint is skipped. apps/fouria/iac/access.tf declares
public_control_apps with a bypass policy (same posture as the Lerma
/api/* CI bypass), so the path is reachable without an Access login; the
Worker still enforces its own internal bearer token on top. Lerma must send that
bearer, so its deploy sets INTERNAL_DISPATCH_INVALIDATE_TOKEN to the dispatch’s
hardcoded key (internal-fouria-dispatch-key) — without it the internal calls
403 and the checkpoint cannot run.
Checkpoint outcome telemetry. A successful tenant checkpoint returns the
archive backupId, quiesced (whether killGateway completed), and
maintenanceWindow (whether the cross-isolate R2 guard was opened). Dispatch
forwards only those checkpoint fields, and Lerma logs them as
backup_id/quiesced/maintenance_window; skipped is reported separately
when the tenant is not running. Require both quiesced: true and
maintenanceWindow: true to claim the guarded quiesce path ran; a successful
snapshot by itself only proves archive creation.
The checkpoint takes the R2 lease before quiescing. A live createBackup
can race the gateway’s own writes, so after acquiring the shared backup-lock
lease the action records a durable, cross-isolate maintenance window
(src/maintenance.ts, maintenance/replacement.json in BACKUP_BUCKET) and
then stops the gateway (killGateway) before snapshotting. A busy lock fails
before maintenance is opened or the gateway is stopped; it must never drain the
tenant while waiting for another snapshot. While the window is open,
ensureGateway refuses to start the gateway
(GatewayMaintenanceError) so a request served by another isolate cannot
resurrect it mid-snapshot; the window has an explicit expiry (15 min) so a
crashed controller can never wedge a tenant, and a malformed/expired record is
discarded (fail open). The checkpoint action always restores service and closes
the window in a finally, even when the snapshot fails, so a failed checkpoint
never leaves the tenant drained. Its recovery restarts the existing live
filesystem with skipRestore: true: the snapshot attempt did not replace the
container, and trying to restore while recovering from a lock timeout would
enqueue another waiter behind the same lock. The proxy maps maintenance refusal
to a retryable 503.
restoreIfNeeded also coalesces concurrent calls within a Worker isolate by
R2 bucket. The startup WebSocket allows only one in-flight
ensureGateway({waitForReady:true}) warm-up, retrying only after a failed attempt
settles and with bounded exponential backoff. Its recurring health snapshots are
passive/observation-only, and startup-progress shell reads run only when the
startup wrapper is actually visible; it does not start a fresh restore attempt
or shell RPC on every poll. Together these prevent a long-lived startup stream
from multiplying 120-second waiters behind one in-flight persistence operation
or waking an empty container ahead of the restore.
Checkpoint recovery restarts the existing container filesystem with
ensureGateway({ bypassMaintenance: true, skipRestore: true }). A checkpoint
does not replace or clear that filesystem; if its snapshot cannot acquire the
shared backup lock, recovery must not call restoreIfNeeded and enqueue another
waiter on the same lock it just timed out waiting for.
Only the snapshot is fail-closed. The maintenance window and the gateway stop
are best-effort guards: a failure to open the window or to stop the gateway
falls back to the previously-shipped live snapshot instead of failing the
replacement (a broken guard must never block an upgrade). The post-snapshot
meta.json check is retried briefly before it is treated as a real failure, so a
transient R2 visibility gap cannot fail an otherwise-successful snapshot. The
checkpoint returns quiesced: true only when the gateway was actually stopped.
This closes most of the planned-replacement loss window, but it is not yet a
full drain-to-replacement fence: the gateway is restarted after the checkpoint,
so writes between the checkpoint and the rollout are still outside the archive,
and the shared control-plane coordination record is not implemented. Durable
primary storage for unexpected container loss remains the gated next step in
docs/plans/legacy-sandbox-zero-data-loss.md.
The pre-snapshot legacy-stage shell runs the node:sqlite online backup plus a
symlink-preserving tar pipe of the legacy roots, with
--exclude='*/node_modules/openclaw'. It deliberately preserves symlinks
rather than dereferencing them (cp -aL) and excludes the image-provided
shared OpenClaw package (extensions/<id>/node_modules/openclaw → the image’s
/usr/local/lib/node_modules/openclaw). -L expanded that package into every
archive (three full copies), which pushed the stage past its bound and — worse —
made the restore materialization a multi-minute copy (observed on
staging-client-dev: cp -a … /root/openclaw/.openclaw/ running 8m45s, starving
the gateway launch, leaving restore-needed set, and re-running the restore
every boot). The restore bridge (materializeLegacyRootsAfterRestore) uses the
same tar --exclude pipe, so an old backup’s dereferenced copy is never
materialized; a copy-then-rm -rf is equally slow because deleting a
squashfs-lower tree requires copying each file up first. restoreIfNeeded clears
the restore-needed marker immediately after a successful materialization
(before the analytics bookkeeping) so an isolate eviction cannot leave a stale
marker. On a large tenant the stage can exceed the
old 300s bound (observed killing the tarjetero stage at exactly 300s — issue
#344 follow-up). The bound is LEGACY_STAGE_TIMEOUT_MS (default 600000), the
idempotent stage is retried once on a transient timeout/transport failure, and
a failure logs exit/timedOut/stderr plus a structured process event with the
Sandbox processId for cross-dataset correlation. The stage wait is
transport-resilient (runShellResilient → waitForExitResilient): a Sandbox
DO↔container drop mid-stage reattaches by process id and resumes within the stage
deadline, so a mid-stage DO reset no longer ends the request. The Dispatch
backup CPU budget (BACKUP_CONTROL_CPU_MS) is aligned to the same 10 minute
bound.
The scheduled backup control action acknowledges immediately (202 { ok: true, accepted: true, jobId }) and enqueues the snapshot as a durable job
on the per-tenant ActivityScheduler DO, so neither the dispatch cron nor an
operator request is bounded by it. The DO alarm runs the job past the request
and retries under a bounded lease — the previous waitUntil approach was
cancelled ~30s after the response and could drop a snapshot that was still
cold-booting. Combined with the crash-consistent catalog commit and orphan
reconciliation (#344) the archive stays durable even if the isolate is replaced
mid-snapshot. POST /debug/backup/async and POST /debug/backup/sync
(DEBUG_ROUTES, ROOT) expose the two modes on demand.
Config-version snapshots. Alongside the archive, each snapshot captures
/home/openclaw/.openclaw/openclaw.json into R2 under configs/<timestamp>.json
(30-day retention) for the Backups page’s Config Version History / diff. The
capture normalizes sandbox.readFile() across SDK generations
(readSandboxFileText in persistence.ts): the legacy SDK returns
{ content, … }, SDK 1.0 returns a Response, and older helpers return a bare
string/Blob. Calling String() on the structured result collapsed it to
"[object Object]", which was stored verbatim and then rejected by the diff with
"[object Object]" is not valid JSON. A JSON.parse write-time guard now
refuses non-JSON config versions, and the diff viewer (DiffTool) shows a clear
“corrupted” message if a pre-existing bad snapshot is opened.
POST /api/admin/storage/test-connectivity (ADMIN/ROOT) is a read-only
diagnostic that verifies the R2 backup bucket binding is reachable by HEADing the
backup catalog handle — a 404 (no handle yet) still counts as connected. It is
whitelisted for ROOT through the diagnostics gate (DIAGNOSTICS_PATHS in
src/routes/api.ts) so platform operators can test backup connectivity from the
Diagnostic Dashboard without explicit access to the ADMIN-gated Backups page.
Hard restart (POST /debug/restart-openclaw) takes the shared lease, then opens a
restart maintenance window and kills the gateway before cutting the archive, so
the checkpoint is consistent. It sets restore-needed before closing the window, so no
request can relaunch the gateway on pre-restart state in between. If the checkpoint
fails after the stop, the gateway is restarted in place on the untouched filesystem and no restore is armed. It refuses with 409 while a restore is in flight
or a maintenance window is open, and while a restore is pending over the recovery point.
After a successful checkpoint, the next activation restores the committed snapshot.
Restore outcome. sandbox.restoreBackup is bounded by RESTORE_BACKUP_TIMEOUT_MS.
Because withTimeout only stops waiting, a timeout is recorded in
restore-target.json as outcome: 'unknown', not as a failure. While that record is
younger than GATEWAY_RESTORE_PHASE_BUDGET_MS, getRestoreInFlightReason blocks every
restore (RestoreInFlightError, 409 on the admin route). A definitive failure is
recorded as outcome: 'failed' and does not block. The in-lease checks (restore
entry, restoreIfNeeded, restoreBackupById) re-read this under the lease, and
restoreIfNeeded also re-checks the maintenance window after it waits for the lease.
Destructive guard. getPersistenceBusyReason is the single busy check for freeze
(/control), POST /debug/destroy-container, and POST /debug/reprovision. It is
busy while any maintenance window is open or a restore is in flight. Reprovision also
re-checks after its checkpoint. A snapshot-only lease is not busy: an abandoned
snapshot leaves no half-applied filesystem behind, whereas an abandoned restore does.
flowchart TD
DC["Dispatch Worker cron<br/>backup control action"] --> RC{"runtime active?<br/>isRuntimeActive()"}
RC -- no --> SKIP["skip (never wake)"]
RC -- yes --> S["scheduleBackupIfNeeded"]
S --> G{"credentials + interval + lock"}
G -- no --> N["no-op"]
G -- yes --> L{"backup-lock free?<br/>(non-blocking)"}
L -- busy --> SKIP["scheduler skips this tick"]
L -- free --> HOLD
MAN["POST /api/admin/storage/sync"] --> W["acquire lease<br/>wait ≤120s w/ backoff"]
HR["POST /debug/restart-openclaw"] --> CP["checkpoint first"]
CP --> W
W -- busy > 120s --> ERR["error (no snapshot)"]
W -- acquired --> HOLD["lease held over legacy stage<br/>+ sandbox.createBackup"]
HOLD --> A["Sandbox snapshot<br/>/home/openclaw"]
A --> E["optional F4E1 encryption"]
E --> P["commit backup-handle.json"]
P --> REL["release lease (CAS tombstone)"]
CP --> K["graceful kill + restore marker"]
K --> R["restore committed snapshot<br/>on next activation"]
RP["POST /debug/reprovision"] --> RCP["checkpoint first<br/>(or require existing backup)"]
RCP --> RD["destroy instance<br/>409 unless restorable backup or force"]
SR["POST /api/admin/gateway/restart"] --> SOFT["SIGUSR2 in-process reload<br/>no snapshot"]
Do not perform repeated hard restarts without verifying a successful
checkpoint. The checkpoint protects only files inside /home/openclaw and only
after the backup catalog has been committed.
Durable async backup/restore jobs + stable recovery point
POST /api/admin/storage/sync?async=1 and
POST /api/admin/storage/restore/:id?async=1 no longer rely on request
waitUntil() (Cloudflare only guarantees ~30s of background time after a
response). Instead the route commits a job record to the per-tenant
ActivityScheduler Durable Object (src/activity/backup-jobs.ts) and returns
202 { accepted: true, jobId }. The DO alarm performs the work and retries with
a bounded lease (BACKUP_JOB_LEASE_MS) so a multi-minute snapshot survives
isolate eviction and Worker redeploys. GET /api/admin/storage/jobs/:id reads
the persisted status. The ActivityScheduler owns one alarm, so the DO merges
the next automation wake and the next backup job and arms the earliest
(scheduleCombinedAlarm); a queued job is never dropped when the activity
schedule clears the alarm.
Every durable job runs one bounded step per alarm invocation. A Durable
Object alarm handler has a hard 15-minute wall limit, while a snapshot is
stage (≤10 min) + createBackup + config capture + prune — and the alarm also
runs the boot, maintenance and activity queues. A single-invocation job could
therefore be killed by the platform, stay running until its lease expired, and
restart the whole operation from zero (the orphaned-archive class of incident).
src/activity/job-steps.ts persists a cursor: { step, attempts, startedAt }
on the job record; each alarm runs at most one step, commits the cursor, and
re-arms one second later so the other queues get a fair turn.
| Job | Steps |
|---|---|
| snapshot (manual) | repair → prepare → archive → finalize |
| snapshot (scheduled) | repair → gate → prepare → archive → finalize |
| deprovision | repair → prepare → archive → finalize → stop |
| restore | restore → launch |
| boot | restore → launch |
| maintenance | preflight → execute |
repairbrings the catalog back to a consistent state before any snapshot work: it sweeps archive objects that no catalog entry can use (the SDK writesbackups/<id>/meta.jsononly after a successful upload, so an upload that lost its metadata write is invisible to both the reconciler and the retention prune — it would sit in the bucket forever, billed and unrestorable) and it encrypts the archives that are not encrypted at rest.- The lease covers one step, not the whole operation (
STEP_LEASE_MS= 6 min). A Durable Object alarm has a hard 15-minute wall limit, so the previous 14-minute lease meant a step killed at minute 3 stalled the job for another 11 minutes before it was retried. Alarm invocations are serialized per object, so a healthy step is never duplicated. - A step recovered after its lease expired had an unknown outcome: the job
says so (
detail: 'step-interrupted', or'step-deadline-exceeded'past the 12-minute soft deadline) and emits ajob-step.interruptedevent instead of silently replaying. Every step emits ajob-step.completed/job-step.failedevent with its duration — the measurement the step budget is derived from (max(configured, 2 x the last snapshot duration)for the cadence, 12 min for the per-step deadline). prepare/archive/finalizeare the phased snapshot (prepareSnapshot/archiveSnapshot/finalizeSnapshotinsrc/persistence.ts). The catalog entry is still committed inarchive, immediately after encryption, so a crash beforefinalizeleaves a restorable archive (andfinalizere-adopts it if the catalog write was lost).- Each step takes the shared R2 lease on its own (
withSnapshotLease); the stage is idempotent andarchivere-checks therestore-neededlatch, so a hand-off between steps cannot capture a half-restored container. A scheduled snapshot refuses to archive behind a pending restore; a manual/deprovision checkpoint deliberately still archives (without promoting), which is the documented recovery case. - The retry budget is per step (
isStepExhausted) as well as per job, so a permanently broken step fails the job instead of consuming attempts that belong to healthy steps. A completed step is never re-run: a boot that is interrupted duringlaunchresumes atlaunch, not at the destructiverestore. - The legacy stage runs with
maxAttempts: 1inside a durable step (the step cursor owns the retry), so its worst case stays at the 10-minuteLEGACY_STAGE_TIMEOUT_MSinstead of 20 minutes. - The activity alarm now enqueues a scheduled snapshot instead of running
scheduleBackupIfNeededinline, so a 10-minute stage no longer shares an invocation with the automation wake.
Archives that are not encrypted at rest are visible and repaired. While a
master key is configured, an archive whose meta.json does not carry
encrypted: true (a legacy plaintext archive, or one whose upload finished
while the encryption step never ran) is catalogued with unencrypted: true and
queued in pendingEncryption. It is never dropped (it stays restorable) and
never promoted to the recovery point; the durable repair step encrypts it
in place and clears the flag. The dashboard shows a “not encrypted” badge for
such an archive.
Scheduled cadence is a floor, not a promise. BACKUP_INTERVAL_MINUTES
(default 5) is the minimum; resolveEffectiveIntervalMs stretches the effective
interval to at least 2× the last observed end-to-end snapshot duration
(recorded as lastSnapshotDurationMs in the catalog). Without that, a tenant
whose snapshot takes 10 minutes would start a new one the moment the previous
committed: continuous R2 lease contention, continuous container work, and a
starved automation wake. A fast tenant is unaffected.
createSnapshot promotes a stable recovery point only after the SDK
archive, optional F4E1 encryption, and the catalog commit have all succeeded
(promoteStableBackup → stable-backup.json). A cold restore prefers that
pointer and never restores a newer archive whose promotion was interrupted;
retention (cleanupOldBackups) always keeps the stable entry, so a
just-uploaded-but-unpromoted archive cannot evict the last committed recovery
point. The pointer is also monotonic (an older or replayed archive never
moves it backwards) and fails closed: a malformed stable-backup.json
aborts restore and retention rather than falling forward to an unpromoted
archive. Unit-tested in src/persistence.test.ts (the “stable recovery point
promotion” block). The latest validated config is also mirrored to a fixed-key
hot copy (stable/config.json), kept out of the version-history listing and
30-day prune.
A targeted restore (POST /api/admin/storage/restore/:id) force-promotes the
restored archive (the exception to the forward-only pointer) so the next cold
boot restores the same archive. ?setCurrent=0 (sync or ?async=1) threads
{ setRecoveryPoint: false } through restoreBackupById (and the durable
restore job) to apply an archive without moving the recovery point — the
next fresh container then reverts to the unchanged pointer. A restore also opens
a cross-isolate maintenance/replacement.json window (operation restore) and
stops the gateway right after taking the shared restore lease, closing the
window in a finally; restoreIfNeeded and ensureGateway both refuse to run
under it, so a refetch cannot restart the gateway and remount the latest
recovery point mid-restore.
This makes the async workflow durable and the recovery point monotonic. It does not turn interval snapshots into zero-RPO replication: writes acknowledged after the last committed snapshot are outside the guarantee. Closing that gap requires a write-through storage path and is tracked in issue #377.
Sandbox → R2 Connectivity (outboundByHost)
Sandbox code can reach Worker bindings through a virtual hostname without
any SDK/client library or credentials inside the container. @cloudflare/sandbox
(via @cloudflare/containers) exposes the canonical per-host outbound handler:
// src/index.ts
Sandbox.outboundByHost = {
[RELEASES_R2_HOST]: (request, env, ctx) => handleReleasesR2Request(request, env),
};
// catch-all: internet passthrough
Sandbox.outbound = async (request) => fetch(request);
The handler runs in the Workers runtime with full env bindings, so the
sandbox never sees R2 credentials. Unmatched hosts fall through to the catch-all
Sandbox.outbound.
Release bridge (releases.r2.internal). src/sandbox-r2-bridge.ts maps
plain HTTP from the sandbox onto the tenant’s BACKUP_BUCKET under a fixed
releases/ prefix (the bucket is per-tenant, so the prefix is already
tenant-isolated):
| Method | Path | Effect |
|---|---|---|
GET | / | JSON list of release objects |
GET | /<key> | Stream an object (200) or 404 |
HEAD | /<key> | Size/etag metadata (200) or 404 |
PUT | /<key> | Upload an object (201) |
DELETE | /<key> | Remove an object (204) or 404 |
Keys are sanitized against traversal before touching R2; uploads use a
FixedLengthStream (bounded memory) when content-length is advertised.
Docs bridge (docs.r2.internal). src/sandbox-docs-bridge.ts exposes the
tenant’s client documentation (customers/<slug>/docs/) for on-demand fetch by
the Fouria Builder agent / sandbox tooling, reusing R2ClientDocsStore
sanitization. The tenant slug is resolved via the shared resolveTenantSlug
(src/sandbox-tenant.ts): an explicit customer-<slug> container id wins,
otherwise it is derived from the per-tenant Worker env (CUSTOMER_SLUG, else
WORKER_NAME / the always-provisioned WORKER_URL, with the tenant- /
staging- prefixes and -dev suffix stripped), since production provisions the
sandbox under the fixed id openclaw. GET / lists,
GET/HEAD /<name> reads, PUT /<name> writes, DELETE /<name> removes. This
lets the sandbox read docs without R2 credentials and without waiting for
Worker-side materialization.
Leads bridge (leads.r2.internal). src/sandbox-leads-bridge.ts exposes the
global paso4-leads proposal bucket (the fouria-leads skill) over
LEADS_BUCKET. The bucket folder layer is the tenant slug, and the bridge
resolves the caller’s slug via the shared resolveTenantSlug in
src/sandbox-tenant.ts (explicit customer-<slug> container id, else the
per-tenant Worker env CUSTOMER_SLUG / WORKER_NAME / WORKER_URL). Every path
the agent sends is relative to its own <slug>/ root — the handler prepends
the slug,
so a sandboxed agent cannot list, read, or write another client’s proposal. The
only bucket-root object a tenant may read is the shared, proposal-agnostic
build-pdf.sh (SHARED_LEADS_READ_KEYS); writes/deletes of it resolve under the
tenant’s own folder. Routes: GET / (or GET /?prefix=<p>) lists relative
keys, GET/HEAD /<key> reads, PUT /<key> writes, DELETE /<key> removes. Keys
reuse the shared sanitizeLeadsKey / sanitizeLeadsPrefix rules from
leads/operations.ts. This is how the sandbox reaches proposal docs without a
public route — the legacy wss://<worker>/api/leads/ws proxy
(src/routes/leads.ts) is intercepted by Cloudflare Access on tenant hostnames
(a per-tenant Access app shadows the *.fouria.io/<path> apps) and is now used
only by the host-side E2E fixture (test/e2e/o_leads_proxy.txt, which
authenticates with the CDP_SECRET persisted by server/start).
Binding requirement.
LEADS_BUCKETmust be provisioned as anr2_bucketbinding (lermainstances.exinstance_bindings/2, configLEADS_R2_BUCKET, defaultpaso4-leads). A plain-text var holding the bucket name makesenv.LEADS_BUCKETastring, sobucket.list()throws inside theoutboundByHosthandler and the container sees an opaqueECONNRESET. All three bridges validate the binding withisR2Bucket(src/r2-binding.ts) and wrap their handler body in a try/catch, returning a structured JSON503/500instead of resetting the socket.
R2 S3 credentials are deprecated for the container. buildEnvVars no longer
forwards R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY into the sandbox: the
bridges run in the Workers runtime with the BACKUP_BUCKET / LEADS_BUCKET
bindings, so the container never sees credentials. The S3 credential remains a
Worker binding for the Sandbox SDK presigned-URL backup path only.
SDK 1.0 (preview). The canonical
static outboundByHostAPI is unchanged in the 1.0 preview, so these bridges run as-is on@cloudflare/sandbox@0.13.0-next; see the Sandbox SDK 1.0 section below.
Client Docs (dev-only admin surface)
Client documentation is managed from the Admin Client Docs page
(/_admin/docs, ADMIN-only, dev instances only) backed by
src/routes/api/fouria-docs.ts (dev-gated by isDevAgentsEnabled, so it 404s in
production). It manages two tenant-scoped namespaces directly from the Worker
— no sandbox/CLI round-trip:
- Documentation —
customers/<slug>/docs/inBACKUP_BUCKET(markdown, text, and binary files like PDFs/images; content type from R2 metadata or the extension). Store:R2ClientDocsStore(listDocs/getDoc/getDocBinary/putDoc/putDocBinary). - Proposals (leads) —
<slug>/in the globalLEADS_BUCKET(PDFs and assets), scoped to the caller’s tenant.
The SPA renders markdown client-side (marked + DOMPurify, sanitized) and opens
PDFs with the browser’s native viewer through the /raw endpoints. Raw responses
set X-Content-Type-Options: nosniff; HTML/SVG are forced to attachment with a
sandbox CSP so client-authored files can never execute in the admin origin.
Versioning (DocsGitStore). Every doc write is committed to a Worker-side,
git-like object store in BACKUP_BUCKET:
customers/<slug>/docs-git/objects/<blobSha> content-addressed file bytes
customers/<slug>/docs-git/commits/<commitSha> commit JSON ({ tree, parents, … })
customers/<slug>/docs-git/refs/heads/dev mutable HEAD
customers/<slug>/docs-git/refs/promoted promoted ref
The working namespace always mirrors HEAD. History is exposed in the dashboard’s
commit-tree shape (GET /commits); POST /rollback/:sha restores an older commit
as a new, non-destructive revert commit.
Backups (DocsBackupStore). POST /backup snapshots the working namespace to
customers/<slug>/docs-backups/<id>/ (files + manifest.json), independent of the
git store; GET /backups lists (retention keeps the newest 10), and
POST /backups/:id/restore re-applies a snapshot as a new commit.
Sandbox SDK 1.0 side-by-side foundation (SandboxV1)
Alongside the default legacy Sandbox path, the Worker carries an additive,
not-routed Sandbox SDK 1.0 foundation so the side-by-side move can be built
without a big-bang cutover. It is a foundation only — no tenant is on it and
it carries no durability claim.
- Class —
SandboxV1(src/sandbox-v1.ts), a user-ownedDurableObjectthat drivesthis.ctx.container(the 1.0 model). Exported fromsrc/index.tstogether withDirectoryBackupGateway(needed for the DO to passctx.exports.DirectoryBackupGatewaytoDirectoryBackup). - Registration —
wrangler.jsoncmigrationv7(new_sqlite_classes: ["SandboxV1"]), aSandboxV1DO binding, and a separatescheduling_policy: "durable_object"container application per environment (fouria-sandbox-v1,fouria-sandbox-v1-staging) that buildsimages.sandboxfromapps/fouria/Dockerfile.v1. - Default routing unchanged — the legacy
Sandboxclass +getSandbox()still serve every tenant. No route selectsSandboxV1; there is no customer routing change and no cutover. - Package / image pairing — root catalog installs
@cloudflare/sandbox@1.0.0(this DO) and@cloudflare/sandbox-legacy@0.13.0-next.776.1(the still-default legacy class, imported via the-legacyalias).Dockerfile.v1usesnode:24-trixie-slimand COPYs thesandbox-shimhelper from the matchingcloudflare/sandbox:1.0.0image (the shim tag must track the package version); the legacy base image no longer applies because 1.0 removes the in-image SDK server. - TTL (derived, capped at 6h) —
resolveInactivityPolicy()reusesresolveSleepAfter(): non-billable instances get a finite idle timeout (staging 15m, dev 2h) clamped to the 6h platform maximum accepted bysetInactivityTimeout(). Billable production gets no inactivity timeout (timeoutMs: null): the container stays awake until an explicit lerma freeze. There is noSANDBOX_SLEEP_AFTERbinding. - Real
DirectoryBackupwith F4E1 at-rest wrapping — the additive DO instantiates the realDirectoryBackup(from@cloudflare/sandbox@1.x) againstctx.container+ctx.exports.DirectoryBackupGatewaywith a tenant-scopedsandbox-v1/<encoded-do-name>/R2 prefix inBACKUP_BUCKET. The SDK writes one flat plaintext object per record (<prefix><id>.tar.zst); the Worker then post-encrypts it in place to<prefix><id>.tar.zst.enc(F4E1/AES-GCM) plus a readable<prefix><id>.meta.jsonsidecar, and deletes the plaintext canonical object. Restore transiently decrypts.encback to the canonical key for the SDK, then always deletes that plaintext in afinallyblock; delete removes all three objects (strict for explicit user deletion, best-effort on cleanup). Plaintext therefore exists only in the short post-upload window and the restore/decrypt window. The master key stays Worker-only (env.BACKUP_ENCRYPTION_KEY, never passed to the container); the record’ssha256/sizeremain plaintext and the SDKDirectoryBackup.restore()verifies them on restore. WithBACKUP_ENCRYPTION_REQUIRED=truea missing key/meta fails closed before any upload or restore. - Still no scheduling, conversion, or cutover — Worker-owned periodic
snapshot scheduling + retention is not implemented (no approved defaults
wired; the intended targets are a 5m snapshot interval, retaining the latest
interval snapshots plus a 24h rolling window and daily 30d history, once a
scheduler exists). There is no 0.13
.sqsh.enc→ DirectoryBackup conversion, no customer cutover, and no default-routing change: the legacySandboxclass +getSandbox()still serve every tenant, no route selectsSandboxV1, and the complete restore-before-Gateway path is not wired for the v1 line. This is not a final durability/no-loss guarantee. Seedocs/plans/sandbox-v1-scaffold.md.
Sandbox SDK runtime lines
The default tenant runtime is the legacy Sandbox class, installed via the
@cloudflare/sandbox-legacy catalog alias of npm:@cloudflare/sandbox
0.13.0-next.776.1, paired with the docker.io/cloudflare/sandbox:0.13.0-next.776.1
image. The Worker package and the container image must stay on the same line —
the stable and 1.0 control protocols are mutually incompatible.
The side-by-side SandboxV1 Durable Object uses the stable @cloudflare/sandbox
1.0.0 package (see the Sandbox SDK 1.0 section above). The intended runtime is
selected by the SANDBOX_RUNTIME binding (legacy default | v1); until V1
implements the full Sandbox surface the legacy class keeps serving and
resolveSandboxRuntime() reports the requested V1 as pending (see
src/sandbox-runtime.ts).
Every tenant-facing site that needs a Sandbox obtains it through the single
runtime acquisition boundary acquireSandbox() in src/sandbox-contract.ts —
the request middleware (src/index.ts), the task buffer (src/tasks/index.ts),
the ActivityScheduler Durable Object (src/activity/scheduler.ts), and the
per-customer gateway launcher (src/gateway/process.ts). The boundary resolves
the runtime but always returns the legacy stub for now (a v1 request is still
pending), preserves the container identity (openclaw for the single-tenant
instance, customer-<slug> for a per-customer gateway) and the legacy idle
policy (defaults to buildSandboxOptions(env)), and never starts a container —
getSandbox() returns a Durable Object stub, so acquiring a runtime never wakes
a sleeping tenant or touches the v1 DO namespace. This is the single flip point
for the cutover.
Key differences from the 0.12 stable line, all funneled through
src/gateway/sandbox-exec.ts:
- No
startProcess/execStream/getLogs().sandbox.exec(argv)starts a process and resolves once it is launched, not once it finishes; it never implies a shell.startShell(sandbox, script)wraps a shell command line in['/bin/bash', '-lc', script]and returns the handle;runShell(sandbox, script)waits and buffers{stdout, stderr, exitCode};outputOf(proc)/collectLogs(proc)read an existing handle. listProcesses()returnsProcessStatus[](.state, argv.command,.startedAt) with no methods.findExistingGatewayProcessfetches a live handle viagetProcess(id)and returns aGatewayProcesswrapper (.state,.commandstring,.refresh(),.waitForPort(),.kill(),.getLogs()).waitForProcessdelegates towaitForExit({ timeout })and kills the process on a local wait timeout (the SDK timeout cancels the wait only).- Busy-DO hardening for gateway reads. A contended container-control channel
can hold an
execlaunch for the SDK default (120s) and stall the Sandbox DO. Read routes therefore bound their sandbox work and degrade instead of hanging past the client timeout:ensureGateway(waitForReady:false)(every API read route) is bounded byENSURE_GATEWAY_READ_TIMEOUT_MS(30s) and returnsnullon expiry; the caller serves its normal “not ready” payload. Cold-start (waitForReady:true) is exempt.- A short-TTL positive readiness result (
GATEWAY_READINESS_CACHE_TTL_MS, 5s) collapses concurrent read-path probes onto one; it is cleared on kill/restart/recovery. - The gateway port probe (
isGatewayPortOpen) is bounded byPORT_PROBE_TIMEOUT_MS(10s) with a container-sidecurl -m, and marks the expected closed-port exit (curl7 = connection refused) withreportNonZeroExit: false(operationgateway.port-probe) so the normal “gateway down, start it” outcome is not logged as asandbox.process.failederror on every cold start — matching thegateway.health-probebehavior. - Expected CLI non-zero exits are not logged as failures either:
runShell(viareportNonZeroExit) andwaitForProcess(same option) let a call site mark an exit it handles as normal. The knowledgewiki statusprobe uses it (operation: 'knowledge.wiki-status') because an uninitialized vault exits non-zero and the handler reportsinitialized:false. Timeouts always log. - Route flows that poll the container wrap their whole flow in a budget
(
/api/admin/devices25s,/debug/container-config20s).
- No transport selection.
SANDBOX_TRANSPORTwas removed; the preview uses RPC automatically. SandboxOptions.labelswas removed upstream (worker identity is no longer attached at container start).
Deploy cutover: because the protocols are incompatible, a production deploy
must roll containers in one step — wrangler deploy --containers-rollout=immediate
— with rollout_active_grace_period at 0. In-flight processes/terminals stop
and pre-deploy process IDs become invalid.
Container Image Build Profiles (dev vs production)
The Dockerfile takes a BUILD_PROFILE build arg (default production):
- production (default) installs only
xz-utils— the slim image (~1.36 GB) used by production and by E2E. - dev additionally installs the optional PDF/document tooling
(
pandoc+weasyprint— a large Python/cairo/pango stack, ~760 MB) that the dev-only Fouria Builderfouria-leadsskill needs (apps/fouria/skills/agent-only/fouria-leads/SKILL.md; the Builder is injected only whenOPENCLAW_DEV_MODE=true).
The profile is selected per wrangler environment via image_vars (Cloudflare’s
build-time variables, equivalent to docker build --build-arg):
| Environment | BUILD_PROFILE | Image |
|---|---|---|
env.staging (dev/test) | dev | with pandoc/weasyprint |
env.production | default (production) | slim |
| default / local Vite dev (local + E2E) | default (production) | slim |
There is a single image definition (apps/fouria/Dockerfile); CI
(test.yml) builds it with no build args when it pushes unstable-<sha>, and
the release CD (deploy-fouria.yml) promotes that same image by retagging it, so
released images are the slim production profile. The E2E basic (PR) and
production (develop) profiles both provision the slim production image (only
instance type/env differ). To build a dev image locally:
docker build --build-arg BUILD_PROFILE=dev -f apps/fouria/Dockerfile .Sandbox Recovery (stale + orphaned)
When the platform terminates a container behind a live Sandbox DO (idle
recycle, maintenance, SIGTERM), every RPC into it fails with the SDK’s
HTTP error! status: 500 wrapper — and retrying against the same instance
fails forever, bricking the tenant. gateway/sandbox-recovery.ts classifies
these dead-container errors (isStaleContainerError) and recovers
(recoverStaleSandbox): destroy the instance, reset the startup-probe and
persistence caches. The next RPC auto-provisions a fresh container and the
normal cold-start flow runs (R2 restore → start-openclaw.sh → gateway).
A container destroyed out-of-band (Cloudflare Containers dashboard “Destroy”,
failed operator force-stop) is a second failure class: the DO is alive but points
at a placement that no longer exists, so RPCs fail with the SDK’s retryable
admission errors (no container instance available, maximum number of running container instances exceeded) instead of the 500 signature. Those are legitimate
during a cold start, so they are not treated as stale on their own.
isContainerUnavailableError + trackProvisioningFailure escalate them to a
recovery once the streak reaches ORPHANED_INSTANCE_MIN_FAILURES and spans
ORPHANED_INSTANCE_GRACE_MS (120s — longer than the SDK’s ≈110s provisioning
budget, so a slow-but-healthy cold start is never cancelled mid-provision). The
streak is cleared whenever a sandbox RPC succeeds. maybeRecoverSandbox unifies
both classes for callers.
Recovery is wired into three failure points in gateway/process.ts:
findExistingGatewayProcess (listProcesses failures), checkGatewayHealth
(exec failures), and the ensureGateway startShell path (recover + retry
once). It is cooldown-guarded (STALE_RECOVERY_COOLDOWN_MS, 60s) and never
fires for deploy-window OperationInterruptedError. Permanent image errors still
require the rescue tool (rescueContainer, see OPERATOR-GUIDE §5.1). Logs use the
[SandboxRecovery] prefix.
Transient transport drops (retry, don’t reprovision)
A third failure signature is not a dead container: Network connection lost.
(and similar socket/WebSocket transport messages). It appears when the DO↔container
connection drops while the container is alive — most visibly while the Gateway is
applying a config/plugin reload (e.g. adding a model provider in the Control UI),
which restarts the Gateway and briefly tears down the container RPC/WS transports.
isContainerTransportError (gateway/sandbox-recovery.ts) classifies these, and
isStaleContainerError explicitly excludes them (even when the SDK wraps the
message in its sandbox.<op> error prefix) so a transient drop never destroys a
live instance. ensureGateway instead retries with a short backoff: the port
probe (isGatewayPortOpenResilient, up to TRANSPORT_RETRY_ATTEMPTS) and the
startShell spawn (once). Before this, a reload that dropped the transport mid
probe fell straight through to start-openclaw.sh → Failed to start process →
a 502 to the user, even though the Gateway came back seconds later. The retry
keeps the request alive across the reload window.
A Durable Object code update (a normal Worker/DO redeploy, or a Lerma
redeploy of the tenant script) resets the control session between the DO and its
container even though the container itself is not replaced. The cold start then
loses the transport on the supervised port wait: the atomic
start-openclaw.sh start process (config setup + gateway under one lock) keeps
running, but waitForPort() on the stale handle rejected and previously aborted
the cold start — killing a gateway that would have completed (observed as
Network connection lost.). The atomic start path has no separate
setup-process wait, so only the gateway port wait reattaches via
getProcess(id) and resumes (waitForPortResilient,
gateway/sandbox-exec.ts), bounded by the shared STARTUP_TIMEOUT_MS deadline.
The generic waitForExitResilient helper remains in sandbox-exec.ts for the
legacy/boot-smoke supervised waits. Only a non-transport error, an
un-reattachable process, or an exhausted deadline still fails the cold start.
The openclaw doctor diagnostics run through the same transport-resilient path:
runShellResilient, which retries the exec launch (startShellResilient) and
reattaches the wait (waitForExitResilient). Previously a transient
The operation was aborted drop on the doctor startShell surfaced as a hard
500 { ok: false, error: "The operation was aborted" }; it now reattaches to the
process and returns the structured diagnostic instead. GET /api/doctor still
runs inline this way; POST /api/admin/config/fix and POST /debug/sqlite-integrity
now enqueue a durable maintenance job on the ActivityScheduler DO
(activity/maintenance-jobs.ts + activity/maintenance-executor.ts) and return
202 { accepted, jobId }, because a large integrity check or doctor repair can
exceed a Worker request’s lifetime (the old 30s/90s shell timeouts surfaced as
“timeout in under 2 minutes”). The DO alarm runs the command under a 14-minute
lease with bounded retries (runMaintenanceOperation, still runShellResilient
internally) and stores the bounded result on the job record, polled via
GET /debug/sqlite-integrity/jobs/:id / GET /api/admin/config/fix/jobs/:id.
Both endpoints fall back to an inline run when the ACTIVITY binding is absent
(local dev / tests). The doctor repair runs openclaw doctor --fix --non-interactive
(the supported command for index-only integrity_check corruption — missing or
wrong-count index entries — which verifies the table data, preserves a backup,
and REINDEXes) and quiesces the gateway first: runMaintenanceOperation
opens the cross-isolate maintenance window (beginMaintenance(..., 'doctor-repair')),
kills the gateway so the repair can take exclusive state ownership, then closes
the window so the next request re-launches the gateway.
Restore → repair ordering (restore-config-pending.json). A restore replaces
the container’s writable layer but launches nothing, so a config repair applied
around a restore is silently overwritten by the next re-apply of the recovery
point (observed on tenant-tarjetero-dev: a config fix was lost on the next
gateway restore). Every successful restore (restoreIfNeeded and
restoreBackupById) now writes a durable restore-config-pending.json marker.
The durable boot job and the durable restore job launch the gateway on the
restored state through the shared launchRestoredState helper
(src/post-restore.ts); the admin sync/async-fallback restore route ends there
too, so every restore path has the same ordering guarantee. The helper calls
ensureGateway({ waitForReady: true }) and then
captureRestoredConfigVersion — writing configs/<ts>.json and
stable/config.json, so the restored config becomes the newest and stable
config version and the dashboard’s diff/history reflects what is actually live.
The marker is cleared only after that capture. A gateway-launch failure never
re-runs the destructive restore: restore jobs stay succeeded with
detail: "gateway-launch-pending" and the sync route returns
gatewayReady: false; the pending marker drives the next boot / maintenance
tick. A doctor-repair job checks,
before running: an active restore job, an open maintenance window / in-flight
restore (getPersistenceBusyReason), or a pending restored-config marker. In any
of those cases it defers (MaintenanceJobDeferredError): the job keeps its
attempts and queue position (60s re-check) instead of failing its three attempts
while the restore runs, and executes only after the gateway is up on the restored
state.
Job status is pushed over a WebSocket — GET /api/admin/config/fix/jobs/:id/ws
(and the /debug/sqlite-integrity/jobs/:id/ws variant) — instead of the
dashboard polling the JSON endpoints above every 2s (one request + Worker log
line per tick for the whole multi-minute repair). The streamer
(routes/api/maintenance-ws.ts) reads the durable record from the
ActivityScheduler DO, sends a frame only when it changes, and closes 1000
once the job is terminal; the dashboard’s subscribeMaintenanceJob falls back to
a slow HTTP poll only while the socket is down. While a job is active the
streamer also calls the DO’s rearm() self-heal, so a job whose alarm was
lost is scheduled again immediately.
The shared alarm is written through an optimistic-concurrency guard
(armWithGenerationGuard): the ActivityScheduler owns one alarm, so three job
enqueues, the warm-path schedule refresh, and the alarm handler’s own trailing
re-arm can interleave at await points. A stale arm that read “no pending job”
could execute setAlarm(later)/deleteAlarm() after a just-enqueued job’s
setAlarm(now) and strand the job until the later time — observed as a
doctor-repair job enqueued at 12:57:30 that only began at 13:02:11 (exactly the
previous boot job’s 14-minute lease). The guard bumps a stored generation and
abandons a superseded arm’s write, so the newest arm (fresh read) owns the
alarm.
While the control session reconnects, the loading page’s GET /api/startup/ws
stream treats a transport-drop snapshot as transient: it re-sends the last known
state (or a synthetic starting view) and backs off exponentially
(STARTUP_WS_POLL_INTERVAL_MS → STARTUP_WS_MAX_BACKOFF_MS, 30s) instead of
re-probing — and re-logging the same error — every 5s. A successful probe resets
the cadence.
The manual lever is POST /debug/reprovision (DEBUG_ROUTES, ROOT): it
checkpoints the live container first and fails closed — unless it just
committed a snapshot or a restorable backup already exists it refuses with
409 rather than silently destroying tenant state; {"force":true} is the
explicit override for an already-unrecoverable container. The checkpoint is only
attempted when isRuntimeActive() reports a live runtime (so an idle container
is never woken) and is time-bounded (REPROVISION_CHECKPOINT_TIMEOUT_MS); the
restorability check HEADs the newest catalog entries’ meta.json rather than
trusting the entry count. On success it destroys the instance, clears the
startup/persistence caches, and resets the orphan streak without ever contacting
the gateway — so it works when the container is entirely gone. Compare
POST /debug/restart-openclaw (gateway-aware, checkpointed) and
POST /debug/destroy-container (teardown only, used by E2E).
Activity-Driven Wake (ActivityScheduler)
Automations (“jobs”) run inside the Gateway, so the Gateway must be running
for a schedule to fire. Historically a Worker cron trigger woke every tenant on
every tick (expensive and since removed). src/activity/ replaces it with a
per-tenant activity clock:
ActivitySchedulerDurable Object (SQLite-backed, bindingACTIVITY) — stores a mirror of the tenant’s automation schedule and owns a single alarm. Deliberately a separate DO from the SDKSandboxDO, which already uses its own alarm forsleepAfter.- Mirror — warm paths (
GET /api/health, only whenreadiness.ok) readopenclaw automations list --jsonand push the earliestnextRunAtMsinto the DO. The DO throttles this to once per 5 min, and it never wakes a sleeping container. - Alarm (
activity/alarm.ts) — firesCRON_WAKE_AHEAD_MINUTES(default 10) before the next job, wakes + heals viaensureGateway(which recovers an orphaned instance — see above), then re-reads the schedule and re-arms. No enabled jobs → alarm cleared → the container sleeps. A due-but-pending job is woken once per occurrence and re-checked on a bounded retry, so it can never spin.
Net: healing/wakeup is independent of user traffic, gated on real job activity,
and free when idle. Traffic may cold-start the container; jobs get a pre-warmed,
healed one. Requires the ACTIVITY binding in wrangler.jsonc and in the lerma
provisioner (tenant_provisioner.ex resource_bindings) + the
new_sqlite_classes migration (dispatch_namespace.ex @do_sqlite_classes).
Change-Aware Backups (feature-flagged)
Scheduled snapshots can be gated on whether tenant state changed. Two independent flags, both off by default:
| Flag | Effect |
|---|---|
BACKUP_CHANGE_DETECTION=true | Phase 1 (cost). The scheduled snapshot path (scheduleBackupIfNeeded) runs a container-side scan first. Unchanged state ⇒ no squashfs, no legacy staging, no lease. |
BACKUP_CHANGE_WATCH=true + BACKUP_WATCH binding | Phase 2 (RPO). The per-tenant BackupWatch DO checks every BACKUP_CHANGE_CHECK_MINUTES (2) and requests a durable snapshot from ActivityScheduler once a change has been quiet for BACKUP_CHANGE_DEBOUNCE_MS (60 s) and the floor (BACKUP_INTERVAL_MINUTES) has elapsed. |
Scan (src/change-detection.ts). Runs find over the real state roots (/home/openclaw, /root/openclaw/.openclaw, /root/clawd; the home directory is symlink-only for state, and find does not follow symlinks). It prunes the transient legacy stage and .npm. It compares against the baseline marker /tmp/.fouria-last-backup and prints unchanged, changed, or active (changed within the debounce window). Only entries inside the roots count (-mindepth 1), so a root’s own mtime, which moves when the legacy stage is created, cannot read as a change. Known limitation: a pure deletion with no accompanying write is not detected until the next write in the same tree, because find -newer sees mtimes, not removals. Every ambiguous outcome is changed: a missing baseline, no roots, a non-zero exit, a timeout, or a thrown error.
Baseline. /tmp/.fouria-pending-backup is stamped before a snapshot captures state and promoted to the baseline only after the catalog commits. A failed snapshot leaves the baseline untouched. This runs on every snapshot, whether or not a flag is set, so enabling a flag later has a valid baseline.
Watch (src/backup-watch/). watch.ts is the pure state machine. durable-object.ts is the adapter, and arm.ts holds gating and policy with no DO imports, so warm paths can call it without an import cycle.
- Never wakes a container: liveness is the non-waking
isRuntimeActive(). A not-running tenant is disarmed and not scanned. - Keepalive guard. Every scan is an
execthat renews the container’s sleep timer. A tenant unchanged forBACKUP_CHANGE_IDLE_DISARM_MS(10 min, below the staging 15 min sleep window) is disarmed, and re-arming never resets the idle clock. The watch can therefore delay a quiet container’s sleep by at most that window. - No starvation. An
activetenant defers snapshots for at mostBACKUP_CHANGE_MAX_DEFER_MS(10 min), so constant writes still get snapshotted. - No hot-path work. Snapshots are never taken from the watch directly. It enqueues the existing
ActivitySchedulerdurablesnapshotjob, which keeps that job’s gateway start, lease, retention, and restore-pending guards. - Arming happens on a running tenant only: the dispatch
backupcontrol action (/control) and the warmrefreshActivitySchedulepath (gateway healthy).
Wiring. wrangler.jsonc declares the BackupWatch class (migration v8) and the BACKUP_WATCH binding in every environment. In production the tenant’s bindings come from lerma, which binds BACKUP_WATCH only when the backup_change_watch provisioning flag is on (see apps/lerma). Both flags are plain-text vars that lerma always sets to true or false, so turning a flag off explicitly reverts it on redeploy.
Backup before deprovisioning
A container is never deprovisioned without a committed snapshot. The durable
backup-job queue (src/activity/backup-jobs.ts) gains a deprovision operation
(mode: 'sleep' | 'freeze'); the ActivityScheduler alarm runs it as
snapshot → stopIfIdle() (sleep) or snapshot → destroy() (freeze), and a
failed snapshot retries rather than tearing the container down (fail closed).
Reuse a snapshot from inside the idle window. hasRecentSnapshot() (a pure
catalog read in src/persistence.ts) answers whether the newest committed
archive is newer than the instance’s idle timeout (resolveSleepAfterMs():
staging 15m, dev 2h, production 6h) plus IDLE_SNAPSHOT_REUSE_SLACK_MS (5m, so
the archive finalized just before the previous stop attempt still counts).
onActivityExpired() uses it to stop
inline instead of enqueueing a snapshot when the container has already been idle
for a whole window — the filesystem cannot have changed, so a second archive is
pure cost. A read failure is treated as “no recent snapshot” (fail closed). This
also breaks the loop where the deprovision snapshot’s own RPCs renewed the
activity window, stopIfIdle() refused, and the next expiry snapshotted again
(observed on tenant-tarjetero-dev: three 422 MB archives per idle expiry while
the container never stopped).
Deprovision archives stamp the schedule. finalizeSnapshot receives
stampScheduled: true for deprovision jobs as well as trigger: 'scheduled'
ones, so a committed deprovision archive resets lastScheduledBackup and the
next cron tick does not archive the same state again.
Stop only on inactivity (stop-time re-check). The snapshot can take minutes
on a large tenant, so the container may be used again between enqueue and stop.
The sleep branch therefore does not call sandbox.stop() directly: it calls
TenantSandbox.stopIfIdle(), an RPC on the Sandbox DO (the activity authority).
That re-reads the DO’s own idle window — renewed by every request, including a
proxy request for an in-flight chat — and the container process list, and stops
only when the window has genuinely elapsed and no transient work is running. If
the tenant became active during the snapshot the job returns skipped
(detail: active:<reason>) and leaves the container running; the next idle
expiry enqueues a new job. Refusals: activity (window renewed),
transient-activity, probe-failed (unknown is not idle),
billable-production (never auto-sleeps), not-running.
- Idle sleep.
TenantSandbox.onActivityExpired()stops inline when a recent snapshot covers the idle window, and otherwise enqueues adeprovision(sleep) job, so the Sandbox DO leaving memory never loses an in-flight multi-minute snapshot. With noACTIVITYbinding it falls back to the previous stop-now path. - Freeze.
POST /controlfreeze enqueues adeprovision(freeze) job and acknowledges immediately (202 { ok, accepted: true, frozen: false, jobId }), failing closed at enqueue (409when a durable job is already active). The caller does not hold a multi-minute request open: it polls the tenant’sGET /api/health/backup/:jobIduntil the job reaches a terminal status. A snapshot failure surfaces as afailedjob and the container stays running, so lerma never marks a still-running instance frozen. - Platform stops (an inactivity timeout firing after the DO left memory, a container crash, or main-process exit) have no Worker hook; the periodic + change-watch snapshots bound that residual RPO.
The pollable endpoint (GET /api/health/backup/:jobId) carries no in-Worker
authentication — the same posture as /api/health — and is reachable by the
lerma control plane’s health-monitor service token through the health Access
tunnel (the *.fouria.io/api/health path application in
apps/fouria/iac/access.tf). Cloudflare Access path inheritance covers the
/backup/* sub-path, and it is also listed explicitly so the tunnel allowlist
documents the endpoint.
Health & Status Endpoints
GET /api/health is the unified health endpoint: it returns the startup/readiness/liveness
probes plus the canonical status (running|starting|stopped), processId and processType.
It requires no authentication inside the Worker, but at the Cloudflare edge it is
Access-protected by path-scoped applications (*.fouria.io/api/health) that carry
no role requirement: they admit any authenticated user (browser) and the lerma
health-monitor service token (headless).
GET /api/status, GET /api/admin/gateway/status/polling and GET /api/admin/gateway/status/ws are thin
aliases over the same snapshot. Browser clients use the sibling paths outside /api/health/* —
GET /api/health-ws (WebSocket stream every ~2s), GET /api/health-snapshot (HTTP snapshot), and
GET /api/health-components (Cloudflare component states) — while /api/health, /api/health/ws,
and /api/health/components remain mounted for headless service-token consumers (lerma). The
browser-facing sibling paths exist because /api/health and /api/health/* are covered by the
path-scoped FourIA Health - <slug> Access app whose immutable AUD differs from the host app’s: the
browser’s host-app session is invalid there, so Access answers with a 302 login redirect to
*.cloudflareaccess.com — which WebSocket handshakes cannot follow and browser fetch() calls
reject on cross-origin CORS.
By default /api/health warms the container (lerma keep-alive). ?passive=1 probes without warming
— the ROOT /_admin/diagnostics surface uses it so an operator can debug an instance whose container is
down. The admin dashboard subscribes to the health WebSocket (with an HTTP polling fallback) and never
blocks the UI on gateway status.
GET /healthz is a Kubernetes-style liveness probe: one non-waking
isRuntimeActive() read, always 200 { ok, ts, status } (status = active |
idle), never warming the container or probing the gateway. It exists because
/healthz otherwise fell through to the gateway proxy (app.all('*')), which
warmed the container and waited for the port — ~108 s on a cold instance, so the
platform killed the request before it answered. Readiness stays on /api/health.
A single /api/health request issues one gateway /health curl exec, shared by the
readiness and liveness probes. They previously each ran their own, so every poll spawned a
pair of identical curl processes (visible in /debug/processes) — a duplicate on the
frequent lerma probe. The process scan and the /health probe are now both fetched once, in
parallel, and reused across the snapshot.
Gateway memory (gateway: { pid, rssBytes, threads })
The gateway is one Node process in a 4 GiB / no-swap container whose RSS is dominated by
per-worker_thread V8 heaps that an in-process restart never reclaims
(docs/plans/sandbox-memory-usage.md). A dev tenant “rebooted” once and the only way to answer
“was it memory?” was to SSH in and read /proc, dmesg and the gateway log by hand.
src/gateway/memory.ts makes that a first-class signal:
| Piece | Behaviour |
|---|---|
buildGatewayResourceScript() | One shell exec: pgrep -x 'openclaw-gatewa' → gw:<pid>:<rssKb>:<threads> per live gateway server (same exact comm as the soft reload, so it never measures the openclaw gateway run launcher; zombies skipped). |
readGatewayResources() | Runs it (gateway.memory-read, bounded) and returns the highest-RSS process. Never throws: a failed read is null, not a broken health probe. |
sampleGatewayResources() | Throttles the read to GATEWAY_RESOURCE_SAMPLE_INTERVAL_MS (60s), so the frequent health poll issues at most one extra exec per interval and only a fresh sample writes a metrics row. A cached null is served inside the window too, so a broken probe is not retried every poll. |
maybeRestartGatewayForMemory() | The dev-gated watchdog (below). |
attachGatewayResources() in routes/public.ts attaches the sample to the snapshot as
gateway and writes a gateway-resources Analytics Engine row (double1 = RSS bytes,
double2 = threads, blob1 = gateway-resources, blob2 = customer, trailing
platform/environment blobs like every other cost row). It is applied by both /api/health
(and /api/health-snapshot) and the /api/health-ws stream, and only when the snapshot is
running — a sleeping/stopped container is never woken for a diagnostic.
Dev-gated memory watchdog. When the sample reaches GATEWAY_RSS_RESTART_BYTES (3 GiB of the
4 GiB container) on a dev instance (isDevAgentsEnabled: -dev hostname, OPENCLAW_DEV_MODE,
or local DEV_MODE), the gateway is replaced — killGateway + resetStartupCache +
ensureGateway({ skipRestore: true, bypassMaintenance: true }). That is the only mechanism that
actually returns RSS to baseline.
Design constraints worth keeping:
- The threshold guards the container limit, not OpenClaw’s warning. OpenClaw’s own
rss_thresholdis 1.5 GiB and fires permanently on a healthy-but-large dev gateway (~2.1 GiB observed, ~2.6 GiB peak). Using it as the action threshold would restart the gateway in a loop. - No soft-reload step. A measured in-process
SIGUSR2restart does not reclaim RSS (1.14 → 1.38 → 1.18 GiB), so reloading first would only add a failed attempt. skipRestore. A kill leaves the filesystem untouched; re-entering the R2 restore barrier would deadlock on the backup lock for no benefit.- Cooldown claimed before acting.
GATEWAY_MEMORY_WATCHDOG_COOLDOWN_MS(30 min) is recorded before the replacement, so two concurrent health polls cannot start two restarts. - Detached. The watchdog is dispatched through
executionCtx.waitUntil(or left detached when there is no context), so a multi-minute replacement never delays or fails a health response.
Every decision returns a stable machine reason (not-dev, no-resources, below-limit,
cooldown, rss-above-limit), and an actual restart logs a structured
gateway.memory.pressure event with pid, rssBytes, rssMib, threads and the limit.
Cloudflare component states
The dedicated GET /api/health/components endpoint returns a quick view of the
Worker-bound platform pieces rendered by the ROOT diagnostics horizontal
Cloudflare Components widget. GET /api/health also carries the same
(redacted) components entry — but only on the non-passive path. Lerma’s
HealthMonitor polls /api/health?passive=1 frequently and never reads
components, so the passive probe skips the component probe entirely (no
isRuntimeActive read, no R2 HEAD) to keep the frequent poll slim:
| Component | Probe |
|---|---|
| Container | Non-waking sandbox.isRuntimeActive() (up / sleeping / unknown) |
| Durable Object | Sandbox + ACTIVITY bindings present and the control channel answered |
| R2 | Bindings-shaped BACKUP_BUCKET plus a HEAD of backup-handle.json |
| AI Gateway | Present only when all three gateway credentials are configured (up = configured) |
Every probe is non-waking and individually bounded (3s), so the endpoint is
safe on a sleeping instance and can never hang the health response; on failure
the components entry is omitted instead of blocking. The AI Gateway is
inclusion-gated on credentials and reported as up (“configured”) — the probe
deliberately does not issue an authenticated Cloudflare API call on every poll,
and a missing account-scoped token must not read as a false outage.
Each component also carries binding-derived facts (a { label, value }[]
shown in the widget’s hover panel) and a Cloudflare dashboard logsUrl (a
per-product logs/metrics deep link opened on click). Both are derived from
Worker bindings/env only — never an authenticated Cloudflare API call — so
the surface stays non-waking and free (src/gateway/cloudflare-links.ts builds
the links: Container → Workers Logs/observability, Durable Object → DO
metrics/logs, R2 → the BACKUP_BUCKET_NAME bucket page, AI Gateway → the
gateway page). When the account id is not bound the links fall back to the
dashboard’s documented ?to=/:account/… form; a component without a resolvable
link renders as a plain (non-clickable) card. The unified GET /api/health
snapshot redacts facts/logsUrl (redactComponentDetails) — it is a
broadly-consumed public probe — and serves only the state summary; the enriched
view is exclusive to GET /api/health/components.
On phones the widget collapses into a compact status widget that shares a
row with the gateway and persistence-lock status widgets at the top of the
diagnostics page (not floated or overlaid): a single pill carrying the Cloudflare
icon, a worst-case rollup status dot and an up/total summary. Tapping it
expands the same component strip as a compact panel. To build that parallel row,
AdminLayout omits the topbar gateway + persistence indicators on the
diagnostics page on phones (App.tsx) and DiagnosticDashboard renders
GatewayStatusLed + PersistenceLockIndicator beside the Cloudflare widget. The
gateway pill gains an identifying icon and the lock an
Idle/Backing up/Restoring label (its color still encodes the state); both
use compact phone sizing so the three widgets stay on one line on normal phones
and wrap gracefully on very narrow ones. The desktop layout is unchanged — the
topbar keeps its indicators and the Cloudflare card renders full-width.
The ROOT /_admin/diagnostics page renders its operational tools as separate
clickable Fouria DevOps widget cards (backup connectivity + configuration
editor + paired devices). Each card’s title row toggles its body while the
action buttons stay live, and the cards load their data asynchronously and
render their chrome immediately — there is no blocking loading animation for the
data that populates them. On phones each option collapses into a vertical
icon + name menu row (collapsed by default; tap to open) with the option’s
actions revealed inside the opened body. The Configuration Editor reflows to the
viewport: action buttons stack full-width, the section tabs scroll horizontally,
long JSON wraps, and the raw textarea shrinks. The Debug Tools grid drops to a
single column so nothing forces sideways scrolling, and the admin bottom nav
uses flexible items (with safe-area insets) so every nav entry fits. The
floating dev role switcher (DevUserSelectorCompact) is lifted above the bottom
nav on phones so it neither covers the last nav item nor overflows the viewport.
The gateway status indicator lives in the admin topbar beside the
persistence-lock indicator and the notification bell (GatewayStatusLed, mounted
in App.tsx’s AdminLayout). Its color-coded LED reflects the gateway status
(running/starting/sleeping/stopped/unknown); clicking it reveals the
complete status value (status, process, process type, the startup/readiness/
liveness probes, restore state and last update) and a reboot icon that
performs the same soft-reload/restart as the removed Gateway Controls banner. The
reboot action is hidden behind an Admin badge when /debug/* is disabled,
matching the previous gating.
Live gateway detection
On the Worker-driven cold start the gateway is a supervised sandbox.exec
process (/bin/bash -lc 'exec openclaw gateway run …'), so it is visible to
listProcesses() for the life of the container. The legacy full mode (manual
runs, test:gateway:boot) still launches it as a detached background child of
start-openclaw.sh (… &), and a container provisioned by an older image may
still be serving a detached gateway during a rolling deploy — for those, after
the script exits the gateway binary is invisible to listProcesses().
To avoid false no_process reports on log/status surfaces, those endpoints
(GET /api/admin/logs, GET /debug/logs, GET /debug/stop-gateway,
POST /api/admin/gateway/restart, GET /api/admin/audit-logs) call
findExistingGatewayProcess(sandbox, { probeLive: true }), which falls back to
a bounded /health probe and — when the gateway responds — returns a synthetic
gateway-live process whose logs are read from /tmp/gateway-stdout.log. The
passive health snapshot path deliberately does not probe so a down container
fails fast.
Log Levels
| Level | When to use |
|---|---|
debug | Local development |
info | Production default |
warn | High-traffic production |
error | Security-sensitive production |
TypeScript & Tooling
Type Generation
Worker types are auto-generated:
bun run types # wrangler types → worker-configuration.d.ts
Never manually edit worker-configuration.d.ts — it’s regenerated from wrangler.jsonc.
Testing
bun run test # Unit tests (Vitest)
bun run test:coverage # With coverage report
bun run test:client # SPA component/page tests (jsdom)
bun run test:ui # Playwright UI suite (Vite dev server + mocked API)
bun run test:ui:headed # Same, with a visible browser
bun run test:e2e # E2E tests (cctr + Playwright)
bun run test:e2e:cli # CLI-only E2E (cctr)
The Playwright UI suite (test/ui/, config playwright.config.ts) drives the
real SPA served by bun run dev with per-test /api/* mocks — no Worker,
container, or Cloudflare credentials. It covers the app shell, role-based
navigation and redirects, every admin page’s render, and the notification bell.
See docs/TESTING.md for the
full description and the Nix PLAYWRIGHT_BROWSERS_PATH notes.
Formatting & Linting
bun run format # oxfmt (auto-format)
bun run format:check # Check formatting
bun run lint # oxlint
bun run typecheck # tsc --noEmit
Common Tasks
Adding a New API Endpoint
- Create route handler in
src/routes/api/<domain>.ts - Register in
src/routes/api.ts - Add client method in
src/client/api.ts - Write tests (unit + integration)
- If new dependencies: update
package.json
Adding a New Environment Variable
- Add to
OpenClawEnvinterface insrc/types.ts - If passed to container: add to
buildEnvVars()insrc/gateway/env.ts - Update
.dev.vars.example - Document in ../../docs/ENV-VARIABLES.md