FourIA — Worker Dashboard Architecture
On this page
Deep-dive into the apps/fouria component, the core Tenant Worker of FourIA.
See ../docs/DEVELOPMENT.md for project-level overview and ../docs/ENV-VARIABLES.md for environment variable reference.
Architecture Overview
Browser
│
▼
┌─────────────────────────────────────────┐
│ Cloudflare Worker (index.ts) │
│ │
│ Middleware: Logging → Auth → RBAC │
│ │
│ Routes: │
│ /api/* → Hono API routes │
│ /_admin/* → React SPA (Vite build) │
│ /debug/* → Debug endpoints (ROOT) │
│ /* → Proxy to OpenClaw GW │
│ │
│ Activation: Worker restore → onboard │
│ → patch → launch gateway │
└──────────────┬──────────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Cloudflare Sandbox Container │
│ │
│ start-openclaw.sh: │
│ 1. openclaw onboard (auth setup) │
│ 2. Enable plugins (unified billing) │
│ 3. moltlazy patch (config injection) │
│ 4. Start Cloudflare Tunnel (optional) │
│ 5. background gateway + health poll │
│ │
│ ┌────────────────────────────────────┐ │
│ │ OpenClaw Gateway │ │
│ │ - Control UI on port 18789 │ │
│ │ - WebSocket RPC protocol │ │
│ │ - Agent runtime │ │
│ └────────────────────────────────────┘ │
└─────────────────────────────────────────┘
Frontend Architecture (Vite + React SPA)
The dashboard is a React 19 SPA built with Vite 6 and served from /_admin/.
Build & Dev Flow
bun run dev → Vite dev server on :8787 (vite.ui.config.ts)
│ Hot-reload for frontend changes
│ Proxies /api/*, /debug/* to :8788 (worker)
│
bun run dev:worker → Vite dev server on :8788 (vite.config.ts)
│ Runs the actual Worker in workerd via the
│ Cloudflare Vite plugin; Worker changes hot-reload
│
bun start → bun run build:docker && vite dev
Full local stack (build + Worker dev server)
Key insight: both dev servers are Vite.
bun run devonly handles the admin SPA; the Worker (routes, auth, gateway, DOs, containers) runs under the Cloudflare Vite plugin viabun run dev:worker, the same plugin used byvite build. The plugin has nowrangler devflags: local E2E vars are injected through itsconfighook and.dev.varsis loaded from this directory.
Vite Configuration
// vite.config.ts — Worker dev server (bun run dev:worker)
base: '/_admin/' // All assets served under /_admin/
server: {
port: 8788,
allowedHosts, // DEV_ALLOWED_HOSTS (tunnel hostnames)
}
// vite.ui.config.ts — admin SPA dev server (bun run dev, :8787)
server: {
port: 8787,
proxy: {
'/api': 'http://localhost:8788',
'/debug': 'http://localhost:8788',
'/cdp': 'http://localhost:8788',
},
}
Exposing the Worker through a tunnel (e.g. Telegram webhooks): Vite blocks unknown
Hostheaders, so pass the public hostname explicitly —DEV_ALLOWED_HOSTS=foo.trycloudflare.com bun run dev:worker(comma-separated; a leading dot allows a whole domain). Point the tunnel at:8788, not the SPA at:8787(it does not proxy/telegram-webhook).
Client Module Structure
src/client/
├── App.tsx # Root component with React Router
├── main.tsx # Entry point — mounts React app
├── api.ts # Typed API client → /api/admin/*
├── gateway-gate.ts # Health circuit breaker (park/resolve gateway queries)
├── GatewayHealthContext.tsx # App-wide passive health subscription + gate wiring
├── pages/ # Route-level pages
│ ├── HomePage.tsx # Dashboard overview
│ ├── AgentsPage.tsx # Agent management
│ ├── ChannelsPage.tsx # Connections/integrations
│ ├── KnowledgePage.tsx # Knowledge base/wiki
│ ├── AccountPage.tsx # User/team management
│ ├── IntegrationsPage.tsx # Integration config
│ ├── SecretsPage.tsx # Vault management
│ ├── BackupsPage.tsx # Backup management
│ ├── DiagnosticDashboard.tsx # Costs + diagnostics
│ ├── AuditLogsPage.tsx # Gateway ledgers: audit activity + background tasks
│ └── AutomationEditorPage.tsx # Automation workflows
├── components/ # Shared UI components
└── hooks/ # Custom React hooksFrontend ↔ Backend Communication
The frontend (src/client/api.ts) calls /api/admin/* endpoints. These are served by the Worker’s Hono routes (src/routes/api.ts), which:
- Verify RBAC (
requireWriteRole()for mutations) - Execute OpenClaw CLI commands inside the Sandbox container
- Parse CLI output and return JSON
Client circuit breakers (health-gated gateway queries)
Two client-side circuit breakers keep the dashboard responsive when the OpenClaw gateway is not ready (issue #371):
- #1 — gateway queries are asynchronous. Pages paint immediately and never block first paint on a gateway response; mutations are fire-and-notify through
NotificationProvider.api.tsroutes every gateway-touching call throughgatewayApiRequest. - #2 — gateway queries require the passive health stream to be UP. The single app-wide
GatewayHealthProvider(src/client/GatewayHealthContext.tsx) subscribes once to/api/health-ws?passive=1and drives a module-singletonGatewayGate(src/client/gateway-gate.ts). UP is defined assnapshot.readiness.ok === true;starting/stopped/sleepingare not UP (sleepingis healthy-idle but the gateway is not running). Before dispatching,apiRequestawaitswhenGatewayUp(signal); while down the query is parked and theGatewayGateNoticeindicator shows, then the queue flushes automatically on the next UP snapshot. A transient (503/transport) failure is retried once after the next UP.
The gateway-touching boundary is the GATEWAY_TOUCHING_PREFIXES manifest in api.ts (/agents, /models, /channels, /config, /devices, /integrations, /secrets, /knowledge, /audit-logs, /dreaming, /gateway/restart). Diagnostics (/debug/*, /logs, /gateway-api, /gateway/status/polling, /gateway-token, /version) and worker-local surfaces (/storage, /costs, /billing, /users, /vault, /releases, /fouria-docs, /whoami) are deliberately not gated, so an operator can still inspect a down gateway and non-gateway features keep working.
Diagnostics (DiagnosticDashboard). The ROOT dashboard is explicitly skip-not-park: useGatewayRunning() (GatewayHealthContext.tsx) reports snapshot.readiness.ok ignoring the ROOT bypass, and the page only dispatches its container reads (/config/raw, /config) while it is true; they re-run automatically when the gateway transitions back to running. The debug-routes flag is Worker-local and still loads. Every call is dispatched after paint (scheduleAfterPaint → requestIdleCallback, setTimeout(0) fallback) and the widgets render their waiting state first, so nothing is blocked on a container response: the Configuration Editor renders its tabs/editor with a non-blocking “Loading configuration…” / “Waiting for the gateway to start…” notice (gatewayReady prop on ConfigSection) and the device list shows a waiting state until Refresh. Gateway-only actions (device list/approve, config save, channel probe) are disabled/guarded while down; doctor --fix repair is NOT one of them: it is the recovery action for a broken/down gateway (the backend stops the gateway itself), so a dedicated Repair Environment control sits outside the Configuration Editor (always rendered for ADMIN/ROOT, disabled only while a repair is in flight) and the in-editor repair button stays enabled while the gateway is down. The Debug Tools that call the container (logs, processes, version, sqlite integrity) and the recovery actions (restart/rescue/destroy) stay available on purpose — they are how an operator inspects and revives a down gateway — while the ones that require a running gateway (gateway-api, stop-gateway) are skipped with an inline explanation.
ROOT bypass. ROOT operators are never gated: App.tsx passes bypass={whoami?.role === 'ROOT'} to the provider, which flips the gate’s bypass (GatewayGate.setBypass). This preserves the “ROOT can always reach a tenant” rule and keeps the operator’s non-passive device-pair request (GET /api/admin/devices/ws → listDevices → ensureGateway({ waitForReady: false })) able to wake a sleeping container. The notification bell is passive for every non-ROOT role (subscribeDeviceNotifications(..., { passive: true }) → ?passive=1), so a regular dashboard load never wakes a sleeping tenant; only ROOT’s stream is non-passive. Server-side, a passive /api/admin/devices/ws first does the non-waking readRuntimeActive() read and, when the runtime is not live, emits an empty snapshot and skips the gateway event stream. Once the ROOT wake brings the gateway UP, the shared health stream flips to up and every parked query (for all roles) flushes. The GatewayGateNotice is suppressed while bypassed, since ROOT actions are never queued.
The gate is fail-open when no provider is mounted (non-SPA callers, unit tests): the module default is up. The provider installs the shared gate during its first render, before any child page effect runs. It is a module singleton rather than a useState initializer because React StrictMode double-invokes initializers in dev — two gates would leave api.ts parked on a different instance than the provider drives.
Backend Architecture (Hono Worker)
Request Pipeline
Incoming Request
│
├─► Logging middleware (all routes)
├─► Sandbox init (getSandbox() into context)
│
├─► Public routes (no auth):
│ /sandbox-health
│
├─► CF Access Auth middleware:
│ Extract JWT → validate → set accessUser
│
├─► RBAC resolution:
│ resolveCustomerRole → set userRole, customerSlug
│
├─► Route dispatch:
│ /api/* → API routes (auth + RBAC)
│ /_admin/* → Admin UI (SPA assets)
│ /debug/* → Debug routes (ROOT only)
│ /* → Catch-all proxy to gateway
│
└─► Gateway proxy:
HTTP proxy → containerFetch(port 18789)
WS proxy → WebSocket relay with token injectionAPI Route Structure
// src/routes/api.ts — route aggregation
api.route('/admin', adminApi)
├── /whoami, /version, /logs, /gateway-api, /debug-status, /gateway-token (misc)
├── /devices/* → devicesRoutes
├── /storage/* → storageRoutes
├── /releases/* → releasesRoutes
├── /gateway/* → gatewayRoutes
├── /config/* → configRoutes
├── /cron/* → cronRoutes
├── /dreaming/* → dreamingRoutes
├── /users/* → usersRoutes
├── /integrations/* → integrationsRoutes
├── /knowledge/* → knowledgeRoutes
├── /secrets/* → secretsRoutes
├── /agents/* → agentsRoutes
├── /models/* → modelsRoutes
├── /channels/* → channelsRoutes
├── /costs/* → costsRoutes
├── /vault/* → vaultRoutes
├── /billing/* → billingRoutes (plan, checkout, portal, invoices, freeze-status, freeze, unfreeze)
└── /audit-logs/* → auditLogsRoutes (audit activity ledger + task ledger)
// Write gate: POST/PUT/DELETE/PATCH require ADMIN or ROOT
// Read gate: GET/HEAD available to all authenticated roles
Build identity.
GET /api/admin/versionandGET /debug/versionreport the FourIA Worker’s ownfouria_version(component CalVer fromapps/fouria/package.json) andfouria_commit(short git SHA) alongside the container’s OpenClaw/Node versions. Both are injected into the bundle byvite.config.ts(__FOURIA_VERSION__/__FOURIA_COMMIT__, consumed insrc/build-info.ts;GITHUB_SHAwins in CI,git rev-parse --short HEADlocally). The DiagnosticDashboard Version Info tool surfaces them for operators.
Cloudflare error and trace correlation
The Worker emits structured worker.request.started and
worker.request.completed records with the method, path, status, duration, and
Cloudflare Ray ID (cfRay). Unhandled exceptions are logged as
worker.request.failed with the exception name/message/stack and the same
request fields. Cloudflare also attaches its invocation $metadata.requestId
and $metadata.traceId to each Worker log, so the invocation’s request, logs,
and trace can be opened together in Workers Observability. Error records pass an
Error object to console.error as well, so Cloudflare populates its exception
metadata (including $metadata.error) alongside the indexed event fields.
Sandbox command failures are emitted as sandbox.process.failed. These records
include a stable operation label and phase, the Sandbox processId when a
process was launched, and available exit-code/timeout/duration fields. The
processId is the join key to the processId field on the Containers dataset’s
process.complete event; inspect exitCode and timedOut there rather than
trusting the generic event message, which can say success when the process
itself exited non-zero. Raw shell commands, stdout, and stderr are deliberately
omitted because they can contain tenant config or credentials.
Persistence failures use stable events such as persistence.backup.failed,
persistence.config-version.failed, persistence.restore.failed, and
storage.restore.failed. Config-version and restore errors include backupId
when an archive exists; use it to join to the backup.create / backup.restore
container events. A config-version capture failure is best-effort: it identifies
the archive whose supplemental config snapshot was not captured, but does not
mean the archive itself failed to be created.
Backup and restore work also emits paired persistence.phase records. Each
record has an operationId, operation (backup or restore), phase, and
status (started, completed, or failed); terminal records include
durationMs, and phases after an archive handle exists include backupId.
source names the route or job, cfRayId correlates request-triggered work with
worker.request.*, and jobId correlates work running later in the
ActivityScheduler. Use operationId to group the phases; if a started phase has
no terminal record, the Worker may have been interrupted during that phase.
Useful phases include sandbox.createBackup, archive.encrypt,
catalog.commit, sandbox.restoreBackup, and legacy-state.materialize.
For a failure investigation, filter the cloudflare-workers dataset to
event = "sandbox.process.failed", record processId and the Worker
$metadata.requestId/$metadata.traceId, then search the containers dataset
for the same processId (and the tenant’s $container.applicationId). For a
request-level failure, filter to event = "worker.request.failed" or use
cfRay to pivot to the matching edge request. For backup/restore CPU-limit
investigations, filter to event = "persistence.phase", find the last
started event for the relevant cfRayId or jobId, then group by its
operationId to see which phases completed before it.
Worker observability export destinations
Account-level Cloudflare Observability → Destinations export this Worker’s
logs and traces to Lerma over OTLP/HTTP JSON (issue #337). The destination names
are referenced in the observability block:
// staging
"observability": {
"logs": { "destinations": ["lerma-worker-logs"], "head_sampling_rate": 0.1, "persist": false },
"traces": { "destinations": ["lerma-worker-traces"], "head_sampling_rate": 0.1, "persist": false }
}
// production
"observability": {
"logs": { "destinations": ["lerma-worker-logs-prod"], "head_sampling_rate": 0.1, "persist": false },
"traces": { "destinations": ["lerma-worker-traces-prod"], "head_sampling_rate": 0.1, "persist": false }
}
The -prod destinations point at https://lerma.paso4.io/api/ingest/otel/v1/*;
the un-suffixed pair points at https://lerma-dev.paso4.io/....
apps/fouria/wrangler.jsonccarries the matching names in each env block (stagingandproduction, direct deploys).- WFP-deployed tenant workers get the names from Lerma’s
Lerma.Cloudflare.Observability.worker_metadata/0(WORKER_OBSERVABILITY_LOGS_DESTINATIONS/WORKER_OBSERVABILITY_TRACES_DESTINATIONS). The CD sets the staging names for staging and the-prodnames for production, so a redeploy inherits them. - Lerma is the destination of record:
persist: falsestops the duplicate Cloudflare-side copy, and the head sampling rate (10% at rollout) bounds export volume. Lerma drops message text, stacks, URLs/queries, user agents, request/response bodies and DB query text at ingestion; only allowlisted metadata is stored. - Cloudflare gzips each OTLP/HTTP JSON batch (
Content-Encoding: gzipwithContent-Type: application/json).LermaWeb.Plugs.OtlpBodyReaderruns beforePlug.Parserson the ingest routes and decompresses it; without that step the normal JSON parser returns400and the destination retries forever. - The export only feeds diagnostics + the Lerma instance Worker telemetry
widget. It never replaces
HealthMonitor(still the source ofinstances.status) or billing, and it is not a probing path — no container is woken to produce it.
Ops: create the two destinations under Observability → Destinations with the
custom header Authorization: Bearer <OTEL_INGEST_TOKEN>, set the token and the
destination names on Lerma (staging first), then redeploy the tenants. Delivery
status shows Last: n minutes ago; the uptime numbers come from
GET /api/analytics/v1/uptime/<script_name>?window=24h.
Gateway start lifecycle + stuck-start watchdog
A cold start is one shared promise (ensureGateway) spanning the restore
barrier → the supervised start-openclaw.sh start launch → the port wait. A
wrapper that never execs the gateway, or a container control-channel drop, can
leave that promise “in progress” for many minutes while every request logs
Gateway start already in progress … waiting... and waits on it.
The start is now tracked per sandbox key with a phase (probe → restore →
launch → port-wait) and a per-phase timer. post-start runs detached
after readiness (see below), so it is deliberately not part of the awaited start
promise:
gateway.start.begin/gateway.start.phasestructured events record the attempt, phase, elapsed time, and (after launch) the process id; the “already in progress” log carriesphase,age, andattempt.isGatewayStartStuckflags a phase that overruns its budget +GATEWAY_START_STUCK_GRACE_MS(therestorephase getsGATEWAY_RESTORE_PHASE_BUDGET_MSbecause the legacy stage may take 10 min; all other phases useSTARTUP_TIMEOUT_MS).- When the watchdog trips,
recoverStuckGatewayStartemits agateway.start.stuckstructured event, kills the stuck supervised process (killGateway), clears the startup/readiness caches, and returns so the next request starts clean. AfterGATEWAY_START_STUCK_MAX_ATTEMPTSconsecutive resets it reprovisions the container (recoverStaleSandbox). - Separately,
findExistingGatewayProcessusesclassifyGatewayStartto detect astart-openclaw.sh startwrapper older than the cold-start budget with the port closed and kills it immediately (sourcestartup-wrapper-age) instead of waiting out another full readiness budget. - A non-owner isolate that merely observes a running wrapper waits to the
same deadline the owner’s watchdog uses —
STARTUP_WRAPPER_STUCK_BUDGET_MS(STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS) measured from the wrapper’s start — and re-probes/healthbefore any kill. A fixedSTARTUP_TIMEOUT_MSmeasured from observation expired while the wrapper had alreadyexeced the gateway and was still binding, killing it mid-boot, releasing the startup lock, and making the waiting second wrapper re-run the entire setup (observed onstaging-client-dev: a ~14-minute cold boot). The kill now only fires once the wrapper has genuinely overrun the owner’s own budget with the port still closed.
The port-wait also completes on a positive /health probe. The SDK’s
waitForPort drives a container-side port-watch + log-readiness subscription
that can hang indefinitely on some boots even after the gateway is serving.
Waiting on it alone let the shared start promise overrun its budget, so the
watchdog force-killed a healthy gateway and reprovisioned the container
(observed on tenant-client-dev: gateway.start.stuck phase=port-wait after
368 s while HTTP proxying returned 200). The wait races waitForPortResilient
against the same positive /health probe the public proxy uses. If that waiter
reaches its deadline and parks, the watchdog makes one fresh bounded health
check before resetting: a late-positive response emits
gateway.start.late-ready and is adopted without killing the gateway. A failed
health RPC is treated as unknown, not closed, and defers the reset until a
later request can probe again; only a confirmed closed port follows the normal
stuck-start reset path.
Process count is not a health signal. listProcesses() returns every
tracked process for the container’s lifetime — including exited/error
entries the Worker never garbage-collects — so the raw length only grows. The
log line now reads listProcesses returned <total> processes (<running> running)
and every health decision (classifyGatewayStart, the gateway matcher) considers
only running entries. The SDK exposes no completed-process cleanup, so the
list is append-only; never gate health on the total.
Gateway protocol ledgers (/api/admin/audit-logs)
The AuditLogsPage reads the two append-only ledgers from the Gateway protocol
(docs.openclaw.ai/gateway/protocol/ledgers):
GET /api/admin/audit-logs— audit activity ledger (audit.activity.list,operator.read), newest first. SupportsagentId,sessionKey,runId,kind,status,direction,channel,after,before,limit, andcursor; returns{ events, nextCursor? }. The bundledadmin-http-rpcplugin allowliststasks.*but notaudit.*, so this read goes over the Gateway WebSocket RPC using OpenClaw’s reference client (@openclaw/gateway-client/browser+@openclaw/gateway-protocol) bridged throughsandbox.wsConnect, authenticating with the shared gateway token as anoperator(seesrc/gateway/gateway-rpc.ts). It falls back toopenclaw audit --jsonwhen the WS path is unavailable (older Gateway); the CLI fallback gets the resolvedOPENCLAW_GATEWAY_TOKENinjected because a freshsandbox.execshell does not inherit the gateway process env.GET /api/admin/audit-logs/tasks— task ledger (tasks.list,operator.read), newest first. Supportsstatus(repeatable),agentId,sessionKey,limit,cursor, andsortBy; returns{ tasks, nextCursor? }. RPC-first over the admin-http-rpc plugin, with anopenclaw tasks list --jsonCLI fallback (same token injection).
Both surfaces are ADMIN-only (absent from DIAGNOSTICS_PATHS) and return an
empty page with an error string when the ledger is unavailable, so the
dashboard degrades gracefully instead of failing the request.
Why a WebSocket client.
admin-http-rpconly exposes an allowlisted method subset. Methods outside it (notablyaudit.*,plugins.*,secrets.*,agents.files.*) require the wire-v4 Gateway WebSocket handshake (connect.challenge→connect→hello-ok).src/gateway/gateway-rpc.tsadaptssandbox.wsConnectto the reference client’sGatewayProtocolSocket; the client version sent in the connect frame is injected at build time fromcatalog.openclaw(__OPENCLAW_VERSION__).
RPC-first agent board + integrations
Eligible dashboard routes try Admin HTTP RPC first, the official Gateway
WebSocket RPC second, and keep the CLI as the final fallback for older
Gateways (helpers return null/false on transport or method failure):
- Agent board (
/api/admin/agents) — list readsagents.listand channel bindings inconfig.getover Admin HTTP RPC. If either result is unavailable, the Gateway WebSocket client fetches the missing read(s) over one connection; the CLI remains the final fallback. Create →agents.create; delete →agents.delete; identity (name/emoji/avatar) →agents.update; model (+ fallbacks) →config.patchonagents.entries.<id>.model. Explicit bind/unbind operations have no matching RPC method and stay on the CLI; persona read/write are filesystem operations. - Devices (
/api/admin/devices) — list, approve, reject, and approve-all retain Admin HTTP RPC as the first path, use Gateway WebSocket methods (device.pair.list,.approve,.reject) if HTTP RPC is unavailable, and retain the CLI as the compatibility fallback. - Models (
/api/admin/models) and channel probe (/api/admin/channels/probe) —models.listandchannels.statususe Admin HTTP RPC first, then Gateway WebSocket RPC, then the existing CLI path. - Integrations (
/api/admin/integrations) — the whole endpoint is WebSocket-free.GET /reads the allowlistedchannels.status+agents.listover the admin HTTP RPC, and resolves plugin enabled-state fromconfig.get(plugins.entries.<id>.enabled) merged with the staticPLUGIN_FIELDScatalog — noplugins.listsocket. When no channel accounts are configured the supported-channel catalog is synthesized locally fromCHANNEL_FIELDSinstead of shelling out toopenclaw channels list --all. Plugin/channel toggles and non-secret configure fields →config.patch; secrets reload → theopenclaw secrets reloadCLI.secrets.applyand agent bindings have no 1:1 RPC method and stay on the CLI. CLI fallbacks remain for a degraded HTTP RPC transport. The admin RPC fetcher also cancels a latecontainerFetchbody when its timeout wins, preventing the Sandbox DO’s “ReadableStream received over RPC disconnected prematurely” warning and the churn that dropped later sockets. The same cleanup is shared by the public proxy and thegateway-api/diagnostics probes viasrc/utils/container-rpc.ts(containerFetchWithTimeoutcancels a late body,wsConnectWithTimeoutcloses a late socket). Reading a running gateway’s logs must not open its live SDK stream (it never settles and is later disconnected):GatewayProcess.getLogs()tails/tmp/gateway-stdout.log, and/debug/processes?logs=true//api/admin/logs?id=refuse a live stream for a still-running process. The catch-all proxy logs only an allowlisted header subset (safeHeaderLog) — never the forwarded map, which carries CF Access credentials and cookies. - Config / secrets / agent resolution — config set/unset and dreaming
settings use
config.patchover Admin HTTP RPC, then Gateway WebSocket RPC, then the CLI fallback.POST /api/admin/secrets/reloadusessecrets.reload(WebSocket);resolveAgentFlag(shared by themodels/knowledgeCLI calls) resolves the roster withagents.listover Admin HTTP RPC, then WebSocket RPC, then CLI.
secrets applyhas no--yes. The current CLI accepts only--from <plan.json>,--dry-run,--allow-exec, and--json; the plan itself is the confirmation. Bothintegrationsanddreamingapply without--yes.
Agent Studio (/_admin/agents)
The Agent Garage is an all-in-one, RPC-only workbench (roster + Overview, Capabilities, MCP, Memory, Automations, Connections, Health). The old character-builder page was replaced; Channels/Integrations/Automation remain the platform-wide configuration surfaces.
The studio never shells out to openclaw and never runs cat/find/stat:
| Surface | Gateway method | Transport |
|---|---|---|
| Agent entries, bindings, channels, MCP, plugins, skills declarations | config.get | Admin HTTP RPC (WS fallback) |
| Roster / identity / workspace | agents.list | Admin HTTP RPC (WS fallback) |
Bootstrap (AGENTS.md, SOUL.md, …) | agents.files.list / .get / .set | WS |
| Durable notes + dreaming files | agents.workspace.list | WS |
| Tool catalog | tools.catalog | WS |
| Effective surface for a session | tools.effective (sessionKey required) | WS |
| Skills, workshop, trust, required bins | skills.status, skills.workshop.list, skills.securityVerdicts, skills.bins | WS |
| Hooks | hooks.status | WS |
| Plugins + runtime state | plugins.list | WS |
| Sessions + store/WAL sizes | sessions.list, sessions.storage.status | WS |
| Jobs + run history | cron.list, cron.status, cron.runs | Admin HTTP RPC (WS fallback) |
| Memory provider | memory.status | WS |
| Node/process/disk | system.info | WS |
| Update / backups | update.status, backup.status | HTTP / WS |
| Activity (future) | audit.activity.list | WS |
Endpoints (all ADMIN-only, mounted at /api/admin/agent-studio):
GET / (instance + roster), GET /instance, GET /:agentId
(?sessionKey= selects the effective session), GET|PUT /:agentId/files/:name,
GET /:agentId/automations, POST /:agentId/automations/:jobId/:action,
PUT|DELETE /:agentId/bindings, POST /plugins/:pluginId/:action.
The dedicated automations read uses the same job normalization as the agent
detail: schedule is always display text, never the Gateway’s structured
{ kind, everyMs, anchorMs } object. This prevents React error #31 when opening
the Automations pane. The shared gateway status indicator queries the ROOT-only
/api/admin/debug-status endpoint only for ROOT users; ADMIN users retain the
health indicator without an unauthorized debug-status request.
Health lists plugins in pages of 10 with Previous/Next controls and a total count. Pagination is client-side (no additional RPC reads), resets when switching agents, and hides its controls when the list fits on one page.
Capabilities uses independent 10-item pages for Allowed, Denied, Subagents, Skills, Available tools, Effective surface, and discovery Notices. Tool pages count individual tools across groups while preserving their group headings. Filtering applies before pagination and resets the skills/catalog/effective pages; changing agents resets all capability pages. No extra RPC reads are made.
Reads are non-waking. The route checks isRuntimeActive() first and returns
{ available: false, reason: "sleeping" } before opening a socket, so a
sleeping tenant is never started just to paint the dashboard. Every endpoint
opens one withGatewayRpcClient connection for its WS methods and guards each
call independently, so one unsupported method only removes its pane (e.g.
skills: { known: false }) instead of failing the page. Core reads use the admin
HTTP RPC when available and fall back to config.get/agents.list over WS.
Writes: bootstrap files use agents.files.set with the hash from
agents.files.get (409 agent_file_conflict on a stale hash); bindings are a
config.patch on the top-level bindings array (no openclaw agents bind);
automation run/enable/disable use cron.run/cron.update; plugin
enable/disable/refresh use plugins.setEnabled/plugins.refresh with
operator.admin.
RPC gaps (deliberately rendered as unknown, never faked): there is no MCP
probe/reload method, so MCP health is the last-known tools.effective inventory
plus its discovery notices; there is no binary-presence method, so the Health
pane lists skills.bins required binaries and per-skill missing requirements
rather than claiming presence.
Tests: src/agents/studio.test.ts (pure normalizers/builders),
src/routes/api/agent-studio.test.ts (WS/HTTP method map, per-call degradation,
sleeping short-circuit, secret redaction, file conflict, bindings via
config.patch), src/client/pages/__tests__/role-isolation.test.tsx
(BASE_USER/ADMIN), and test/ui/agent-studio.spec.ts (roster, tabs, file editor,
sleeping state).
Realtime device-pairing notifications (/api/admin/devices/ws)
The dashboard NotificationBell subscribes to GET /api/admin/devices/ws:
- On connect it pushes a
{ type: "device.pair.snapshot", pending, paired }frame (best-effort, non-blocking), then relays Gatewaydevice.pair.requested/device.pair.resolved/device.pair.changedevents as{ type: "device.pair.event", event, payload }. - The Gateway subscription uses
openGatewayEventStream(the official@openclaw/gateway-clientbrowser client oversandbox.wsConnect), which reconnects with backoff until the browser socket closes. - Worker-hosted notification sockets rotate after 10 minutes with close code
1012; this is intentionally below the observed Cloudflare load-shed lifetime for long-lived Worker invocations. The dashboard reconnects and receives a fresh snapshot, releasing the old Worker invocation and Gateway subscription. - Admins approve/reject directly from the bell:
POST /api/admin/devices/:requestId/approve/.../reject(RPC-firstdevice.pair.approve/device.pair.reject, CLI fallback). The client helper issubscribeDeviceNotificationsinsrc/client/api.ts.
Dashboard notification centre. NotificationBell also renders application
notifications (backup created/failed, backup restored/failed) from the
NotificationProvider context (src/client/NotificationContext.tsx) that wraps
the admin layout. Long-running admin actions push their result there instead of
blocking on a modal, so the outcome still reaches the admin if they navigate away
before the request settles. The bell badge counts unread app notifications plus
pending device-pairing requests; opening the panel marks app notifications read
and each can be dismissed.
Persistence-lock indicator. PersistenceLockIndicator (rendered beside the
bell) shows a closed red lock while a dashboard-initiated backup or restore
is in flight, and an open green lock otherwise. The state comes from
PersistenceLockContext (src/client/PersistenceLockContext.tsx): the
BackupHistoryPanel raises it (begin('backup' | 'restore')) when it fires an
async snapshot/restore and drops it (end()) when its completion check settles
(success, failure, or timeout), so the indicator is synchronous with the
operation lifecycle — there is no polling and no lease read. A counter keeps
the lock held if two operations overlap. The widget is ADMIN/ROOT only (a
BASE_USER cannot trigger a backup).
OTel Ingest (sandbox bridge http://otel.internal, shared-secret auth)
There is no public /api/otel route. The container’s OpenTelemetry exporter
targets http://otel.internal/v1/{traces,metrics}, a virtual host resolved by the
Sandbox SDK inside this Worker (TenantSandbox.outboundByHost, alongside
docs.r2.internal / releases.r2.internal). The traffic never leaves the
sandbox ↔ Worker control plane, so Cloudflare Access never interposes a login
redirect and no path has to be published.
The per-tenant OTEL_SHARED_SECRET is still verified (constant time) by the
handler: the bridge is only resolvable for this tenant’s sandbox, but a leaked
handler must not become an open sink. buildEnvVars always sets the two
endpoints and the bearer header; WORKER_URL is irrelevant to the exporter now.
POST /api/otel/v1/metrics— OTLP/HTTP JSON metrics (cost/tokens). AI cost rows keepopenclaw.provider+openclaw.modelso lerma can attribute the spend (unified-billing gateway-routed vs. direct provider / OTel-only).POST /api/otel/v1/traces— OTLP/HTTP JSON traces ({resourceSpans:[...]}). Onlyopenclaw.message.processed,openclaw.harness.run,openclaw.tool.executionspans are approved; others are counted asignored. Each approved span is classified (src/billing-classification.ts) into a semantic billing action (telegram_query,signal_analysis,email_draft, …) — an explicitopenclaw.billing.action_typemarker wins, span attributes are the fallback. Approved-but-unmapped spans are accepted and counted asnonBillable. Stableaction_id = lowercase(traceId) + lowercase(spanId)(no separator);action_keydedupes a business unit andtrace_iddrives precedence (email_draft/review_analysissuppresstelegram_query). Tenant is required per span (customer.sluget al.; one batch = one tenant).application/x-protobufis rejected with415(usehttp/json). Seedocs/contracts/otel-trace-action-id.md. Normalized traces are held in-memory for downstream reconcile; Analytics Engine stays source of truth.- One identity authority for both paths.
/v1/metricsand/v1/tracesresolve the caller through the sameresolveCostIdentity(env)(src/sandbox-tenant.ts): the explicit trustedCUSTOMER_SLUG(or theWORKER_NAMEit mirrors) first, then the routing resolver so a legacy worker that only carries a routableWORKER_URLis still attributed to its own tenant instead of thedefaultsentinel. A billable instance whose identity cannot be resolved fails closed with503 TENANT_UNRESOLVEDon both paths, and a metric that names a foreign customer through any ofcustomer.slug/customer_id/customeris rejected with403. The cost summary read side uses the same resolver, so the dashboard reads exactly the identity the write path emitted. Emission guard:isBillableEventsEnabled(env)(src/dev-mode.ts) decides whether an instance may mint billable actions. Billable production always emits and cannot be opted out, and the staging platform always emits in every environment so the billing pipeline runs with production fidelity before release (its rows land in the staging Analytics Engine dataset). A dev environment is non-emitting by default — it accepts spans (200) and the cron drain drops buffered traces — and emits only whenBILLABLE_EVENTS=trueopts it in (e.g. to exercise billing end-to-end from a dev instance). The override is emission-only —getCostContextstill labels rows with the realplatform/environment(staging/devstay non-billable cost axes), andshouldAutoSleepis unchanged, so an opted-in dev instance keeps its idle timeout and staging keeps its 15m one.
Browser Rendering (private http://cdp.internal bridge)
The agent browser is served by Cloudflare Browser Rendering through the Worker’s
BROWSER binding. It is deliberately not a public /cdp dependency for the
container: on a tenant hostname the host Cloudflare Access application answers
/cdp with a 302 login redirect, and adding a path-scoped bypass is not
acceptable. The public /cdp shim remains for direct/host-side CDP clients and
the cloud E2E worker (workers.dev), not for the tenant.
The private path has three parts:
- Container proxy (
apps/fouria/cdp-proxy.mjs, started bystart-openclaw.shinlaunch_gateway_process). It listens on127.0.0.1:9223, serves the small CDP HTTP surface (/json/version,/json/list,/json/new,/json/close/:id) and a WebSocket command channel, and forwards each command tohttp://cdp.internal/command. It is detached withsetsidand closes fd 9 so it never holds the gateway startup lock. - Worker bridge (
src/sandbox-cdp-bridge.ts, registered inTenantSandbox.outboundByHost).POST /commandruns the CDP command through the tenant’s Sandbox Durable Object (cdpCommand),DELETE /sessioncloses the browser, andGET /healthis a liveness probe. The Sandbox SDK resolvescdp.internalinside the Worker, so the hop never reaches the internet and Cloudflare Access never sees it. - Session ownership (
src/cdp-session.ts+TenantSandbox). The browser is launched lazily fromthis.env.BROWSERand kept in the DO across the proxy’s per-action WebSockets, serialized by a command queue. The Browser Rendering credential never enters the container. The CDP handlers are the samehandleCDPMethodused by/cdp, so the two transports cannot drift.
Moltlazy writes the matching remote-CDP profile (fill-only, user-editable):
browser.enabled=true, browser.defaultProfile="cloudflare",
browser.profiles.cloudflare with cdpUrl="http://127.0.0.1:9223" and
attachOnly=true (a loopback cdpUrl without attachOnly is treated as a
local managed Chrome), and
tools.alsoAllow: ["browser"] (the coding profile excludes the browser tool).
A restored boot re-applies --only gateway,browser so the profile self-heals
from an older snapshot.
Known limit: OpenClaw’s Playwright-backed snapshot/evaluate paths expect
CDP session multiplexing (Target.setDiscoverTargets/setAutoAttach,
sessionId routing) that the raw shim does not implement yet; raw open,
tabs, navigation and screenshots work. The E2E covers registration, open,
and cross-call tabs.
Cost-tracking dimensions & development expense
Every Analytics Engine cost-tracking row (ai-query, container-active,
r2-operation, cron-trigger, ws-session, action-completed, action-step,
health-check) carries two trailing blobs — platform (production |
staging) and environment (production | dev) — resolved by
getCostContext(env) (src/metrics.ts → resolveInstanceDimensions in
src/dev-mode.ts). The canonical customer key is the base client slug
(CUSTOMER_SLUG, else tenant-{slug}[-dev] with the prefix/suffix stripped),
so worker-side rows join against the client.slug lerma reconciles. health-check
rows (emitted from non-passive GET /api/health) feed lerma’s uptime/downtime
statistics. Dev/staging rows are non-billable development expense — infra +
AI cost of building for a client, tracked in AE and pulled asynchronously by
lerma, never charged to the client’s credit pool.
Software Provisioning (declared, re-applied every reprovision)
Software that agent skills install at runtime (a plugin, an apt/npm
package, a downloaded binary) lives only in the container filesystem, so it
is lost when the image is upgraded or the container is destroyed on idle-sleep.
The softwareProvisioning feature flag makes that software declarative:
| Step | Where |
|---|---|
| Declaration | customers/<slug>/software.json in BACKUP_BUCKET (PUT /api/admin/software/manifest), { id, check, install, verify?, timeoutSec? } per package |
| Materialization | materializeSoftwareManifest() → /home/openclaw/.openclaw/software.json (inside the backup root, so a warm restore also carries it) |
| Reconcile | moltlazy software provision (packages/moltlazy/src/software/): check → skip, else install → optional verify; per-package timeout, non-fatal, exit 0 |
| Boot trigger | ensureGatewayImpl dispatches the CLI detached (void spawnSoftwareProvision(...)) just before start-openclaw.sh start, so it runs in parallel and never blocks readiness |
| On-demand trigger | POST /api/admin/software/provision (ADMIN/ROOT) runs the same CLI to completion, bounded by SOFTWARE_PROVISION_RUN_TIMEOUT_MS (5 min) |
It applies on every provisioning cycle — cold start, restored boot, idle
wake, image upgrade, and /debug/restart-openclaw — because it lives in the
Worker’s ensureGatewayImpl rather than in the (skippable) setup body of
start-openclaw.sh. A soft reload (SIGUSR2) does not re-run it; nothing is
lost there.
Outputs: /tmp/software-report.json (structured; read by
GET /api/admin/software/report) and /tmp/software-progress.log (ticks; read
by /api/startup-progress as softwareLog). The CLI truncates the progress log
at the start of every run because /tmp survives an in-place restart, so
otherwise ticks from earlier boots would be interleaved.
Operational notes:
- Both
GETendpoints returnenabledso the dashboard can distinguish “flag off”, “instance asleep”, “not run yet” and “nothing declared”; the homepage widget’s Run provisioning button (ADMIN/ROOT) calls the on-demand endpoint and renders the fresh report. - A failed package stays
failedin the report and does not abort the rest of the manifest or the boot. A report is only rewritten when the CLI actually runs, so a stale report can exist inside one container after the flag is switched off — theenabled: falsemarker is what the UI keys on. - The manifest is arbitrary shell executed in the tenant container. The upload and on-demand routes are ADMIN/ROOT-only; there is no command allowlist by design (the container is single-tenant).
Container Startup Flow (start-openclaw.sh)
start-openclaw.sh takes a phase argument ($1):
| Mode | Behaviour |
|---|---|
start | Production cold-start (single entry point): acquire the container-wide startup lock first, then run the full setup body, then become the single OpenClaw owner — execing the gateway while still holding the lock. |
setup | Standalone/smoke only: run everything up to (not including) the gateway launch, then exit. |
launch | Standalone/smoke only: acquire the container-wide launch lock and become the single OpenClaw owner (or wait for the current owner to become healthy), then exec the gateway. |
post-start | After the gateway is ready — dev automation + boot-log reset, then exit. Dispatched detached (fire-and-forget) by the Worker so it never extends the shared start promise. |
full (default) | Manual/back-compat + boot smoke: run setup, then launch + poll the gateway in the background, then exit. |
The Worker’s ensureGateway drives production cold-start as a single
/usr/local/bin/start-openclaw.sh start call followed by post-start. Unlike
the standalone phases, start acquires the container-wide startup lock before
running setup and holds it through the Gateway exec, so setup and launch are
serialized under the same owner — the Worker no longer invokes setup and
launch separately. The gateway is not a shell background job: the start
owner execs openclaw gateway run while holding the lock. The lock is held for
the Gateway process lifetime via the inherited fd (released exactly when the
gateway exits), so the supervised process is the gateway — tracked by
listProcesses/getProcess for the life of the container and killable via its
handle. A concurrent invocation that loses the lock polls /health and exits 0
without starting a duplicate owner. The serial
test/e2e/ac_gateway_startup_concurrency.txt suite checks that exactly one owner
exists and that neither the migration-input-change error nor the SQLite
offline-maintenance error appears. The standalone setup/launch modes (used
only by smoke/manual runs) keep the older split behaviour — a setup flock
that serializes the slow setup body and is released before a separate launch —
which is why ownership there is enforced solely by the dedicated launch lock.
full keeps the legacy background-launch behaviour for manual runs and
test:gateway:boot.
Restored containers skip the setup body (zero-data-loss). The R2 restore
happens in the Worker (restoreIfNeeded → sandbox.restoreBackup()), before
start-openclaw.sh runs — the script never performs the restore. When the
launch runs against a container that already holds tenant state, ensureGateway
passes OPENCLAW_RESTORED=true in the launch env (see
hasRestoredTenantState), and the script skips the entire setup body — onboard →
plugin installs → moltlazy patch → config validate — emitting those phases as
skipped and launching the gateway directly from the restored filesystem. The
restored state is therefore authoritative: the cold-start budget is spent
launching it, not re-mutating it (which could diverge from or clobber the
snapshot, and which wastes startup time). A brand-new tenant (no state, no
backup) still runs the full setup body and onboards normally; the image ships no
openclaw.json, so “state present” is exactly “already initialized/restored”.
Each boot rewrites a durable receipt at /tmp/openclaw-restored-setup-skipped
(written on the skip path, removed on the setup path) so an operator can confirm
which path ran; the local backup E2E
(test/e2e-backup/zzz_backup_restore_skip_setup.txt, opt-in via
E2E_BACKUP_RESTORE=1) asserts the receipt and that the restored setup state is
left byte-for-byte intact.
OPENCLAW_RESTORED is decided by hasRestoredTenantState(), which combines
the in-memory restore status (getRestoreStatus() — hadBackup && restored, set
by restoreIfNeeded) with the container-state probe (containerHasState — the
restored openclaw.json, legacy /root or v1 /home layout). The in-memory
result alone is not sufficient, because the launching isolate is frequently
not the one that restored:
- the durable boot job is two steps (
restore, thenlaunchin a later alarm), so a DO eviction in between loses_restoreState; - any request that reaches the proxy while the restored container has not been
launched yet (the loading page polls, the browser keeps its tab open, an API
call arrives) calls
ensureGatewayfrom its own isolate, whererestoreIfNeededonly runs the cheap request-scoped check and never records a restore status; clearPersistenceCache()(freeze / resume / stale-container recovery) resets it too.
Without the durable fallback those launches passed no OPENCLAW_RESTORED, so
start-openclaw.sh re-ran the whole setup body — re-onboarding and re-installing
plugins over the just-restored snapshot (observed on restored staging tenants
after a rollout/reprovision). The restore decision is deliberately not keyed
on the durable R2 restore-needed latch, which is a separate cross-isolate
retry/fence (written before sandbox.restoreBackup(), cleared only after restore
- legacy-state materialization succeed): its absence is normal while idle and
after a successful restore, and the restore itself is triggered by the
container-state probe plus the committed recovery point (
stable-backup.json). The loading page’s “Restoring…” spinner observes the in-memory restore status the same way, so it appears the moment a cold restore is in flight regardless of whether the marker was present in R2.
One targeted exception: the restored boot re-applies only the gateway
section. The Worker-critical gateway config (trustedProxies, auth,
controlUi.allowedOrigins) must self-heal even on a restored boot: a
warm-restored openclaw.json can carry a stale gateway.trustedProxies (e.g.
the pre-2026-09 default ['127.0.0.1','::1']) that omits the Cloudflare Sandbox
bridge range (10.1.0.x) the Worker proxy connects from, so OpenClaw rejects
every proxied request with proxy_attribution_required (symptom: the tenant
serves the gateway’s JSON error {"error":{"type":"proxy_attribution_required"}}
on the root HTML/WS proxy). The skip path therefore runs
moltlazy patch --only gateway --file "$CONFIG_FILE" (bounded, non-fatal): it
heals trustedProxies (drop catch-alls, union the required ranges — see
healTrustedProxies) and re-asserts auth/origins, but never regenerates the
user-editable agents/session/tools/logging sections nor re-seeds
workspaces, so the snapshot’s user data is untouched (the backup E2E fingerprints
moltlazy-session.json + a sentinel, both unaffected).
The skip is guarded on the restored config actually existing. A “restored”
boot can carry partial container state but no materialized openclaw.json — an
interrupted restore (a container-image rollout changes the runtime mid-restore),
or restoreIfNeeded’s “container already holds tenant state; skipping duplicate
restore” short-circuit, which skips the legacy-state materialization yet still
reports restored=true and therefore sets OPENCLAW_RESTORED=true. Skipping the
setup body then launches OpenClaw with no config at all (observed on
tenant-tarjetero-dev after the v2026.10.13 rollout). start-openclaw.sh
therefore only honours OPENCLAW_RESTORED=true when $CONFIG_FILE exists;
otherwise it logs a warning and falls through to the setup body, which
onboards/repairs the config instead of launching a configless gateway.
The cold boot is a durable ActivityScheduler DO job, not a request
waitUntil(). Cloudflare cancels a request’s background (waitUntil) work a
short time after the response, so a cold boot driven from a request was killed
mid-restore and re-attempted forever: [Gateway] Gateway start already in progress … (phase=restore, age=686365ms) with start-openclaw.sh never run,
141 waitUntil() tasks did not complete warnings, 14× re-decrypt of the same
archive, and 64 probe processes hammering the Sandbox DO (observed on
tenant-client-dev). The boot is now a durable job (src/activity/boot-job.ts)
on the ActivityScheduler DO: POST /api/warmup, non-passive GET /api/health
and /api/startup/ws only request it (requestBoot, deduped) and return;
the alarm performs restoreIfNeeded → ensureGateway and retries on a bounded
lease independent of any request. When the ACTIVITY binding is absent (local
dev / unit tests) the routes fall back to the request-scoped waitUntil boot.
A request never owns a destructive restore. Even with the durable boot job,
every ensureGateway call on a cold path re-asserts the restore barrier. A
request that would actually need an archive restore now defers it:
restoreIfNeeded(..., { restoreOwner: 'request' }) (the default for
ensureGateway) performs only the cheap non-destructive checks — the
restore-needed latch and containerHasState() — returns immediately when the
container already holds tenant state, and otherwise throws
RestoreDeferredToBootJobError. ensureGateway catches it, calls
requestDurableBoot(), and throws GatewayBootPendingError, so the proxy/route
answers 503 (or the loading page) while the DO alarm performs the restore +
launch. Durable callers (the ActivityScheduler boot/wake jobs and the queue
consumer) pass restoreOwner: 'durable' and run it inline. The proxy no longer
calls restoreIfNeeded itself. Motivation: on tenant-tarjetero-dev a 350 MB
encrypted archive was re-decrypted and sandbox.restoreBackup aborted at
~30.1 s (backup.restore error … The operation was aborted) on every cold
request, because the restore ran under a request execution context that
Cloudflare cancels shortly after the response — a reboot/restore loop while the
tenant was actively used.
Three supporting bounds keep a stalled boot from hiding behind the long restore
phase: the SDK sandbox.restoreBackup RPC is bounded by
RESTORE_BACKUP_TIMEOUT_MS (fail fast on a hung control channel); the restore
reports sub-phase progress (onProgress) so isGatewayStartStuck judges the
restore phase on GATEWAY_RESTORE_NO_PROGRESS_MS of no progress rather
than the full 11-min budget; and the persistence withLease releases the shared
R2 lease when its enclosing request aborts (PersistenceDiagnosticContext.signal),
so a cancelled operation cannot leave restore-needed + a held backup-lock
re-triggering the loop.
Snapshot lease renewal and admin snapshot mode. The shared R2
backup-lock TTL is STEP_DEADLINE_MS (12 minutes), with a heartbeat every
quarter-TTL (3 minutes). Every acquire/CAS write is bounded by
LOCK_WRITE_TIMEOUT_MS (10 s) and a timeout is reported as
persistence.lock.write-timeout + a lost acquire: a conditional put that never
settles would otherwise hold the whole durable step (6 min of silence, then
job-step.interrupted) instead of failing the acquire. This was observed in
local dev, where a takeover of an expired lease wrote the object server-side
while the promise never resolved — the operator saw only
Restore — Waiting for the backup lock for the entire step. Snapshot work
explicitly renews before/after legacy
staging, sandbox.createBackup, and encryption, and before final config capture;
restore renews around decrypt and archive application. Renewals are serialized on
one promise chain, awaited before the release tombstone, and best-effort (a failed
renewal is logged but does not abort the snapshot). Structured events report
renewed, late (at least half a TTL), and failed renewals. A 12-minute TTL trades a
longer stale-holder recovery window for avoiding false lease expiry during a
legitimate long step; lock waiters can still hit their bounded acquisition timeout.
POST /api/admin/storage/sync uses the durable backup job by default when the
ACTIVITY binding is configured (202 with jobId, 409 if another backup/restore
owns the slot). ?inline=1 explicitly requests synchronous completion; ?async=1
keeps the accepted-then-background behavior. Environments without ACTIVITY retain
the inline fallback. E2E scenarios that need an immediate backup id must use
?inline=1.
Never call output()/runShell on the live gateway handle. The gateway never
exits, so the SDK’s buffered output() only opens an RPC log stream, waits for
the local timeout, and disconnects it, surfacing workerd’s ReadableStream received over RPC disconnected prematurely. Read diagnostics from the
/tmp/gateway-stdout.log / /tmp/startup.log tails (readLogTail) instead.
The setup/post-start phases must stay child-free. The Worker waits on
them with waitForExit, which — despite its name — only settles once the
supervised terminal exits, which never happens while the script leaves a live
background child. setup is awaited on the critical path; post-start is
dispatched detached but still waited with a bounded waitForExit (a lingering
child would keep the post-start process alive until the
GATEWAY_POST_START_TIMEOUT_MS kill). So neither phase may start a tail -f log
mirror: the mirror is started by the native launch (buildGatewayRunScript). A
lingering child in setup stalls cold start for the full STARTUP_TIMEOUT_MS
and makes the Worker re-run setup on retry (observed: ~11 min to first gateway
launch).
post-start never blocks readiness. Once the port wait succeeds and
markGatewayReady runs, the gateway is serving; ensureGatewayImpl dispatches
start-openclaw.sh post-start with void runGatewayPostStart(...) instead of
awaiting it. Dev instances run openclaw automations list/create there (30–60s),
and awaiting it previously held the shared start promise so every request logged
Gateway start already in progress … waiting... while /health already returned 200. startShell dispatches the sandbox.exec synchronously before its first
await, so the work still runs server-side even if the isolate is evicted;
runGatewayPostStart swallows its own errors (best-effort).
The entrypoint runs sequentially:
1. Fast-path check: Is gateway already running? → exit 0 (skipped for `post-start`)
2. Serialize concurrent startup/setup/full invocations (exclusive `flock` on
/tmp/openclaw-setup.lock, ~900s wait). The Worker's in-process
`gatewayStartPromises` guard is per-isolate, so bursty cold-start polling can
land several runs in different isolates at once; without the lock they
all run the slow setup body concurrently, contend on OpenClaw's native
config/state locks, and stall cold start for many minutes (observed: ~18 min
to gateway-ready in run 36692775633). Only one body runs; a waiter re-checks
the gateway fast-path after acquiring and exits 0 once the winner is up.
In production, `start` takes this lock **before** the setup body and holds it
through the Gateway `exec`; `post-start` is outside the lock (it runs after
the gateway is reachable). The standalone `launch` mode has its own dedicated
launch lock and must not double-lock. `full` releases fd 9 before
backgrounding the gateway so the child does not inherit the setup lock.
3. Worker-side lifecycle coordination
- Restore, snapshot, restart, and destroy are coordinated by the Worker.
- OpenClaw's native lock protects gateway ownership; the startup `flock`
above serializes onboarding/config/plugin filesystem mutations **and** the
launch across concurrent Worker isolates — `start` holds it from before
setup until the gateway is `exec`'d, so a single owner covers both. The
standalone `setup`/`launch` smoke modes still split the lock (setup lock
released before the separate launch lock), so within that path single-owner
launch is enforced by the dedicated launch lock alone.
4. Pre-backup optimization:
- Move logs to /tmp (excluded from squashfs snapshots)
- Clean stale backup files
5. Onboard (if needed):
- Detect Cloudflare AI Gateway auth from env vars
- Ensure the `cloudflare-ai-gateway` provider plugin is present before
onboard. It is **pre-installed into the image at build time** (see
Dockerfile), so this is normally a no-op detection; the install is a
resilience fallback. Non-interactive onboarding cannot approve plugin
capabilities, so without the plugin its plugin phase fails and burns the
60s onboard cap on every cold start. With it present, the phase is a no-op
and onboarding completes cleanly.
- Run: openclaw onboard --non-interactive --accept-risk
6. Enable plugins (single working `config patch`):
- All three plugins are **pre-installed into the image at build time**
(Dockerfile, unpinned → latest ClawHub release). The installs below run
only as a fallback when the extension directory is missing (e.g. a warm R2
restore replaced state). Detection (`plugin_extension_present`) checks that
directory — the artifact the gateway actually loads — and deliberately does
**not** read `openclaw plugins list --json`: the registry refresh hangs for
its full `CLI_TIMEOUT` when ClawHub/npm are slow, and a hung/partial list
reads as "not installed" and triggers a needless network install (~57s
ai-gateway + ~74s otel on a full cold boot on `staging-client-dev`). See
`docs/plans/cold-start-latency.md`.
- **Build/boot guard:** every ClawHub install is best-effort. A ClawHub
outage, a transient network error, or a plugin that requires a newer plugin
API than the pinned OpenClaw runtime (observed: `@openclaw/cloudflare-ai-gateway-provider`
resolving to 2026.9.9 against the 2026.9.8 runtime) logs a `WARNING` and
continues — the Docker build no longer fails, and the boot proceeds. The
enablement patch below only enables a plugin whose extension directory is
present, so a failed install cannot invalidate the whole patch.
- cloudflare-unified-billing (when installed) — the Paso4 plugin, served from
https://clawhub.ai/paso4/plugins/cloudflare-unified-billing. Entry opts
into `hooks.allowConversationAccess=true` so the
`agent_end`/`message_received` hooks run (OpenClaw 2026.8.2 blocks
non-bundled conversation hooks without it)
- cloudflare-ai-gateway (when installed)
- admin-http-rpc (always) — the Worker's RPC-first admin read path
- diagnostics-otel (enabled only when `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`
is set and not in E2E mode; otherwise explicitly disabled). Exporter
protocol is always `http/json`. When `WORKER_URL` is set (and
not in E2E mode) `buildEnvVars()` also derives
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT={WORKER_URL}/api/otel/v1/traces`.
`diagnostics.otel.traces` is opt-in via `OTEL_TRACES_ENABLED=true`
(default: disabled); the config shape is unit-tested in
`src/gateway/diagnostics-otel.ts` (`buildDiagnosticsOtelConfig`).
- The patch must NOT carry `plugins.bundledDiscovery` (removed in OpenClaw
2026.9.x — including it fails schema validation and drops the whole patch)
nor `plugins.allow` (an allowlist disables every bundled provider not
listed). Memory search is disabled via `memory.search.enabled=false`.
- **memory-core dreaming cadence (dev):** the same patch slows the dreaming
sweep to weekly (`plugins.entries.memory-core.config.dreaming.frequency =
"0 3 * * 1"`, default `0 3 * * *`) when `OPENCLAW_DEV_MODE=true`. The full
sweep is a heavy isolated agent turn (~6 min, `toolsAllow: ["*"]`) that the
cron replays on every wake (`cron: running missed jobs after restart`) — a
major RSS spike source on the 1-vCPU dev sandbox (observed 1.6 → 2.08 GiB
during a replayed sweep). Production keeps the OpenClaw default cadence.
- **Version drift:** the plugin-API mismatch above means a build on the
pinned runtime ships **without** the Cloudflare provider plugins until
OpenClaw is bumped (see the openclaw-migration skill) or the specs are
pinned. The guard keeps the image and boot healthy in the meantime; it does
not restore the missing provider by itself.
7. Patch config via moltlazy:
- Run: node /app/moltlazy/dist/cli.js patch
- Generates per-section includes (moltlazy-session/tools/logging.json) injected
via per-section $include; gateway/agents written inline (no root $include)
- References the gateway token as an env SecretRef (kept in the container environment)
- Run: node /app/moltlazy/dist/cli.js validate
- **Validation is FATAL**: an invalid config would prevent the gateway from
booting, so the script aborts (exit 1) instead of proceeding.
8. Harden file permissions:
- chmod 600 moltlazy-*.json, chmod 700 .openclaw
9. Start gateway:
- `start` mode (production): after setup, `exec` the gateway under the same
startup lock the phase acquired at step 2 — the Worker no longer calls
`launch` separately (its stdout log mirror / `post-start` run as usual)
- `setup` mode: smoke/manual-only standalone path — `mark_step gateway`,
then exit 0 without launching the Gateway. The production Worker uses the
atomic `start-openclaw.sh start`, which holds the startup lock and `exec`s
the Gateway, followed by `post-start`
- `full` mode: background `openclaw gateway run --port 18789 --verbose
--allow-unconfigured --bind lan`, poll `/health`, then exit while the
gateway child remains detachedStartup Progress (/api/startup-progress)
The loading page shows a transparent boot stepper + live log while the gateway
starts. start-openclaw.sh declares its boot phases in a single STEPS array
(single source of truth) and emits progress to two /tmp files on each phase
(kept in /tmp so they stay out of squashfs backups and R2):
| File | Content |
|---|---|
/tmp/startup-progress.json | { steps:[{id,label}], currentStep, currentStepId, totalSteps, skipped, ts } |
/tmp/startup.log | Clean, secret-free, chronological boot detail (verbosity-gated) |
mark_step <id> [skipped] resolves the label/index from STEPS, writes the
progress JSON atomically (mktemp + mv), and echoes the transition to the
container stdout. log_boot <level> <msg> writes clean boot detail to
/tmp/startup.log. Step transitions are intentionally not written to the
log — the loading page’s stepper shows step progress, so the log stays clean
(no duplicated step/checked info). Adding a phase = one STEPS row + one
mark_step call — the loading page renders the list dynamically, so no other
code changes.
GET /api/startup-progress (public, no warm-up) reads both files and returns
{ available, steps, currentStep, currentStepId, totalSteps, skipped, ts, log }.
It is bounded by STARTUP_PROGRESS_TIMEOUT_MS and never calls
restoreIfNeeded/ensureGateway, so it adds no load to the health path.
The loading page consumes this data over the GET /api/startup/ws WebSocket
(not by polling). Each push is a combined { ...HealthSnapshot, progress, boot, jobLog, restoreProgress?, recovery?, repair? } payload — the health snapshot
(which drives readiness/redirect) plus the boot steps + log, the durable boot
job, and the formatted job-status log. On socket open the endpoint registers
one ensureGateway warm-up with waitUntil; every subsequent health
snapshot is passive/read-only. It does not re-enter restoreIfNeeded on
every push. This distinction matters when restore is blocked on an R2
backup-lock: the former per-push warm-up queued a new 120-second restore-lock
waiter every 5–30 seconds, which could starve gateway startup and the
pre-replacement checkpoint while the UI kept its socket open.
The loading page shows the boot log panel while the gateway is starting up —
both while start-openclaw.sh is running its setup body (processType === 'script') and while the gateway binary is binding its port (processType === 'gateway'). The processType === 'gateway' state no longer hides the log:
start-openclaw.sh start execs the gateway and a start_boot_log_stream
mirror tails the gateway’s own stdout (/tmp/gateway-stdout.log) into the
curated boot log, so the log keeps filling with real launch progress (SQLite
migration, plugin load, port bind) instead of going blank for the minutes the
gateway takes to bind — the exact “Launching the gateway — no progress” window
that previously looked stuck. The mirror is killed (stop_boot_log_stream) and
the buffer cleared once the gateway is ready.
The terminal never depends on /tmp/startup.log alone. That file does not
exist during the durable boot job’s restore phase (no start-openclaw.sh
process yet) and can stay quiet while the gateway binds its port, so the panel
used to sit blank next to a slow-step notice like Starting OpenClaw Gateway (188s). The Worker now formats the durable state it already reads into a
jobLog string (activity/job-status-log.ts, pure and unit-tested):
| Line | Source |
|---|---|
[HH:MM:SS] Boot queued / running / retrying / failed — … | ActivityScheduler.getBootJob() (step, attempt, elapsed, next attempt, reason) |
[HH:MM:SS] Restore — <phase label> (<percent>%) | getRestoreProgress() (idle annotation and terminal failure included) |
[HH:MM:SS] Repair running — openclaw doctor --fix, attempt N/3 | getActiveMaintenanceJob() |
/api/startup/ws pushes jobLog on every frame, and the non-passive
not-yet-ready /api/health / /api/health-snapshot carries the same fields for
the page’s HTTP polling fallback. loading.html renders the container’s own log
when it has content and otherwise the formatted job status, labelling the
terminal source (fouria@gateway: /tmp/startup.log vs boot job status). The
panel is therefore visible for the whole restore phase instead of being replaced
by the brand logo, and the brand logo only appears when there is nothing at all
to show (no process and no durable job state).
While a cold restore is in flight, the page also shows a restoring spinner
beside the status line (title “Restoring FourIA” / status “Restoring data from
backup…”). It is driven by the health snapshot’s in-memory restoreStatus
(hadBackup && !restored) — the same signal the startup.ok probe uses to
report “Backup found but not restored” — or the durable recovery.restoring
marker (below), and is cleared the moment the restore commits (or the start
fails/redirects). It is intentionally not keyed on the R2 restore-needed
latch, so the spinner shows for a restore that would have run anyway (a fresh
container with a committed recovery point).
The page also names the backup + date being loaded, so a cold restore is
legible instead of a generic spinner. /api/health (non-passive, not-yet-ready)
and /api/startup/ws attach a durable recovery object read from R2
(readRecoveryState): { restoring: { id, created } | null, committed: { id, created } | null }. restoring is the in-flight restore-target.json written
by restoreIfNeeded/restoreBackupById just before sandbox.restoreBackup and
cleared on success; committed is the stable-backup.json recovery point a cold
boot restores. It is read from R2 (not the per-isolate restoreStatus) because
the durable boot/restore runs in the ActivityScheduler DO, whose in-memory
state is invisible to the request isolate serving the page. Lerma’s frequent
passive poll never pays the two R2 reads; they are bounded by
RECOVERY_STATE_TIMEOUT_MS and fail soft to null.
During a restore the loading UI is a progress bar synchronized with the DO’s
own phase events (not a client-side timer). restoreIfNeeded /
restoreBackupById call their onProgress(phase) hook on every
persistence.phase; the DO wires that to recordRestoreProgress, which maps the
phase to a label + a monotonic fraction (describeRestoreProgress, in
activity/restore-progress.ts) and exposes it via getRestoreProgress().
/api/startup/ws pushes it as restoreProgress: { phase, label, fraction };
loading.html renders the bar and hides the stepper while a restore is in
flight, then switches back to the container’s startup-progress.json stepper
once the gateway launch begins. A request-scoped restore with no DO feed shows an
indeterminate bar. /api/admin/boot-state also returns restoreProgress for
operators.
The label stays honest when a restore is interrupted. A failed attempt is marked
before the durable job rethrows (recordRestoreFailure), so the bar reads
Restore attempt failed during "<phase>" — retrying instead of the last phase.
A kill that skips that catch (e.g. a Worker CPU-limit kill) leaves the last
durable phase in place, so getRestoreProgress() appends
— no progress for N min once the record is idle for 2 minutes. The fraction
never moves, so a stale bar is labelled rather than presented as live.
A terminal failure must not be presented as a retry. — retrying is only
true while the durable job still has attempts left. When the boot job (or a
manual restore job) exhausts them, the DO records
describeRestoreTerminalFailure (recordRestoreTerminalFailure) with
terminal: true: the label becomes Restore failed — <reason> (or
Gateway launch failed — … for the launch step), describeRestoreProgressIdle
stops appending an idle note (a finished failure is not stalled), and the next
boot.start clears the flag. readRecoveryState likewise stops reporting a
restore-target.json whose outcome is failed as restoring, and
loading.html gives a failed boot precedence over the leftover progress bar
and status line (Boot failed after N attempts: <reason>. Refresh the page to retry.). Without this, a tenant whose recovery point could not be applied sat on
Restoring the backup — no progress for 10 min forever: the three failed
attempts took ~70 s, and the page then rendered the stale record (with the
restoring spinner, because any restore-target.json counted as restoring)
instead of the terminal failure the job had already recorded.
Archive decryption (decryptStream / encryptStream in
crypto/backup-encryption.ts) assembles each 4 MiB chunk from R2 reads with a
byte queue that copies each byte once. The earlier concat-per-read buffer
recopied the pending chunk on every ~64 KB read, which exhausted the Worker CPU
budget on large recovery archives (Worker exceeded CPU time limit during
Decrypting backup …). Measured in isolation on a 96 MiB archive: decrypt
1507 ms → 140 ms at 64 KB reads, and 5614 ms → 137 ms at 16 KB reads.
The R2 binding delivers the archive in 4 KiB reads (local workerd: 64 MiB
arrives as 16,385 reads), so the old per-read concat was the dominant cost.
Each restore logs [crypto] Decrypt read profile with the read count, bytes,
largest and average read, and chunk count, so the read pattern can be checked
on a live tenant.
The loading page uses a single push channel — GET /api/startup/ws — and
does not poll while the WebSocket is open. Flow:
- Handshake — the first WS push reports whether the gateway is already running; if so the page redirects immediately, otherwise the stream keeps warming and pushing step updates.
- Boot phase — while boot steps are in progress, the sandbox pushes the
health snapshot +
progress(steps + log) everySTARTUP_WS_POLL_INTERVAL_MS(~5s). No client polling. If a transient DO↔container transport drop makes the probe fail, the stream re-sends the last known state and backs off exponentially up toSTARTUP_WS_MAX_BACKOFF_MS(30s) until the control session reconnects, so the page keeps waiting and the same error is not surfaced/re-logged every 5s. - Fallback — if the WebSocket drops (e.g. local dev server, a network
blip), the page falls back to slow HTTP polling (
/api/healthevery 5s) indefinitely. The page has no wall-clock budget or retry cap of its own: the cold boot is driven out-of-band by theActivitySchedulerDO, which owns retries and terminal failure. A run of failed polls does not hard-fail the page (“Unable to reach server”): a container-image / Worker deploy briefly resets the DO↔container control session, so the page keeps retrying and shows “Reconnecting — the instance may be updating…”.
The page only declares failure on a permanent container error
(degraded: true in the /api/health snapshot — deleted image, missing app,
where a rescue is required), surfaced immediately via the container error, or on
the durable boot job’s terminal boot.status === 'failed' state. Every other
state — still booting, gateway reachable but warming up, restore in progress —
keeps waiting, because a cold start legitimately takes 3–5+ minutes (onboard +
gateway warmup; the provider plugins are pre-installed in the image, so plugin
installs only run on the fallback path). Mixed probe states (e.g. startup.ok
lagging behind readiness.ok while a backup restore is pending) are not
treated as failures, and there is no client-side timeout that interrupts a
long-but-progressing boot.
The loading page treats the canonical gateway status (status === 'running',
driven by the readiness probe) as “ready” rather than trusting startup.ok
alone, because startup.ok is gated on backup-restore state that can lag a
healthy, already-serving gateway.
A slow boot step is surfaced as an alert. Once the current boot step has run
longer than SLOW_STEP_MS (120s), the loading page shows a non-fatal notice
naming the step and its elapsed time (e.g. “Still working on “Installing
plugins” (185s)…“), driven by the script’s own progress.ts. It is not a
failure — cold starts legitimately take minutes — and it is hidden on ready/error.
It exists so an operator can tell a slow-but-progressing boot from a stuck one
without opening logs or the Workers Observability console. (See
docs/plans/cold-start-latency.md.)
The Worker’s root HTML route (app.all('*') in src/index.ts) uses the same
ready predicate (snapshot.readiness.ok, i.e. status === 'running') — it must
not additionally require startup.ok. Requiring both made the two disagree
whenever readiness.ok was true but the startup probe lagged (a restore still
settling, a stale cached startup result, or a start-openclaw.sh start wrapper
not yet replaced by the gateway binary): the loading page redirected to /
while the root route re-served the loading page — an infinite reload loop
where OpenClaw never appeared to boot (observed on tenant-client-dev). Keep the
two predicates in lockstep.
The gateway WebSocket proxy gates on the same readiness predicate. The
catch-all WS path in src/index.ts must call isGatewayReadyForProxy() before
sandbox.wsConnect(). wsConnect (and containerFetch) wait for their target
port inside the Sandbox DO for containerTimeouts.portReadyTimeoutMS
(CONTAINER_PORT_TIMEOUT_MS, 90s). During a container-image rollout the
container is healthy — Cloudflare reports it active and the container logs are
clean — but the gateway has not bound 18789 yet, so that one RPC parks for the
full 90s and every other RPC (health, components, other proxies) queues behind
it, surfacing as “Unable to reach server” on the loading page. When the gateway
is not ready the WS path returns a fast, retriable 503 + Retry-After: 5
(and fires ensureGateway({ waitForReady: true }) via waitUntil so the cold
start still progresses) instead of calling wsConnect. The HTML and non-HTML
HTTP proxy paths already gate the same way.
A "Timed out waiting for runtime port" error — the DO’s port wait expiring
because the container is up but the service has not begun listening — is
classified by isRuntimePortTimeoutError() (src/gateway/sandbox-recovery.ts)
as transient: the WS path returns a retriable 503, never a 502, and it
must not trigger killGateway/stale-container recovery, which would discard
a healthy container mid-rollout. It is deliberately excluded from
isStaleContainerError.
Likewise, Network connection lost, WebSocket connection failed, and the
Worker’s bounded Operation timed out after … from sandbox.wsConnect() are
transient Sandbox DO WebSocket-transport failures, not proof that the gateway
process died. The proxy returns 503 with Retry-After: 5 and leaves the
container and R2 state untouched. This matters when the admin HTTP RPC
health/status path still works: it uses containerFetch, a separate transport
from the browser’s WebSocket connection.
Security invariant: the boot log contains curated log_boot lines — the
script’s raw stdout (which can contain the AI Gateway API key as an openclaw onboard --cloudflare-ai-gateway-api-key … argument) must never be written to
/tmp/startup.log. In addition, once the gateway launches, a
start_boot_log_stream mirror tails the gateway’s own --verbose stdout
(/tmp/gateway-stdout.log) into the boot log so the loading page can follow
launch progress; this is the same output already surfaced to ADMIN/ROOT via
/debug/gateway-logs and /api/admin/logs, and it is stopped (stop_boot_log_stream)
and the buffer truncated at readiness. The gateway’s startup output is a distinct
stream from the script’s onboard stdout (which is where the API key appears as a
CLI argument and is still never captured).
Verbosity is controlled by STARTUP_LOG_VERBOSITY (steps | detailed (default) | verbose),
passed through from the Worker via buildEnvVars().
moltlazy patch — Config Injection
moltlazy patch is the mandatory config system. It runs every startup and generates:
| Config Section | Type | Content |
|---|---|---|
gateway.* | Mandatory | Port, mode, bind, auth |
session.* | Mandatory | Session scope, DM settings |
tools.toolSearch | Mandatory | Tool search configuration |
logging.* | Mandatory | Log levels, format, destinations |
agents.defaults | Always-on | Default model, system prompt |
agents.entries | Always-on | Premade agents keyed by id (dev-only fouria-builder, fouria-qa) |
skills.entries | Always-on | Built-in skills |
memory.* | Feature-gated | Memory backends (knowledge graph) |
channels.* | Env-gated | Channel tokens (Telegram, Discord, etc.) |
plugins.* | Feature-gated | Plugin management config |
The moltlazy-owned sections (session/tools/logging) are generated into per-section include files and injected into openclaw.json via per-section $include directives — not a root $include (root includes fail closed for Control UI saves / config.apply). They are regenerated from scratch every startup — never edit manually. gateway.* and agents.* are written inline into openclaw.json; the gateway token is an env SecretRef (resolved from the container environment, which start-openclaw.sh keeps set).
Memory-pressure limits. Two moltlazy-owned defaults keep the session store
from growing into the sandbox’s memory/disk: agents.defaults.compaction
(maxActiveTranscriptBytes: '8mb', keepRecentTokens: 12000,
midTurnPrecheck.enabled: true) and session.maintenance
(maxDiskBytes: '512mb', highWaterBytes: '384mb', maxEntries: 500,
pruneAfter: '14d', resetArchiveRetention: '14d',
coldStorage: { enabled: true, afterDays: 7 }). OpenClaw’s own defaults
(20 MB transcript, 10 GB per-agent disk budget, 5000 entries, 30-day retention)
are sized for a developer machine and are too large for a 4–12 GB container.
These limits bound growth but cannot reclaim worker-thread heaps that already
grew; the hard restart is still the only RSS reset (see
docs/plans/sandbox-memory-usage.md).
Admin RPC (admin-http-rpc) — RPC-first read path
The Worker reaches the gateway’s admin control plane over the gateway’s loopback
HTTP listener using OpenClaw’s bundled admin-http-rpc plugin
(POST /api/v1/admin/rpc). The typed client lives in packages/moltlazy/sdk
(createClient, HTTP RpcClient); apps/fouria/src/gateway/admin-rpc.ts
supplies the transport — a fetch-shaped adapter over sandbox.containerFetch
(global fetch cannot resolve the container’s localhost).
This replaced the CLI-first read path for the hot dashboard routes. CLI
invocations (startShell(sandbox, 'openclaw ...')) reap a fresh process and
re-establish a WebSocket on every call (~10–15s); the HTTP RPC path is a single
container request. Routes attempt RPC first and fall back to the CLI
(runCliFallback / the existing parsers) whenever the transport is unavailable
or returns an unexpected shape:
GET /api/admin/devices→device.pair.listPOST /api/admin/devices/:id/approve→device.pair.approveGET /api/admin/models→models.listGET /api/admin/channels/probe→channels.status { probe: true }
Enablement: start-openclaw.sh enables the plugin with a dedicated
plugins.entries["admin-http-rpc"].enabled = true config patch. It is kept
separate from the batched plugin patch because OpenClaw 2026.9.x rejects that
batch (plugins.bundledDiscovery was removed upstream), and setting
plugins.allow would disable every bundled provider that is not listed. Enablement
applies to a running gateway automatically; no restart.
Security: the plugin exposes a full operator control-plane surface on the
gateway listener. It is only reached in-process via sandbox.containerFetch;
src/index.ts shadows /api/v1/admin/rpc with a 404 before the public
catch-all gateway proxy, so it is never reachable through a tenant hostname.
Never remove that guard.
Cold-start guard (health probe). sandbox.containerFetch auto-starts the
container and blocks until the requested port is ready. The RPC-first health
probe (getRpcHealthSnapshot, src/gateway/health-probes.ts) therefore only
issues the RPC once a gateway process is actually detected; while the container
is cold, or the production start-openclaw.sh start wrapper is still running
setup, it returns null — RPC health is skipped until the local Gateway
health/port is open — so /api/health falls back to the exec-based
getHealthSnapshot (which never waits on the gateway port). Callers that already hold a process record pass it as
{ proc } to skip the extra scan — buildHealthSnapshot does this with the
findExistingGatewayProcess result it already fetched.
Cold-start boot lifetime. #318 split the boot so the Worker launched the
gateway after start-openclaw.sh setup exited. /api/health is bounded to a
5s snapshot, so the request returns long before the (multi-second) setup
finishes; workerd then cancels the unregistered ensureGateway continuation and
the container-side setup completes with no gateway ever launched (the
observed cold-start stall: 68 processes, setup exit 0, every curl /health exit
7). The fix was to keep the boot alive past the response, and the current runtime
contract collapses the production path into a single
/usr/local/bin/start-openclaw.sh start call (startup lock acquired before
setup and held through the Gateway exec) followed by post-start — the Worker
no longer invokes setup and launch separately. Any route that triggers a cold
boot must still keep it alive past the response with
c.executionCtx.waitUntil(ensureGateway(sandbox, env, { waitForReady: true }))
— waitForReady: true returns the shared start promise unwrapped (no read
budget) and dedupes onto the promise the bounded snapshot already started.
/api/health and /api/warmup do this; do not replace it with a bare
await ensureGateway(..., { waitForReady: false }).
The SDK’s WebSocket transport (createWsClient) is not used: the current
OpenClaw wire protocol (v4) requires a connect handshake and a paired
operator.admin device identity, which the shared-token loopback path does not
provide. HTTP RPC over the shared token is the least-effort RPC-first transport
and is what admin-rpc.ts implements.
RBAC & Access Control
Roles
| Role | Permissions | Access Scope |
|---|---|---|
| ROOT | admin:platform, admin:customer, api:all | All tenants, all debug endpoints |
| ADMIN | admin:customer, api:all | Own tenant only |
| BASE_USER | api:own | Own tenant, read-only |
Dashboard Pages by Role
| Page | ROOT | ADMIN | BASE_USER |
|---|---|---|---|
| Dashboard Home | Yes | Yes | Yes |
| Agents | Yes | Yes | Read-only |
| Connections | Yes | Yes | Read-only |
| Knowledge | Yes | Yes | Read-only |
| Account/Team | Yes | Yes | Read-only |
| Integrations | Yes | Yes | No |
| Secrets Vault | Yes | Yes | No |
| Backups | Yes | Yes | No |
| Releases | No | Yes | No |
| Diagnostics | Yes | No | No |
Debug (/debug/*) | Yes | No | No |
| Audit Logs | Yes | No | No |
ROOT is the maintainers’ intended role. It has full access to debugging tools, container inspection, and platform-level diagnostics at
/_admin/diagnosticand/debug/*. ROOT is restricted to diagnostics-only API paths (DIAGNOSTICS_PATHSinsrc/routes/api.ts) — management pages (Agents, Connections, Backups, etc.) are ADMIN-gated. As a diagnostic exception, ROOT can still verify backup connectivity (Test Backup Connectivityon the Diagnostic Dashboard →POST /api/admin/storage/test-connectivity) without the Backups page.
Workflow commit tree. The tenant’s workflow git history (commit tree, branches, tags, releases) is client data. It is served by
GET /api/admin/releases/commits, which is deliberately not inDIAGNOSTICS_PATHS: ROOT is blocked at the diagnostics gate and BASE_USER byrequireAdminOrRoot, leaving ADMIN-only access. The tree is produced from the sandbox git repository by the release tooling; until that substrate ships the endpoint returns an empty tree. It is no longer rendered on the Backups page — the ADMIN-gated Releases page (/_admin/releases) is the promotion surface, and no dashboard view renders the commit tree today.
In a harness/dev mode (
DEV_MODEorE2E_TEST_MODE),/debug/*additionally permitsADMINso the automated E2E suite can drive CLI/operator routes from a bareworkers.devworker that has no Cloudflare Access or Dispatch Worker in front of it. Real production (DEV_MODE=false,E2E_TEST_MODE=false) stays ROOT-only per the table above.
Agent Push (dev → production promotion)
A tenant owns two instances: its dev environment builds and validates agents
(the dev-only fouria-builder / fouria-qa personas), and its production
environment serves them. Agent push promotes the built content from dev to
production.
The promoted artifact is the full promotable manifest, owned by
@moltlazy (packages/moltlazy/src/promotion/index.ts): openclaw.json, the
moltlazy per-section $include files, and the agent workspace/. The dev-only
agents and all runtime state — state/, credentials/, agents/, sessions/,
secrets/ — are never promoted. Provider/channel credentials carried inside
openclaw.json do travel; only the runtime secret stores are excluded.
Promotion is driven entirely from the Worker API (operator calls on the dev and
production instances); no promotion command runs inside the container or through
OpenClaw. The transport is private R2, never HTTP. The dev instance stages the
sanitized bundle in its own bucket (promotion/staged/<id>.tar.gz +
promotion/pending.json); the production instance pulls it through a read-only
PROMOTION_SOURCE_BUCKET binding to that sibling bucket. Both buckets stay
separate — production only reads the prefix — and no tenant route is exposed for
the transfer.
Production applies the release as a custom backup: it keeps its own runtime
state (sessions, transcripts, the control-plane SQLite) while overwriting config
/ agents / tools and re-applying the gateway section for its own WORKER_URL,
then soft-reloads. The apply happens on boot, after the restore barrier and
before the launch (so a restored tenant still loads the deployed release), or
when an operator calls POST /api/admin/releases/apply. It runs under the shared
persistence lease (withPersistenceLease), so a scheduled snapshot can never
capture a half-applied tree; it is skipped while the tenant is parked or while a
maintenance window / restore is in flight; and it is idempotent — the container
release pointer (which is part of the snapshot) is the applied marker, so a
restore cannot silently roll it back and an interrupted apply simply reruns on
the next boot.
Routes (src/routes/api/releases.ts, src/services/promotion/r2-source.ts):
| Route | Instance | Behavior |
|---|---|---|
GET /api/admin/releases/commits | both | Workflow commit tree (client data; ADMIN-only, not a ROOT diagnostic). |
GET /api/admin/releases/status | both | The release this instance runs (container pointer, falling back to the applied pointer in R2), plus available when the sibling bucket holds a newer release. Also describes this instance: environment, canPromote (dev), canApply (production) and sourceBinding (whether PROMOTION_SOURCE_BUCKET is provisioned). |
GET /api/admin/releases/history | both | The push history this instance can see, newest first. Dev reads its own promotion/history/<id>.json entries; production merges the sibling dev bucket’s staged entries with its own applied/failed entries. |
POST /api/admin/releases/promote | dev | Export a sanitized bundle and stage it in this instance’s own bucket, targeted at its production sibling. Body { message?: string } records the deployer’s commit message; the verified Access identity is stored as the author. Returns 202 { staged: true }. Refuses to run on a production instance (409 NOT_DEV_INSTANCE). |
POST /api/admin/releases/apply | production | Pull the pending release (R2), apply it under the persistence lease, checkpoint it as a recovery point, and reload the gateway. Idempotent. |
Every push is content-hashed (SHA-256 of the bundle, computed with
crypto.DigestStream while the bundle is uploaded) and carries the deployer’s
optional commit message. The hash identifies the push across both instances,
and the history makes the sequence of pushes auditable without exposing either
bucket to writes from the other side:
- the dev instance writes a
stagedentry (promotion/history/<id>.json) into its own bucket; - the production instance reads those entries through the read-only
PROMOTION_SOURCE_BUCKETbinding and writes its ownapplied/failedentry into its own bucket; - the history endpoint merges the two views by release id, with the local entry
authoritative (production’s
appliedwins over the devstagedrecord).
There is no POST /api/admin/releases/import and no PROMOTION_PUSH_TOKEN:
nothing is pushed across the wire, so there is nothing to authenticate on the
production side. The R2 binding is the authority and is scoped to the tenant’s
own sibling bucket.
All promotion routes (plus the commit-tree endpoint) are ordinary operator routes
behind Cloudflare Access and requireAdminOrRoot(). Two practical consequences:
- On the production platform any ADMIN user whose JWT resolves to the tenant works; on staging the Dispatch gate is ROOT-only, so an operator session with the Paso4 Root group is required.
- An Access service token (
CF-Access-Client-Id/Secret) gets past the edge but resolves no role (it has no email and nocustom.role), so the Worker refuses it. Use a browser session or a forwarded root identity.
Export builds a gzip bundle: the config files + workspace are staged in the
container (dev-agent workspaces skipped), openclaw.json is sanitized (dev-only
agent entries removed, ambient-owner pointers repaired), then archived and
streamed into R2 with sandbox.readFile(…, { encoding: 'none' }). That call
returns the raw stream and its authoritative byte size from the same RPC, so
the FixedLengthStream used for the R2 upload can never disagree with the bytes
it writes (measuring the file separately with wc -c did, and workerd rejected
the upload with “Attempt to write too many bytes through a FixedLengthStream”).
The production side reads the object, writes it with sandbox.writeFile,
extracts over /home/openclaw (state untouched — it is not in the archive),
re-sanitizes, runs moltlazy patch --only gateway (restored-boot self-heal), and
soft-reloads.
src/services/promotion/index.ts keeps its own copy of the exclusion list so the
Worker bundle never depends on the moltlazy config module; a unit test pins both
lists together.
E2E: the former test/e2e-push/ suite drove the HTTP push and is retired with the
transport (two local Workers cannot share a miniflare R2 bucket). The R2 pull is
covered by src/services/promotion/r2-source.test.ts (stage → pending → apply →
applied pointer, plus the park/busy/idempotency skips) and the route tests in
src/routes/api/releases.test.ts.
Releases admin page (/_admin/releases)
ReleasesPage (src/client/pages/ReleasesPage.tsx, ADMIN-only, in the
Governance group) is the dashboard face of the routes above. It reads
GET /api/admin/releases/status and GET /api/admin/releases/history in
parallel and renders the steps that apply to the instance it is running on, so
the operator no longer needs curl:
- Dev instance (
canPromote): a Commit message field and a Stage release for production button (POST /promote). The message and the verified operator identity are stored on the hashed push, and the page notes that production applies the result on its next cold boot or on demand. - Production instance (
canApply): an Apply release now button (POST /apply), the release waiting in the sibling bucket (hash, message, author), and — whensourceBindingis false — an explicit warning that the instance cannot see the staged release and must be upgraded/re-provisioned. - Both: the Push history (newest first, with short hash, commit message,
author, status and dates), the current release pointer (status badge, source,
dates, bundle size, promoted agents and the excluded dev-only agents), what is
and is not promoted, and the post-apply verification checklist. A skipped
apply is explained from the route’s
reason(parked,busy:<phase>,already-applied,nothing-pending,bundle-missing, …).
The page never gates on the gateway: /releases/* are Worker-local routes
(absent from GATEWAY_TOUCHING_PREFIXES) and stay usable while the container is
asleep. Results — including failures — also surface in the notification bell.
UI coverage: src/client/pages/__tests__/ReleasesPage.test.tsx and
test/ui/releases.spec.ts (mocked API, no Worker).
Auth Flow
- Cloudflare Access intercepts request → user authenticates (email OTP, Google, GitHub)
- Access injects
Cf-Access-Jwt-Assertionheader - Worker verifies JWT against Cloudflare Access JWKS endpoint
resolveCustomerRole()resolves the RBAC role — Paso4 ROOT users (platform Access AUD) are alwaysROOT; client users get their lerma-assigned role (ADMIN/BASE_USER) forwarded by the Dispatch Worker viaX-FourIA-RoleuserRoleandcustomerSlugare set in Hono context- Route handlers check role before serving content
Dev-instance role preview (ROOT → ADMIN)
On a dev instance — a -dev hostname, or an explicit OPENCLAW_DEV_MODE
override (see isViewAsEnabled in src/dev-mode.ts) — a Paso4 ROOT user may
view and perform as a lower role from the dashboard’s “View as” switcher:
the X-Mock-Role cookie (POST /api/admin/dev/set-mock) downgrades a resolved
ROOT role to ADMIN (or BASE_USER) for that session. This lets support/ops
inspect and drive a tenant’s dev dashboard as the client’s ADMIN without the
production diagnostics-only restriction.
The preview is a strict downgrade and dev-only:
resolveCustomerRole()only honours the cookie when the role it already resolved isROOT(src/auth/middleware.ts). A non-ROOT user setting the cookie cannot escalate — e.g.ADMIN→ROOTis ignored.POST /api/admin/dev/set-mockreturns404outsideDEV_MODE/E2E_TEST_MODE/ dev instances, and403for non-ROOT callers on a dev instance.- The tenant slug is pinned to the instance’s own tenant;
X-Mock-Customeris ignored in preview mode, so the preview can never cross tenants. - Production instances are unaffected —
canViewAsstays false and ROOT keeps its diagnostics-only scope.
GET /api/admin/whoami reports canViewAs: true for a ROOT user on a dev
instance so the SPA renders the switcher (DevUserSelectorCompact); local
DEV_MODE keeps the existing full mock-role switcher (devMode).
Billing Freeze Gate (F4)
Durable per-customer billing_frozen flag (src/billing-freeze.ts). Frozen blocks
new billable actions only — in-flight actions complete, basic non-billable service
continues. This is distinct from tenant suspension (Dispatch/Lerma-level), which
denies everything.
- Durability: persisted per-customer as
billing-freeze/<slug>.jsoninBACKUP_BUCKET(R2 — survives cold starts; never in-memory only). No CustomerRegistry DO exists. - Lerma → Fouria: Dispatch Worker propagates
billing_frozeninsideX-FourIA-Metadata(shared Postgres is the source of truth; Fouria never calls Lerma directly). Effective freeze = stored flag OR header flag (fail closed). - Fouria → Lerma:
GET /api/admin/billing/freeze-status(ROOT-readable diagnostics path) plus the R2 object itself. - Writes:
POST /api/admin/billing/freeze|/unfreeze(ADMIN-only; ROOT is blocked by the diagnostics-only gate). - freezeGate(env, customer) is implemented in src/billable-actions.ts; it reads the durable per-customer R2 freeze record (no request headers available, fails open on read error); enforcement happens at emit time inside emitBillableActions, which filters billing-frozen customers’ events before writing Analytics Engine and enqueueing to BILLING_QUEUE, so in-flight work completes and basic service continues.
Development Tips
Debugging the Container
# Get container ID from Worker logs
docker ps | grep sandbox
# Enter container shell
docker exec -it <container_id> bash
# Inside container: check openclaw status
openclaw status
openclaw devices list --json
openclaw config get gateway.port
# Check logs
tail -f /tmp/openclaw-logs/*.logUnderstanding Boot Logs
During cold start, these log patterns are normal:
[Gateway] listProcesses returned 0 processes
[Gateway] Starting new OpenClaw gateway...
[ERROR] Uncaught ProcessReadyTimeoutError: Waiting for: port 18789 (TCP)
The gateway takes up to 3 minutes to become ready. The Worker retries automatically. If errors persist beyond 3 minutes:
- Check
docker logs <container_id>for OpenClaw errors - Verify CF AI Gateway config (
CLOUDFLARE_AI_GATEWAY_API_KEY+CLOUDFLARE_API_TOKEN+CF_AI_GATEWAY_ACCOUNT_ID+CF_AI_GATEWAY_GATEWAY_ID) - Run
bun run dev:workerto see full Worker output
Service Integrations (Cloudflare Tunnel)
Cloudflare Tunnel support for tenant containers was removed. The container does
not start cloudflared, and no CLOUDFLARE_TUNNEL_TOKEN binding is provisioned.
The obsidian-sync skill still runs its sync server on port 18790 locally.
CRON and Scheduled Tasks
The Worker runs one scheduled handler against two crons (see
wrangler.jsonc → triggers.crons). The handler branches on controller.cron
(CRON_BILLABLE_DRAIN / CRON_OPENCODE_CREDITS in src/config.ts):
| Cron | Job |
|---|---|
*/2 * * * * | Drain buffered billable OTel traces to BILLING_QUEUE (retry safety net for the sync emit). |
0 */8 * * 1-5 | Enqueue the OpenCode credits check onto TASKS (the queue consumer runs it in the sandbox). |
The credits enqueue is deliberately not on the frequent tick — otherwise the
OpenCode credits notification would fire every couple of minutes. Staging and
production keep only the weekday 8-hourly cron; the frequent drain is a
local/E2E aid because the synchronous /api/otel/v1/traces emit already covers
the normal path.
Containers sleep per a derived idle timeout (staging 15m, dev 2h — the Sandbox SDK 1.0
setInactivityTimeout() platform cap). The billable production environment does not
auto-sleep; it is stopped only by an explicit lerma freeze (POST /control →
sandbox.destroy()), so a warmed gateway never incurs a cold-start/downtime window after
an idle period. There is no SANDBOX_SLEEP_AFTER binding; resolveSleepAfter() derives
the value from the instance’s own WORKER_URL (src/dev-mode.ts), the same source the
dev/billable axes use. Waking is driven by the per-tenant ActivityScheduler Durable Object
alarm, which fires ahead of the next automation and then runs an interval-gated
R2 checkpoint.
The container’s idle-sleep decision is owned by the Worker’s Sandbox Durable
Object class, TenantSandbox (src/sandbox-do.ts), exported under the DO
binding name Sandbox (wrangler.jsonc → class_name: "Sandbox"). It extends
the SDK’s legacy Sandbox and overrides onActivityExpired(): on a non-billable
instance the hook ignores the persistent gateway (its lock wrapper, the gateway
binary, and the tail -f stdout mirror) and stops the container once only that
persistent work remains — matching Sandbox SDK 1.0, where code inside the instance
does not keep it alive. Any other running process (an in-flight CLI/backup
command) still counts as activity and defers to the SDK gate. On the billable
production environment the hook defers to the SDK’s keepAlive gate instead,
so the container never stops itself.
When a non-billable instance is due to idle-sleep, the hook first proves that an
existing archive already covers the current filesystem: either a committed
snapshot inside the idle window (the container has had no activity for the
whole window), or — with change detection on, the default — an unchanged result
from the same change scan the scheduled path uses (a quiet tenant takes no
scheduled snapshot, so its newest archive can age past the window). Either way the
hook stops inline through stopIfIdle() without re-archiving. That shortcut
exists because the previous unconditional snapshot was itself renewing the DO
activity window, so the stop-time re-check refused and the next idle expiry
snapshotted again — a loop of 422 MB archives that kept tenant-tarjetero-dev
awake. Otherwise (state changed, no archive, no BACKUP_BUCKET, or a probe
failure) it enqueues a durable deprovision job (mode sleep) on the
ActivityScheduler DO, which snapshots the tenant and only then calls
sandbox.stopIfIdle() — fail closed, so a container is never slept without a
committed archive. Without the ACTIVITY binding (local/dev) it falls back to
stopping inline. See Backup before deprovisioning below.
Lerma’s HealthMonitor probes the passive GET /api/health?passive=1 every
few minutes (authenticated at the Cloudflare edge with a service token — see the
path-scoped public-health Access applications in apps/fouria/iac/access.tf).
Passive probes never run the exec RPCs that would cold-start a stopped
container: getHealthSnapshot first does the non-waking isRuntimeActive()
read and reports status: "sleeping" (HTTP 200, healthy-idle) for an idle
container. Health probes are observability-only and never schedule a backup.
The container runs scheduled tasks via OpenClaw’s cron system.
Task Buffer (Cloudflare Queues)
The Worker is a durable buffer for OpenClaw-side jobs (src/tasks/): it never
implements the job logic itself.
- Producer —
scheduledenqueues the recurring OpenCode credits check ontoTASKS;POST /api/admin/tasks/opencode-credits-check/enqueueis the manual trigger (ADMIN/ROOT). - Consumer —
default.queuewakes/heals the sandbox viaensureGateway, runsopenclaw automations run opencode-credits-checkinside it, then acks (Cloudflare Queues own retries/backoff;max_retries: 5). POST /api/admin/tasks/opencode-credits-checkis the manual consumer trigger (runs the automation directly, bypassing the queue). It clamps its total wall-clock toTASK_TOTAL_TIMEOUT_MS(90s) — the gateway wake, exec launch, CLI wait, and log recovery are each clamped to the remaining budget — so a cold/contended boot always answers within the endpoint’s ceiling instead of outliving the caller (the e2e suite’scurl --max-timehit 120s exactly on cold runs). The shared gateway start promise keeps booting in the background.- Automation definition —
start-openclaw.sh’sensure_opencode_credits_automation(dev-only) creates the recurringopencode-credits-checkcron job; the Worker only runs it. The guard queriesopenclaw cron list --all --jsonin addition to the legacyautomations list --json, which missed a prior boot’s job and produced two duplicateopencode-credits-checkjobs on a dev instance. - Heartbeat cadence —
start-openclaw.sh’sensure_dev_heartbeat_cadence(dev-only) reconciles theheartbeat-fouria-builderjob on every post-start. OpenClaw 2026.10.5 retiredagents.entries.*.heartbeat: a heartbeat is now an ordinary automation job, and config reloads/restarts never rewrite an existing job, so the instance created by the migration kept the default 30-minute cadence even though moltlazy declaresheartbeat.every: 12h(observed ontenant-tarjetero-dev: 30-minute heartbeats for 9 days, ~317k tokens each on the main session, until the OpenCode subscription hit 0%). The reconciler readsopenclaw automations get <id> --json, comparesschedule.everyMs/sessionTarget/payload.lightContext, and only then runsautomations edit --every 12h --session isolated --light-context(split so a rejected--light-contextcannot lose the cadence fix). It never resets the schedule anchor when the job already matches.
Telegram Webhook Ingress (#364)
Managed tenants switch Telegram to webhook mode so an inbound update wakes a
sleeping container (long polling runs inside OpenClaw and cannot wake it). The
Worker is the always-on ingress (src/telegram-ingress/):
- Route —
POST /telegram-webhook/:accountis mounted before Cloudflare Access (secret-auth, not JWT). It resolves the account’s config from a Worker-readable F4E1 registry, constant-time validatesX-Telegram-Bot-Api-Secret-Token, and rate-limits — all before any container wake, so a rejected request never starts or renews the container. - Registry —
telegram-ingress/<slug>/<account>inBACKUP_BUCKET(F4E1-encrypted,BACKUP_ENCRYPTION_KEY). Holds the per-account webhook secret- bot token, readable by the Worker without the container running. Populated
by
POST /api/admin/telegram-ingress/:account(multi-bot).
- bot token, readable by the Worker without the container running. Populated
by
- Hot path — when the gateway is running and ready, the update is forwarded
straight to OpenClaw’s webhook route (port 18789) and its
x-openclaw-delivery-acceptedresponse is passed back to Telegram. - Cold path (“instance is not in hot mode”) — the Worker notifies the sender
via
sendMessage(“waking up…”), enqueues the update on theTELEGRAM_INGRESS_QUEUE(Cloudflare Queue), acks Telegram (the message is now durable), and kicks the cold start. The queue consumer wakes/heals the gateway, waits for readiness, and forwards — acking only after OpenClaw confirms receipt, so a message is never “popped” until it has arrived (at-least-once, mirroring Telegram’s own delivery).max_retries: 10, retry_delay: 60scovers a multi-minute cold boot. - Freeze policy — enforced upstream by the Dispatch Worker: a
frozen/operator-stopped instance is never woken by a Telegram update (no user
role → always non-ROOT); it gets an explicit
503and the message stays in Telegram’s retry queue. - Dispatch ingress — a dedicated host (
hooks.fouria.io/hooks.dev.fouria.io,TELEGRAM_INGRESS_HOST) excluded from the wildcard*.fouria.ioAccess app. It resolves/v1/<slug>/<account>, applies the freeze gate, and forwards (rewritten toPOST /telegram-webhook/:account) with no Access JWT. - moltlazy —
start-openclaw.shpasses the Worker-injectedTELEGRAM_WEBHOOK_MANIFESTto moltlazy, which writeschannels.telegram.accounts.<account>.webhookUrl/webhookSecret/webhookPath/legacyWebhook(fill-only, preserving the user’sbotToken/dmPolicy). - Queue observability — the per-tenant
TelegramIngressDO records every ingress event (bounded counters + the last 20):queued(a cold update was enqueued and woke the container),forwarded(hot path),delivered(queue consumer confirmed receipt),failed(a delivery attempt failed and will retry).GET /api/admin/telegram-ingress/statusreturns the snapshot, andGET /api/admin/telegram-ingress/wsstreams it (a frame per change) for the dashboard. The admin shell’sTelegramIngressProvideropens that socket only while the feature is in use, raises a notification-bell entry (“Telegram message queued — waking your instance”, or a delivery-failure alert), and feeds the Telegram Queue widget on the home page (state + queued/delivered/failed- last event), which is itself hidden until the ingress is in use.
- Coalesced status writes — the status object is not rewritten per update
(durable-agents audit finding F8).
telegram-ingress/coalesce.tsbatches events in memory and flushes them on one alarm,TELEGRAM_INGRESS_STATUS_FLUSH_MS(1 s) after the first event of a burst;TELEGRAM_INGRESS_STATUS_MAX_PENDING_EVENTS(25) forces an earlier flush so memory and staleness stay bounded.getStatus— and therefore the status route and its WebSocket — merges the unflushed batch, so an event is visible as soon as it is recorded. A failed flush puts the batch back and the alarm retry persists it; only an eviction inside the window can lose counters, and those are UI state, not billing records.
Backup and Hard-Restart Policy
Backups are snapshots of /home/openclaw, not a live filesystem journal. Agent
workspaces are therefore kept under /home/openclaw/workspace; files outside
that tree are not protected by the Sandbox snapshot.
The Sandbox SDK’s createBackup returns a handle marked with localBucket: true
when local-bucket mode is enabled (persistence.ts). That mode is an explicit
BACKUP_LOCAL_BUCKET=true opt-in (falling back to DEV_MODE only when unset, for
local development) — it must not be keyed on DEV_MODE alone, because the
cloud E2E deploy sets DEV_MODE=true merely to bypass Access and its backups are
real remote-R2 snapshots (see ENV-VARIABLES.md). The flag is persisted in
backup-handle.json and forwarded to restoreBackup(), so a local backup is
always restored through the local binding — never through the remote
presigned-URL path, which 404s for archives that exist only in the local bucket.
Legacy catalog entries without the flag default to the current local-bucket mode,
and a presigned-URL 404 during restore is treated as a gone backup and pruned from
the catalog.
Health probes are observability-only: GET /api/health must never write the
R2 backup catalog — no scheduleBackupIfNeeded, passive or active. A snapshot is
only meaningful for a live instance (a sleeping container has no workload, so
its filesystem has no changes to store), and waking one purely to snapshot would
restart billing. Snapshots therefore run only against a running runtime:
- the Dispatch Worker cron sends a
backupcontrol action; the tenant assertssandbox.isRuntimeActive()(a non-waking bridge read) and returnsskipped: 'not-running'for a sleeping container instead of waking it. A running instance is acknowledged immediately (202) and the snapshot is enqueued as a durable job on the per-tenantActivitySchedulerDO — awaitUntiltask is cancelled ~30s after the response (platform cap) and was observed dropping the snapshot mid-cold-boot, so the cron never blocks on it and the alarm runs it to completion; POST /debug/restart-openclawquiesces the gateway and checkpoints inside arestartmaintenance window (see the hard-restart paragraph below);POST /api/admin/storage/syncis the explicit admin checkpoint.?async=1acknowledges with202and snapshots inwaitUntil; the default (no flag) blocks and returns the result, which the E2E checkpoint relies on;POST /debug/backup/syncandPOST /debug/backup/async(ROOT) trigger the blocking and non-blocking checkpoint modes on demand;POST /api/admin/storage/reconcile(ADMIN/ROOT) runs the catalog heal (reconcileBackupCatalog) on demand: adopt orphanedbackups/*/meta.jsonarchives, sweep metadata-less uploads, re-encrypt plaintext archives, and enforceBACKUP_MAX_VERSIONS. The durablerepairstep runs the same work before every scheduled snapshot, andGET /api/admin/storage/backupsis a pure read that never heals the catalog.
Scheduled snapshots additionally refuse to run while a restore-needed marker
is present, rechecked under the shared lease, so a wake that lands mid-restore
cannot snapshot and promote a partial filesystem over the committed recovery
point. The marker is the durable, R2-backed restore latch — the scheduled gate
does not use a container-state probe, because the legacy exists probe can
fail in a deployed container and failing the gate closed would silently disable
every scheduled backup.
Soft reload (POST /api/admin/gateway/restart) does not snapshot. When no live
gateway is found it starts the gateway through ensureGateway (never arming
restore-needed, so live changes survive) and returns 409 during a maintenance window.
Restore trigger policy (when may a backup be applied?)
A restore is destructive: sandbox.restoreBackup (legacy) and
DirectoryBackup.restore (v1) replace the writable layer, so every write made
after the archive was taken is lost. A gateway reboot is not a restore
trigger: a gateway that exited, crashed, OOM’d, or is being soft-reloaded keeps
its container filesystem, and the boot must launch on that live state. The
policy is enforced in restoreIfNeeded / probeContainerState
(src/persistence.ts) for the legacy lane and
SandboxV1.restoreTenantStateIfNeeded() (src/sandbox-v1.ts) for v1:
| Trigger | Legacy | v1 |
|---|---|---|
Admin POST /api/admin/storage/restore/:id (ADMIN/ROOT) | restoreBackupById — stops the gateway, opens a maintenance window, force-promotes the point | SandboxV1.restoreRecord inside the DO |
| Durable restore job (enqueued only by that route) | restoreBackupById | n/a |
POST /debug/restart-openclaw (legacy) | quiesced checkpoint → signalRestoreNeeded latch | kill + relaunch on the same state, no latch |
| Container positively reports no tenant state (replaced/killed) | probeContainerState() === 'absent' → apply the committed point | tenantStatePresent() false → apply the committed record |
| Gateway down / restart | no restore (probe returns present) | no restore (state file present) |
Probe inconclusive (transport error / malformed answer / success: false) | ContainerStateUnknownError — boot fails closed | the state-probe exec throws or exits non-1 — boot fails closed |
Request-owned boot (restoreOwner: 'request') | deferred to the durable boot job; never runs a destructive restore under a request | the DO owns the boot |
Two consequences worth knowing when operating a tenant:
- A legacy container is only read as empty when every state path answers a
definite “no”. The probe checks
/root/openclaw/.openclaw/openclaw.json,/home/openclaw/.openclaw/openclaw.json, and/home/openclaw/.fouria-restored. Asuccess: falseanswer (the legacy SDK returns the flag alongsideexists) is a failed probe, not a “no”. A runtime that exposes noexists()method is probed through the shell over the same paths instead of being assumed empty; an unavailable shell or unrecognized output isunknown, which fails closed. - A v1 container is stopped by its inactivity policy (
keepAlivefor billable production; finite for dev/staging). Sandbox 1.0 loses an instance’s disk on stop, so a stopped v1 container legitimately boots by restoring the committedDirectoryBackup. The data-loss bound is therefore theSANDBOX_V1_SNAPSHOT_INTERVAL_MScadence, not the gateway process lifetime.
Adversarial coverage of the full trigger matrix (legacy + v1) lives in
src/restore-trigger-adversarial.test.ts and the “a v1 boot restores only an
empty container” section of src/sandbox-v1-adversarial.test.ts.
Pre-reprovision checkpoint (legacy). When the stuck-start watchdog has exhausted its resets (
GATEWAY_START_STUCK_MAX_ATTEMPTS), it no longer destroys the container unconditionally. It first takes a bounded (GATEWAY_STUCK_RECOVERY_CHECKPOINT_TIMEOUT_MS) snapshot — the gateway is already stopped by then, so the archive is quiesced — and proceeds only if the committed recovery point advanced to that archive. A failed checkpoint, a pending restore over the point, or arecovery-holdblocks the reprovision and logsgateway.start.stuck.reprovision-blocked, leaving the container and its state intact for an operator (POST /debug/reprovisioncheckpoints and fails closed the same way). Without this guard, two stuck boots on a gateway with a persistent startup failure silently reverted the tenant to the previous snapshot. v1 has no equivalent automatic destroy.
Snapshot promotion is further gated. A snapshot taken while restore-needed is set
over a committed recovery point is archived but does not move the pointer, and a
pre-replacement checkpoint is refused with 409 (RestorePendingError) before the
gateway is touched. A tenant with no recovery point yet may promote its first snapshot.
A restore-only restore (setRecoveryPoint: false) writes recovery-hold.json: scheduled
snapshots archive without promoting until an explicit snapshot clears the hold.
All three snapshot paths — scheduled, manual sync, and the hard-restart
checkpoint — share one tenant-scoped R2 backup-lock lease, so only one
isolate ever stages the fixed legacy stage directory or calls
sandbox.createBackup at a time. The lease has a 60s TTL, is renewed every 20s
while a snapshot is in flight, and is released by compare-and-swap to a
tombstone (never by deleting the object, which would race a concurrent
takeover). The two entry paths differ only in how they acquire it:
- Scheduled acquisition is non-blocking —
scheduleBackupIfNeededtakes the lease withtryAcquireLockand, if another isolate holds it, skips this tick (returns without waiting). - Manual (
POST /api/admin/storage/sync) and the hard-restart checkpoint wait for the lease with bounded exponential backoff (up to 120s), then fail with an error instead of silently reporting success.
The lease is held across the legacy stage + sandbox.createBackup consumption
and released via the CAS tombstone when the snapshot finishes. The dashboard’s
PersistenceLockIndicator mirrors the lock in the UI from the operation
lifecycle (PersistenceLockContext) rather than reading the R2 lease: the
BackupHistoryPanel raises it when it fires an async backup/restore and drops it
when the completion check settles, so there is no polling.
Async admin operations + non-blocking Backups page. A manual snapshot or a
restore can wait up to the 120s lease and then transfer a large archive, so
holding the admin HTTP request open is a poor UX and a needless failure mode (a
restore that lost the lease race surfaced as a bare 500 after ~132s, with the
underlying error dropped). Both admin routes therefore accept ?async=1:
POST /api/admin/storage/sync?async=1 and POST /api/admin/storage/restore/:id?async=1
return 202 { accepted: true } and run the work in executionCtx.waitUntil. The
BackupsPage reads the fast raw config (GET /api/admin/config/raw, a single
cat) instead of the slow structured /api/admin/config (which awaits
ensureGateway + up to two 30s shell waits) and paints every section
independently, so no single request blocks the page. BackupHistoryPanel fires
the async endpoints, immediately pushes a “started” notification to the bell,
then polls the durable job (GET /api/admin/storage/jobs/:id, getBackupJobStatus)
while a snapshot is in flight (falling back to the history diff when the server
returns no job id) or /api/health (restore) and pushes success/failure when the
operation completes — the outcome reaches the admin via NotificationProvider
even after navigating away. Restore failures are now console.error’d for
Observability.
Crash-consistency (issue #344). If a Worker isolate is killed by its CPU
budget between sandbox.createBackup() uploading the archive and the catalog
write, the archive would otherwise be orphaned (present in R2 but absent from
backup-handle.json, hence invisible to listBackups/restore). Two guards
prevent/repair this: the catalog entry is committed immediately after the
archive is created (after encryption, before the config-version capture and
retention prune), and the catalog is reconciled explicitly from
backups/*/meta.json by reconcileBackupCatalog (the durable repair step,
POST /api/admin/storage/reconcile, restoreIfNeeded,
scheduleBackupIfNeeded, and the reprovision guard) whenever an archive is
missing from the handle. getBackupHistory is a pure read: a dashboard poll
never adopts, prunes, sweeps, or writes. The scheduled backup control action
also runs under a dedicated CPU budget (BACKUP_CONTROL_CPU_MS in
workers/fouria-dispatch) so the orchestration can finish; a killed isolate’s
lease is reclaimed via the 60s TTL.
Retention is enforced by the reconcile, not by a catalog read. The two are
coupled: an isolate killed after createBackup() leaves an orphan that
reconciliation adopts, and if later snapshots then fail before their own prune
the catalog and R2 grow past BACKUP_MAX_VERSIONS indefinitely (observed on
tarjetero-dev: 7 archives under maxVersions: 3, the catalog written entirely
by reconcile, with the scheduled snapshots failing at the legacy stage).
reconcileCatalogState (behind reconcileBackupCatalog and
reconcileOrphanedBackups) runs cleanupOldBackups after merging, and the
durable repair step runs it before the gate step of every scheduled snapshot,
so the cap holds however the snapshot path ends. It still performs no
write/delete when the catalog is already within the cap, and no read path can
trigger it.
The catalog is a compare-and-swap, not a last-writer-wins file. Every
backup-handle.json write goes through mutatePersistenceStore(bucket, mutate)
(persistence.ts): it reads the handle and its etag, computes the caller’s
mutation, and writes with onlyIf: { etagMatches } (etagDoesNotMatch: '*' for a
create-only handle). On a conflict it re-reads and re-applies the mutation
against the fresh catalog — never a blind merge, because retention and restore
mutations legitimately remove entries — up to five times before raising
PersistenceStoreConflictError. A mutation returns null to skip the write.
Reconciliation, snapshot finalization (catalog.record-outcome), the retention
prune and the schedule-timestamp commit all run inside this CAS, so a dashboard
poll overlapping a snapshot can no longer drop the newly committed archive (or
resurrect a pruned one).
R2 enumeration follows truncated/cursor. listAllObjects(bucket, prefix)
backs the backup reconcile sweep, the orphan archive sweep, cleanupOldConfigVersions
and listConfigVersions. Previously each site read a single page (1,000 backup
objects, 500 config versions) and silently treated it as the whole result. A
truncated page without a forward cursor logs and stops rather than looping; page
sizes are otherwise unchanged.
Restore robustness. restoreIfNeeded/restoreBackupById persist the
restore-needed marker before changing the mounted filesystem or applying an
archive. This includes an ordinary cold restore where no marker existed before
startup: if the restore or legacy materialization fails after writing partial
state, the next invocation retries instead of treating a leftover config file as
proof of success. The marker is cleared after restoreBackup and legacy
materialization both succeed, or when candidate scanning finds no restorable
archive and there is no stable recovery point (the explicit clean-start case). The
pre-restore umount is best-effort: || true
guards the shell command, but the exec RPC itself can still throw (a
cold-start/platform interruption); that is caught and logged instead of aborting
the restore before the archive is applied. An empty catalog read also never
deletes a pending marker — it can be transient (a reconcile list failure, or
an archive not yet visible), and dropping it would silently lose a requested
retry.
scheduleBackupIfNeeded refuses to run while the durable restore-needed
marker is present (or its status cannot be read), rechecking under the shared
lease, so a scheduled wake that races a restore cannot snapshot a partial
filesystem and replace the stable recovery point. It deliberately does not gate
on a container-state probe: the legacy exists probe is unreliable in a
deployed container, and a false “no state” would silently disable scheduled
backups. The marker is normally absent while idle or after a successful restore;
cold restore selection uses the committed recovery point, not marker presence
alone.
When candidate scanning finds no restorable archive and there is no stable recovery point, restore remains a permitted clean start; the temporary retry marker is removed so onboarding and the next scheduled snapshot can proceed. If a stable recovery point exists but cannot be restored, startup fails closed and the marker remains.
Fail-closed restore (no silent empty boot). A committed stable-backup.json
is the tenant’s recorded recovery point, so its presence means the tenant was
initialized. If restoreIfNeeded finds a recorded recovery point but no
restorable archive (an empty catalog with no stable pointer is only a
brand-new tenant; a catalog drained of its archives, or a stable pointer whose
archive is gone, is not), it now throws TenantStateUnavailableError instead of
reporting a clean start. Letting the container boot empty would run onboarding
against an empty filesystem and the next snapshot would overwrite the last good
recovery point — silent data loss. The cold start aborts (the gateway never
launches), restore-needed is preserved, and the instance stays visibly
unavailable for an operator to recover. When the catalog is empty but the stable
pointer is present, the recorded entry is attempted directly first, so a
transient catalog-read failure still restores rather than falsely failing
closed.
Restore skip is keyed on container state, not runtime liveness. The
cross-isolate cold-boot dedupe previously skipped a restore when
isRuntimeActive() returned true (“Runtime already active; skipping duplicate
restore”). But a freshly provisioned container — started by the prelaunch port
probe, or replaced by an image rollout — is already “active” while its
filesystem is empty, so that check let a replaced container skip its
restore and come up empty (observed on tenant-client-dev: the skip logged 34×
alongside legacy-stage bridge timeouts). restoreIfNeeded now probes the
container’s actual state (containerHasState, the restored/onboarded
openclaw.json under the legacy /root root or the v1 /home root) instead:
absent state means a fresh container that must restore. It is always safe to
attempt an idempotent restore; it is never safe to infer state from a running
process. A probe failure is treated as “no state” (restore), never as “already
restored”. isRuntimeActive() remains the correct read for the non-waking
health/sleep paths, where the question really is “is the runtime up?”, not “is
the tenant state present?”.
Pre-replacement checkpoint (X-FourIA-Control: checkpoint). The /control
route gained a synchronous, verified checkpoint action. The control plane
(Lerma) calls it immediately before an image rollout / container replacement, so
the state about to be discarded is durably captured first. Unlike backup
(interval-gated, 202-accepted, best-effort) it bypasses the interval gate, runs
the full snapshot orchestration, and fails closed (non-2xx) unless the
committed archive’s meta.json is present and restorable. A sleeping/destroyed
instance has no live state and is reported as skipped: not-running rather than
woken. The Dispatch Worker forwards checkpoint with the same raised CPU budget
as backup (BACKUP_CONTROL_CPU_MS). In Lerma, both upgrade paths
(TenantProvisioner’s @upgrade_pipeline :checkpoint step and
Instances.do_upgrade) invoke Cloudflare.Containers.checkpoint/1 before the
rollout. The step is best-effort: a control-plane failure is logged and the
upgrade proceeds, so a checkpoint problem can never brick a valid upgrade.
Control-plane reachability (Cloudflare Access). The Dispatch Worker’s control
endpoints (POST /_control/:slug/{freeze,unfreeze,backup,checkpoint},
GET /_invalidate/:slug) sit behind Cloudflare Access on the platform domain
(dev.fouria.io staging, dash.fouria.io production). Without a path-scoped
Access application those headless calls are redirected to the Access login and
return HTML, so the checkpoint is skipped. apps/fouria/iac/access.tf declares
public_control_apps with a bypass policy (same posture as the Lerma
/api/* CI bypass), so the path is reachable without an Access login; the
Worker still enforces its own internal bearer token on top. The bearer is the
Worker secret INTERNAL_DISPATCH_INVALIDATE_TOKEN (required, no hardcoded
fallback); the Worker compares SHA-256 digests in constant time and answers
403 when the secret is unset or the credential does not match. Lerma must send
the same value (deploy-dashboard passes the production value, or
STAGING_INTERNAL_DISPATCH_INVALIDATE_TOKEN for staging) — without it the
internal calls 403 and the checkpoint cannot run. Roll out the Dispatch Worker
before Lerma in each environment, because a Lerma deploy holding the new value
is rejected until the Worker has it (and vice versa).
Checkpoint outcome telemetry. A successful tenant checkpoint returns the
archive backupId, quiesced (whether killGateway completed), and
maintenanceWindow (whether the cross-isolate R2 guard was opened). Dispatch
forwards only those checkpoint fields, and Lerma logs them as
backup_id/quiesced/maintenance_window; skipped is reported separately
when the tenant is not running. Require both quiesced: true and
maintenanceWindow: true to claim the guarded quiesce path ran; a successful
snapshot by itself only proves archive creation.
The checkpoint takes the R2 lease before quiescing. A live createBackup
can race the gateway’s own writes, so after acquiring the shared backup-lock
lease the action records a durable, cross-isolate maintenance window
(src/maintenance.ts, maintenance/replacement.json in BACKUP_BUCKET) and
then stops the gateway (killGateway) before snapshotting. A busy lock fails
before maintenance is opened or the gateway is stopped; it must never drain the
tenant while waiting for another snapshot. While the window is open,
ensureGateway refuses to start the gateway
(GatewayMaintenanceError) so a request served by another isolate cannot
resurrect it mid-snapshot; the window has an explicit expiry (15 min) so a
crashed controller can never wedge a tenant, and a malformed/expired record is
discarded (fail open). The checkpoint action always restores service and closes
the window in a finally, even when the snapshot fails, so a failed checkpoint
never leaves the tenant drained. Its recovery restarts the existing live
filesystem with skipRestore: true: the snapshot attempt did not replace the
container, and trying to restore while recovering from a lock timeout would
enqueue another waiter behind the same lock. The proxy maps maintenance refusal
to a retryable 503.
restoreIfNeeded also coalesces concurrent calls within a Worker isolate by
R2 bucket. The startup WebSocket allows only one in-flight
ensureGateway({waitForReady:true}) warm-up, retrying only after a failed attempt
settles and with bounded exponential backoff. Its recurring health snapshots are
passive/observation-only, and startup-progress shell reads run only when the
startup wrapper is actually visible; it does not start a fresh restore attempt
or shell RPC on every poll. Together these prevent a long-lived startup stream
from multiplying 120-second waiters behind one in-flight persistence operation
or waking an empty container ahead of the restore.
Checkpoint recovery restarts the existing container filesystem with
ensureGateway({ bypassMaintenance: true, skipRestore: true }). A checkpoint
does not replace or clear that filesystem; if its snapshot cannot acquire the
shared backup lock, recovery must not call restoreIfNeeded and enqueue another
waiter on the same lock it just timed out waiting for.
Only the snapshot is fail-closed. The maintenance window and the gateway stop
are best-effort guards: a failure to open the window or to stop the gateway
falls back to the previously-shipped live snapshot instead of failing the
replacement (a broken guard must never block an upgrade). The post-snapshot
meta.json check is retried briefly before it is treated as a real failure, so a
transient R2 visibility gap cannot fail an otherwise-successful snapshot. The
checkpoint returns quiesced: true only when the gateway was actually stopped.
This closes most of the planned-replacement loss window, but it is not yet a
full drain-to-replacement fence: the gateway is restarted after the checkpoint,
so writes between the checkpoint and the rollout are still outside the archive,
and the shared control-plane coordination record is not implemented. Durable
primary storage for unexpected container loss remains the gated next step in
docs/plans/legacy-sandbox-zero-data-loss.md.
Rollout park (X-FourIA-Control: checkpoint-park / resume)
The plain checkpoint restarts the gateway before the rollout, so anything
written in between is discarded with the replaced container. The park variant
is the zero-loss sequence: it is the same verified snapshot, but the tenant is
left drained and write-blocked until the control plane calls resume after the
container has been replaced.
- Ordering.
checkpoint-parkacquires the R2 lease, opens the maintenance window withparked: true(operation: "rollout"), arms the exec lock, and only then stops the gateway and snapshots. The archive therefore describes a filesystem no writer can touch afterwards. - Two enforcement layers (
src/tenant-park.ts):- Request:
tenantParkGuardmiddleware refuses mutating (POST/PUT/PATCH/DELETE)/api/admin/*and/debug/*requests with409 { code: "TENANT_PARKED" }while the durable park record exists. Reads stay available so the dashboard can still show state. This is the authoritative, cross-isolate check (R2 read per mutating request). - Exec:
startShellcallsassertTenantExecAllowedbefore the launch RPC, so background/spawned work that never passes a route (schedulers, promotion helpers, fire-and-forget tasks) cannot start a process either. The park’s own path runs insidewithTenantExecBypass(exception-safe, nestable), which is the only way past the lock. That lock is bounded (PARK_EXEC_LOCK_TTL_MS, 60 s): it cannot be released cross-isolate —resumelands in whichever isolate receives it — so it closes the in-isolate race during the park and then lets the durable record (middleware +ensureGatewayrefusal) be the only gate. Background exec is additionally gated byensureGateway, which refuses to run during any maintenance window.
- Request:
- The gateway is the other writer, and it is gone: automations and channels
run inside the gateway process the park stopped.
ensureGatewayalready refuses to launch during any maintenance window, so a request in another isolate cannot resurrect it. - Live agent sessions are switched to read-only first. Before the gateway is
stopped,
checkpoint-parkcallsdrainLiveSessions(src/gateway/session-drain.ts): it readssessions.listover the Gateway WebSocket (operator.read+operator.write), patches every session topermissionMode: 'read-only'(exec denied, managed mutation tools omitted), and waits for the turns that were already running to finish — so the next turn picks the mode up instead of being interrupted mid-write. The previous mode per session is recorded durably (maintenance/session-modes.json, never inside the container) and handed back byresume/the post-restore path, because the mode lives on the session row and is therefore part of the snapshot. A session that is already read-only is skipped (nothing to restore), and new sessions are untouched — they inherit the configured policy, so nothing here becomes a default. The drain is bounded (SESSION_DRAIN_TIMEOUT_MS, 120 s): a turn that outlives it is reported assessionsDrained: falseand the park proceeds, since the quiesce aborts it cleanly and every write it already made is inside the snapshot. A long agent turn can therefore never block a release.resumewaits (SESSION_MODE_RESTORE_WAIT_MS) for the replacement gateway, applies the map and clears it; if the gateway is not up yet the map stays andlaunchRestoredStateapplies it once the restored state is serving (so an operator restore of a parked archive also puts the modes back). - Telegram users are told, by the ingress, not the model. While the tenant is
parked,
POST /telegram-webhook/:accountanswers the sender directly withTELEGRAM_PARKED_NOTIFY_MESSAGE(Bot APIsendMessage, no gateway call, no AI turn), records aparkedingress event and returns200without enqueuing or waking — the update could not be delivered during the park anyway. The queue consumer does the same for an update that was already queued: it answers and acks instead of retrying for the whole rollout. resumecloses the park, disarms the lock, clears the stale in-memory restore barrier (clearPersistenceCache, asfreezedoes), and requests the boot. It deliberately does not passskipRestore: the replacement container is empty and must rehydrate from the parked checkpoint. Aresumewith nothing parked is a no-op (resumed: false) and never wakes a sleeping tenant.- Failure behaviour. An aborted park (snapshot/lock failure) un-arms the lock, closes the window, and restores service exactly like a failed plain checkpoint, so a failed rollout never leaves the tenant drained. The 15-minute window TTL remains the crash backstop: a controller that dies mid-park cannot wedge a tenant.
- Control surface.
POST /controlacceptscheckpoint-parkandresume; every other action (backup,freeze,unfreeze) is refused with409 TENANT_PARKEDwhile the tenant is parked. The Dispatch Worker allows and forwards both actions (CONTROL_ACTIONS/isControlAction), givescheckpoint-parkthe inline snapshot CPU budget, and echoes only the whitelistedparked/resumedfields.
Lerma drives the sequence from @upgrade_pipeline: :checkpoint calls
Cloudflare.Containers.checkpoint(instance, park: true) and is fail-closed
(a checkpoint that cannot be taken aborts the upgrade before the container is
replaced; the only exemption is a sleeping instance, which has nothing live to
lose), :dispatch_deploy performs the script upload + image rollout, and the
:resume step (immediately after) releases the park. Both error paths call
Containers.resume/1 best-effort so a failed upgrade cannot leave a tenant
parked.
What is still outstanding is the post-rollout readiness gate (waiting for
/api/health to report a running gateway and verifying the deployed image before
declaring success, plus a fleet-level cohort/abort policy) — see the rollout
section above.
The pre-snapshot legacy-stage shell runs the node:sqlite online backup plus a
symlink-preserving tar pipe of the legacy roots, with
--exclude='*/node_modules/openclaw'. It deliberately preserves symlinks
rather than dereferencing them (cp -aL) and excludes the image-provided
shared OpenClaw package (extensions/<id>/node_modules/openclaw → the image’s
/usr/local/lib/node_modules/openclaw). -L expanded that package into every
archive (three full copies), which pushed the stage past its bound and — worse —
made the restore materialization a multi-minute copy (observed on
staging-client-dev: cp -a … /root/openclaw/.openclaw/ running 8m45s, starving
the gateway launch, leaving restore-needed set, and re-running the restore
every boot). The restore bridge (materializeLegacyRootsAfterRestore) uses the
same tar --exclude pipe, so an old backup’s dereferenced copy is never
materialized; a copy-then-rm -rf is equally slow because deleting a
squashfs-lower tree requires copying each file up first. restoreIfNeeded clears
the restore-needed marker immediately after a successful materialization
(before the analytics bookkeeping) so an isolate eviction cannot leave a stale
marker. On a large tenant the stage can exceed the
old 300s bound (observed killing the tarjetero stage at exactly 300s — issue
#344 follow-up). The bound is LEGACY_STAGE_TIMEOUT_MS (default 600000), the
idempotent stage is retried once on a transient timeout/transport failure, and
a failure logs exit/timedOut/stderr plus a structured process event with the
Sandbox processId for cross-dataset correlation. The stage wait is
transport-resilient (runShellResilient → waitForExitResilient): a Sandbox
DO↔container drop mid-stage reattaches by process id and resumes within the stage
deadline, so a mid-stage DO reset no longer ends the request. The Dispatch
backup CPU budget (BACKUP_CONTROL_CPU_MS) is aligned to the same 10 minute
bound.
The scheduled backup control action acknowledges immediately (202 { ok: true, accepted: true, jobId }) and enqueues the snapshot as a durable job
on the per-tenant ActivityScheduler DO, so neither the dispatch cron nor an
operator request is bounded by it. The DO alarm runs the job past the request
and retries under a bounded lease — the previous waitUntil approach was
cancelled ~30s after the response and could drop a snapshot that was still
cold-booting. Combined with the crash-consistent catalog commit and orphan
reconciliation (#344) the archive stays durable even if the isolate is replaced
mid-snapshot. POST /debug/backup/async and POST /debug/backup/sync
(DEBUG_ROUTES, ROOT) expose the two modes on demand.
Config-version snapshots. Alongside the archive, each snapshot captures
/home/openclaw/.openclaw/openclaw.json into R2 under configs/<timestamp>.json
(30-day retention) for the Backups page’s Config Version History / diff. The
history is rendered with client-side pagination (10 versions per page, newest
first) so a long-lived tenant does not paint thousands of rows. The
capture normalizes sandbox.readFile() across SDK generations
(readSandboxFileText in persistence.ts): the legacy SDK returns
{ content, … }, SDK 1.0 returns a Response, and older helpers return a bare
string/Blob. Calling String() on the structured result collapsed it to
"[object Object]", which was stored verbatim and then rejected by the diff with
"[object Object]" is not valid JSON. A JSON.parse write-time guard now
refuses non-JSON config versions, and the diff viewer (DiffTool) shows a clear
“corrupted” message if a pre-existing bad snapshot is opened.
POST /api/admin/storage/test-connectivity (ADMIN/ROOT) is a read-only
diagnostic that verifies the R2 backup bucket binding is reachable by HEADing the
backup catalog handle — a 404 (no handle yet) still counts as connected. It is
whitelisted for ROOT through the diagnostics gate (DIAGNOSTICS_PATHS in
src/routes/api.ts) so platform operators can test backup connectivity from the
Diagnostic Dashboard without explicit access to the ADMIN-gated Backups page.
Hard restart (POST /debug/restart-openclaw) takes the shared lease, then opens a
restart maintenance window and kills the gateway before cutting the archive, so
the checkpoint is consistent. It sets restore-needed before closing the window, so no
request can relaunch the gateway on pre-restart state in between. If the checkpoint
fails after the stop, the gateway is restarted in place on the untouched filesystem and no restore is armed. It refuses with 409 while a restore is in flight
or a maintenance window is open, and while a restore is pending over the recovery point.
After a successful checkpoint, the next activation restores the committed snapshot.
Restore outcome. sandbox.restoreBackup is bounded by RESTORE_BACKUP_TIMEOUT_MS.
Because withTimeout only stops waiting, a timeout is recorded in
restore-target.json as outcome: 'unknown', not as a failure. While that record is
younger than GATEWAY_RESTORE_PHASE_BUDGET_MS, getRestoreInFlightReason blocks every
restore (RestoreInFlightError, 409 on the admin route). A definitive failure is
recorded as outcome: 'failed' and does not block. The in-lease checks (restore
entry, restoreIfNeeded, restoreBackupById) re-read this under the lease, and
restoreIfNeeded also re-checks the maintenance window after it waits for the lease.
Destructive guard. getPersistenceBusyReason is the single busy check for freeze
(/control), POST /debug/destroy-container, and POST /debug/reprovision. It is
busy while any maintenance window is open or a restore is in flight. Reprovision also
re-checks after its checkpoint. A snapshot-only lease is not busy: an abandoned
snapshot leaves no half-applied filesystem behind, whereas an abandoned restore does.
flowchart TD
DC["Dispatch Worker cron<br/>backup control action"] --> RC{"runtime active?<br/>isRuntimeActive()"}
RC -- no --> SKIP["skip (never wake)"]
RC -- yes --> S["scheduleBackupIfNeeded"]
S --> G{"credentials + interval + lock"}
G -- no --> N["no-op"]
G -- yes --> L{"backup-lock free?<br/>(non-blocking)"}
L -- busy --> SKIP["scheduler skips this tick"]
L -- free --> HOLD
MAN["POST /api/admin/storage/sync"] --> W["acquire lease<br/>wait ≤120s w/ backoff"]
HR["POST /debug/restart-openclaw"] --> CP["checkpoint first"]
CP --> W
W -- busy > 120s --> ERR["error (no snapshot)"]
W -- acquired --> HOLD["lease held over legacy stage<br/>+ sandbox.createBackup"]
HOLD --> A["Sandbox snapshot<br/>/home/openclaw"]
A --> E["optional F4E1 encryption"]
E --> P["commit backup-handle.json"]
P --> REL["release lease (CAS tombstone)"]
CP --> K["graceful kill + restore marker"]
K --> R["restore committed snapshot<br/>on next activation"]
RP["POST /debug/reprovision"] --> RCP["checkpoint first<br/>(or require existing backup)"]
RCP --> RD["destroy instance<br/>409 unless restorable backup or force"]
SR["POST /api/admin/gateway/restart"] --> SOFT["SIGUSR2 in-process reload<br/>no snapshot"]
Do not perform repeated hard restarts without verifying a successful
checkpoint. The checkpoint protects only files inside /home/openclaw and only
after the backup catalog has been committed.
Durable async backup/restore jobs + stable recovery point
POST /api/admin/storage/sync?async=1 and
POST /api/admin/storage/restore/:id?async=1 no longer rely on request
waitUntil() (Cloudflare only guarantees ~30s of background time after a
response). Instead the route commits a job record to the per-tenant
ActivityScheduler Durable Object (src/activity/backup-jobs.ts) and returns
202 { accepted: true, jobId }. The DO alarm performs the work and retries with
a bounded lease (BACKUP_JOB_LEASE_MS) so a multi-minute snapshot survives
isolate eviction and Worker redeploys. GET /api/admin/storage/jobs/:id reads
the persisted status. The ActivityScheduler owns one alarm, so the DO merges
the next automation wake and the next backup job and arms the earliest
(scheduleCombinedAlarm); a queued job is never dropped when the activity
schedule clears the alarm.
Every durable job runs one bounded step per alarm invocation. A Durable
Object alarm handler has a hard 15-minute wall limit, while a snapshot is
stage (≤10 min) + createBackup + config capture + prune — and the alarm also
runs the boot, maintenance and activity queues. A single-invocation job could
therefore be killed by the platform, stay running until its lease expired, and
restart the whole operation from zero (the orphaned-archive class of incident).
src/activity/job-steps.ts persists a cursor: { step, attempts, startedAt }
on the job record; each alarm runs at most one step, commits the cursor, and
re-arms one second later so the other queues get a fair turn.
| Job | Steps |
|---|---|
| snapshot (manual) | repair → prepare → archive → finalize |
| snapshot (scheduled) | repair → gate → prepare → archive → finalize |
| deprovision | repair → prepare → archive → finalize → stop |
| restore | restore → launch |
| boot | restore → launch |
| maintenance | preflight → execute |
repairbrings the catalog back to a consistent state before any snapshot work: it sweeps archive objects that no catalog entry can use (the SDK writesbackups/<id>/meta.jsononly after a successful upload, so an upload that lost its metadata write is invisible to both the reconciler and the retention prune — it would sit in the bucket forever, billed and unrestorable) and it encrypts the archives that are not encrypted at rest.- The lease covers one step, not the whole operation (
STEP_LEASE_MS= 6 min). A Durable Object alarm has a hard 15-minute wall limit, so the previous 14-minute lease meant a step killed at minute 3 stalled the job for another 11 minutes before it was retried. Alarm invocations are serialized per object, so a healthy step is never duplicated. - A step recovered after its lease expired had an unknown outcome: the job
says so (
detail: 'step-interrupted', or'step-deadline-exceeded'past the 12-minute soft deadline) and emits ajob-step.interruptedevent instead of silently replaying. Every step emits ajob-step.completed/job-step.failedevent with its duration — the measurement the step budget is derived from (max(configured, 2 x the last snapshot duration)for the cadence, 12 min for the per-step deadline). prepare/archive/finalizeare the phased snapshot (prepareSnapshot/archiveSnapshot/finalizeSnapshotinsrc/persistence.ts). The catalog entry is still committed inarchive, immediately after encryption, so a crash beforefinalizeleaves a restorable archive (andfinalizere-adopts it if the catalog write was lost).- Each step takes the shared R2 lease on its own (
withSnapshotLease); the stage is idempotent andarchivere-checks therestore-neededlatch, so a hand-off between steps cannot capture a half-restored container. A scheduled snapshot refuses to archive behind a pending restore; a manual/deprovision checkpoint deliberately still archives (without promoting), which is the documented recovery case. - The retry budget is per step (
isStepExhausted) as well as per job, so a permanently broken step fails the job instead of consuming attempts that belong to healthy steps. A completed step is never re-run: a boot that is interrupted duringlaunchresumes atlaunch, not at the destructiverestore. - The legacy stage runs with
maxAttempts: 1inside a durable step (the step cursor owns the retry), so its worst case stays at the 10-minuteLEGACY_STAGE_TIMEOUT_MSinstead of 20 minutes. - The activity alarm now enqueues a scheduled snapshot instead of running
scheduleBackupIfNeededinline, so a 10-minute stage no longer shares an invocation with the automation wake.
Archives that are not encrypted at rest are visible and repaired. While a
master key is configured, an archive whose meta.json does not carry
encrypted: true (a legacy plaintext archive, or one whose upload finished
while the encryption step never ran) is catalogued with unencrypted: true and
queued in pendingEncryption. It is never dropped (it stays restorable) and
never promoted to the recovery point; the durable repair step encrypts it
in place and clears the flag. The dashboard shows a “not encrypted” badge for
such an archive.
Scheduled cadence is a floor, not a promise. BACKUP_INTERVAL_MINUTES
(default 5) is the minimum; resolveEffectiveIntervalMs stretches the effective
interval to at least 2× the last observed end-to-end snapshot duration
(recorded as lastSnapshotDurationMs in the catalog). Without that, a tenant
whose snapshot takes 10 minutes would start a new one the moment the previous
committed: continuous R2 lease contention, continuous container work, and a
starved automation wake. A fast tenant is unaffected.
createSnapshot promotes a stable recovery point only after the SDK
archive, optional F4E1 encryption, and the catalog commit have all succeeded
(promoteStableBackup → stable-backup.json). A cold restore prefers that
pointer and never restores a newer archive whose promotion was interrupted;
retention (cleanupOldBackups) always keeps the stable entry, so a
just-uploaded-but-unpromoted archive cannot evict the last committed recovery
point. The pointer is also monotonic (an older or replayed archive never
moves it backwards) and fails closed: a malformed stable-backup.json
aborts restore and retention rather than falling forward to an unpromoted
archive. Unit-tested in src/persistence.test.ts (the “stable recovery point
promotion” block). The latest validated config is also mirrored to a fixed-key
hot copy (stable/config.json), kept out of the version-history listing and
30-day prune.
A targeted restore (POST /api/admin/storage/restore/:id) force-promotes the
restored archive (the exception to the forward-only pointer) so the next cold
boot restores the same archive. ?setCurrent=0 (sync or ?async=1) threads
{ setRecoveryPoint: false } through restoreBackupById (and the durable
restore job) to apply an archive without moving the recovery point — the
next fresh container then reverts to the unchanged pointer. A restore also opens
a cross-isolate maintenance/replacement.json window (operation restore) and
stops the gateway right after taking the shared restore lease, closing the
window in a finally; restoreIfNeeded and ensureGateway both refuse to run
under it, so a refetch cannot restart the gateway and remount the latest
recovery point mid-restore.
This makes the async workflow durable and the recovery point monotonic. It does not turn interval snapshots into zero-RPO replication: writes acknowledged after the last committed snapshot are outside the guarantee. Closing that gap requires a write-through storage path and is tracked in issue #377.
Sandbox → R2 Connectivity (outboundByHost)
Sandbox code can reach Worker bindings through a virtual hostname without
any SDK/client library or credentials inside the container. @cloudflare/sandbox
(via @cloudflare/containers) exposes the canonical per-host outbound handler:
// src/index.ts
Sandbox.outboundByHost = {
[RELEASES_R2_HOST]: (request, env, ctx) => handleReleasesR2Request(request, env),
};
// catch-all: internet passthrough
Sandbox.outbound = async (request) => fetch(request);
The handler runs in the Workers runtime with full env bindings, so the
sandbox never sees R2 credentials. Unmatched hosts fall through to the catch-all
Sandbox.outbound.
Release bridge (releases.r2.internal). src/sandbox-r2-bridge.ts maps
plain HTTP from the sandbox onto the tenant’s BACKUP_BUCKET under a fixed
releases/ prefix (the bucket is per-tenant, so the prefix is already
tenant-isolated):
| Method | Path | Effect |
|---|---|---|
GET | / | JSON list of release objects |
GET | /<key> | Stream an object (200) or 404 |
HEAD | /<key> | Size/etag metadata (200) or 404 |
PUT | /<key> | Upload an object (201) |
DELETE | /<key> | Remove an object (204) or 404 |
Keys are sanitized against traversal before touching R2; uploads use a
FixedLengthStream (bounded memory) when content-length is advertised.
Docs bridge (docs.r2.internal). src/sandbox-docs-bridge.ts exposes the
tenant’s client documentation (customers/<slug>/docs/) for on-demand fetch by
the Fouria Builder agent / sandbox tooling, reusing R2ClientDocsStore
sanitization. The tenant slug is resolved via the shared resolveTenantSlug
(src/sandbox-tenant.ts): an explicit customer-<slug> container id wins,
otherwise it is derived from the per-tenant Worker env (CUSTOMER_SLUG, else
WORKER_NAME / the always-provisioned WORKER_URL, with the tenant- /
staging- prefixes and -dev suffix stripped), since production provisions the
sandbox under the fixed id openclaw. GET / lists,
GET/HEAD /<name> reads, PUT /<name> writes, DELETE /<name> removes. This
lets the sandbox read docs without R2 credentials and without waiting for
Worker-side materialization.
Software bridge (software.r2.internal). src/sandbox-software-bridge.ts
exposes the tenant’s declared-software manifest (customers/<slug>/software.json
— the same key the admin upload and the boot-time materialization use) so the
dev-only Fouria Builder can read and declare packages without the
Access-protected admin API (from the container, /api/admin/* answers 302
to the Access login page). GET / returns a discovery summary (present,
path, containerPath, writeEnabled); GET/HEAD /manifest reads; PUT /manifest validates (the same validateSoftwareManifestInput as the route, with
a 256 KiB cap) and replaces; DELETE /manifest removes. The slug comes from the
shared resolveTenantSlug, so a sandboxed agent can only reach its own
manifest. Writes are dev-only (isDevAgentsEnabled): a manifest entry is
arbitrary shell executed inside the container at provisioning time, so
production stays read-only through the bridge and operators declare software
from the dashboard / admin API. The dev-only software-manifest workspace skill
(packages/moltlazy/workspace/skills/software-manifest.ts) documents the bridge
and is seeded for the Fouria Builder; because workspace skills outrank Skill
Workshop skills, it shadows any Workshop copy of the same name.
Leads bridge (leads.r2.internal). src/sandbox-leads-bridge.ts exposes the
global paso4-leads proposal bucket (the fouria-leads skill) over
LEADS_BUCKET. The bucket folder layer is the tenant slug, and the bridge
resolves the caller’s slug via the shared resolveTenantSlug in
src/sandbox-tenant.ts (explicit customer-<slug> container id, else the
per-tenant Worker env CUSTOMER_SLUG / WORKER_NAME / WORKER_URL). Every path
the agent sends is relative to its own <slug>/ root — the handler prepends
the slug,
so a sandboxed agent cannot list, read, or write another client’s proposal. The
only bucket-root object a tenant may read is the shared, proposal-agnostic
build-pdf.sh (SHARED_LEADS_READ_KEYS); writes/deletes of it resolve under the
tenant’s own folder. Because the bucket is global, the bridge refuses the
bare fixed sandbox id (openclaw) as a fallback namespace when no env identity
resolves: a worker that lost its identity env must not share openclaw/ with
another misconfigured worker. Routes: GET / (or GET /?prefix=<p>) lists relative
keys, GET/HEAD /<key> reads, PUT /<key> writes, DELETE /<key> removes. Keys
reuse the shared sanitizeLeadsKey / sanitizeLeadsPrefix rules from
leads/operations.ts. This is how the sandbox reaches proposal docs without a
public route — the legacy wss://<worker>/api/leads/ws proxy
(src/routes/leads.ts) is intercepted by Cloudflare Access on tenant hostnames
(a per-tenant Access app shadows the *.fouria.io/<path> apps) and is now used
only by the host-side E2E fixture (test/e2e/o_leads_proxy.txt, which
authenticates with the CDP_SECRET persisted by server/start).
Binding requirement.
LEADS_BUCKETmust be provisioned as anr2_bucketbinding (lermainstances.exinstance_bindings/2, configLEADS_R2_BUCKET, defaultpaso4-leads). A plain-text var holding the bucket name makesenv.LEADS_BUCKETastring, sobucket.list()throws inside theoutboundByHosthandler and the container sees an opaqueECONNRESET. All the R2 bridges validate the binding withisR2Bucket(src/r2-binding.ts) and wrap their handler body in a try/catch, returning a structured JSON503/500instead of resetting the socket.
R2 S3 credentials are deprecated for the container. buildEnvVars no longer
forwards R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY into the sandbox: the
bridges run in the Workers runtime with the BACKUP_BUCKET / LEADS_BUCKET
bindings, so the container never sees credentials. The S3 credential remains a
Worker binding for the Sandbox SDK presigned-URL backup path only.
SDK 1.0 (preview). The canonical
static outboundByHostAPI is unchanged in the 1.0 preview, so these bridges run as-is on@cloudflare/sandbox@0.13.0-next; see the Sandbox SDK 1.0 section below.
Client Docs (dev-only admin surface)
Client documentation is managed from the Admin Client Docs page
(/_admin/docs, ADMIN-only, dev instances only) backed by
src/routes/api/fouria-docs.ts (dev-gated by isDevAgentsEnabled, so it 404s in
production). It manages two tenant-scoped namespaces directly from the Worker
— no sandbox/CLI round-trip:
- Documentation —
customers/<slug>/docs/inBACKUP_BUCKET(markdown, text, and binary files like PDFs/images; content type from R2 metadata or the extension). Store:R2ClientDocsStore(listDocs/getDoc/getDocBinary/putDoc/putDocBinary). - Proposals (leads) —
<slug>/in the globalLEADS_BUCKET(PDFs and assets), scoped to the caller’s tenant.
The SPA renders markdown client-side (marked + DOMPurify, sanitized) and opens
PDFs with the browser’s native viewer through the /raw endpoints. Raw responses
set X-Content-Type-Options: nosniff; HTML/SVG are forced to attachment with a
sandbox CSP so client-authored files can never execute in the admin origin.
Versioning (DocsGitStore). Every doc write is committed to a Worker-side,
git-like object store in BACKUP_BUCKET:
customers/<slug>/docs-git/objects/<blobSha> content-addressed file bytes
customers/<slug>/docs-git/commits/<commitSha> commit JSON ({ tree, parents, … })
customers/<slug>/docs-git/refs/heads/dev mutable HEAD
customers/<slug>/docs-git/refs/promoted promoted ref
The working namespace always mirrors HEAD. History is exposed in the dashboard’s
commit-tree shape (GET /commits); POST /rollback/:sha restores an older commit
as a new, non-destructive revert commit.
Backups (DocsBackupStore). POST /backup snapshots the working namespace to
customers/<slug>/docs-backups/<id>/ (files + manifest.json), independent of the
git store; GET /backups lists (retention keeps the newest 10), and
POST /backups/:id/restore re-applies a snapshot as a new commit.
Versioned documentation channels (platform docs)
The platform docs site (webs/fouria-docs, served at fouria.io) is published
per channel so an instance reads documentation and release notes that match
the code it runs:
| Channel | Source | URL |
|---|---|---|
latest | newest v* release tag | /latest/ (the root aliases /) |
staging | develop head | /staging/ |
buildEnvVars forwards FOURIA_DOCS_CHANNEL (latest only for a
production-platform production instance; staging for the staging platform and
every -dev environment — src/docs-channel.ts), FOURIA_DOCS_BASE_URL
(https://fouria.io/<channel>), FOURIA_DOCS_FALLBACK_URL
(https://fouria-docs.pages.dev/<channel>), and FOURIA_VERSION (the Worker
bundle’s CalVer). The fouria-docs skill and the dev-only Fouria Builder/Q&A
personas read those variables, then confirm the build via
<base>/versions.json before quoting release notes. versions.json records the
channel, source commit, and component versions; release-notes/ is compiled from
each component’s CHANGELOG.md, so a channel page only lists what is in that
channel’s code.
Deployment (.github/workflows/deploy-pages.yml): both channels are built from
their own checkouts and assembled by scripts/build-deploy-tree.ts — the
release tag supplies / and /latest/, develop supplies /staging/ — and the
combined tree is deployed to fouria-docs on a release and on every develop
docs change. The staging project previews the staging channel at its root.
Sandbox SDK 1.0 side-by-side foundation (SandboxV1)
Alongside the default legacy Sandbox path, the Worker carries an additive,
not-routed Sandbox SDK 1.0 foundation so the side-by-side move can be built
without a big-bang cutover. It is a foundation only — no tenant is on it and
it carries no durability claim.
- Class —
SandboxV1(src/sandbox-v1.ts), a user-ownedDurableObjectthat drivesthis.ctx.container(the 1.0 model). Exported fromsrc/index.tstogether withDirectoryBackupGateway(needed for the DO to passctx.exports.DirectoryBackupGatewaytoDirectoryBackup). - Registration (class + migration only) —
wrangler.jsoncmigrationv7(new_sqlite_classes: ["SandboxV1"]). There is no deploy surface: theSandboxV1DO binding and itsscheduling_policy: "durable_object"container application were removed in every environment (durable-agents audit finding F9), because they served no traffic while every deploy still built/pushed an extra image and registered an extra application.Dockerfile.v1and the class are kept; restoring a container entry plus the binding is the first step of the cutover (seedocs/plans/sandbox-v1-scaffold.md). - Default routing unchanged — the legacy
Sandboxclass +getSandbox()still serve every tenant. No route selectsSandboxV1; there is no customer routing change and no cutover. - Package / image pairing — root catalog installs
@cloudflare/sandbox@1.0.0(this DO) and@cloudflare/sandbox-legacy@0.13.0-next.776.1(the still-default legacy class, imported via the-legacyalias).Dockerfile.v1usesnode:24-trixie-slimand COPYs thesandbox-shimhelper from the matchingcloudflare/sandbox:1.0.0image (the shim tag must track the package version); the legacy base image no longer applies because 1.0 removes the in-image SDK server. - TTL (derived, capped at 6h) and keep-alive —
resolveInactivityPolicy()reusesresolveSleepAfter(): non-billable instances get a finite idle timeout (staging 15m, dev 2h) clamped to the 6h platform maximum accepted bysetInactivityTimeout(). Billable production sets the 6h maximum and renews it from a recurring alarm on the same DO (keep-alivepurpose): SDK 1.0 has no “never stop” timeout, sotimeoutMs: nullwould mean stop shortly after the DO goes inactive, not stay awake. The renewal alarm keeps the DO active (which resets the inactivity timer), re-applies the timeout in case the DO incarnation lost it, and never starts a stopped container. The snapshot cadence and a queued conversion share the same alarm slot, so clearing one purpose never deletes another’s alarm. There is noSANDBOX_SLEEP_AFTERbinding, and the provisioned instance size is carried inCONTAINER_INSTANCE_TYPE(adurable_objectcontainer chooses its size inctx.container.start(), including every wake). - Real
DirectoryBackupwith F4E1 at-rest wrapping — the additive DO instantiates the realDirectoryBackup(from@cloudflare/sandbox@1.x) againstctx.container+ctx.exports.DirectoryBackupGatewaywith a tenant-scopedsandbox-v1/<encoded-do-name>/R2 prefix inBACKUP_BUCKET. The SDK writes one flat plaintext object per record (<prefix><id>.tar.zst); the Worker then post-encrypts it in place to<prefix><id>.tar.zst.enc(F4E1/AES-GCM) plus a readable<prefix><id>.meta.jsonsidecar, and deletes the plaintext canonical object. Restore transiently decrypts.encback to the canonical key for the SDK, then always deletes that plaintext in afinallyblock; delete removes all three objects (strict for explicit user deletion, best-effort on cleanup). Plaintext therefore exists only in the short post-upload window and the restore/decrypt window. The master key stays Worker-only (env.BACKUP_ENCRYPTION_KEY, never passed to the container); the record’ssha256/sizeremain plaintext and the SDKDirectoryBackup.restore()verifies them on restore. WithBACKUP_ENCRYPTION_REQUIRED=truea missing key/meta fails closed before any upload or restore. - Still no scheduling, conversion, or cutover — Worker-owned periodic
snapshot scheduling + retention is not implemented (no approved defaults
wired; the intended targets are a 5m snapshot interval, retaining the latest
interval snapshots plus a 24h rolling window and daily 30d history, once a
scheduler exists). There is no 0.13
.sqsh.enc→ DirectoryBackup conversion, no customer cutover, and no default-routing change: the legacySandboxclass +getSandbox()still serve every tenant, no route selectsSandboxV1, and the complete restore-before-Gateway path is not wired for the v1 line. This is not a final durability/no-loss guarantee. Seedocs/plans/sandbox-v1-scaffold.md.
Sandbox SDK runtime lines
The default tenant runtime is the legacy Sandbox class, installed via the
@cloudflare/sandbox-legacy catalog alias of npm:@cloudflare/sandbox
0.13.0-next.776.1, paired with the docker.io/cloudflare/sandbox:0.13.0-next.776.1
image. The Worker package and the container image must stay on the same line —
the stable and 1.0 control protocols are mutually incompatible.
The side-by-side SandboxV1 Durable Object uses the stable @cloudflare/sandbox
1.0.0 package (see the Sandbox SDK 1.0 section above). The intended runtime is
selected by the SANDBOX_RUNTIME binding (legacy default | v1); until V1
implements the full Sandbox surface the legacy class keeps serving and
resolveSandboxRuntime() reports the requested V1 as pending (see
src/sandbox-runtime.ts).
Every tenant-facing site that needs a Sandbox obtains it through the single
runtime acquisition boundary acquireSandbox() in src/sandbox-contract.ts —
the request middleware (src/index.ts), the task buffer (src/tasks/index.ts),
the ActivityScheduler Durable Object (src/activity/scheduler.ts), and the
per-customer gateway launcher (src/gateway/process.ts). The boundary resolves
the runtime but always returns the legacy stub for now (a v1 request is still
pending), preserves the container identity (openclaw for the single-tenant
instance, customer-<slug> for a per-customer gateway) and the legacy idle
policy (defaults to buildSandboxOptions(env)), and never starts a container —
getSandbox() returns a Durable Object stub, so acquiring a runtime never wakes
a sleeping tenant or touches the v1 DO namespace. This is the single flip point
for the cutover.
Sandbox v1 migration gate (S1)
The first cutover slice (S1, contract C1/C2/C10) adds the authority and
identity gate, not the conversion (S2 owns convertToV1 and the DO-owned
restore). The gate is src/sandbox-migration/platform.ts:
canMigrate(env)is true only when the trustedWORKER_PLATFORMbinding is exactlystaging. Absent, empty, whitespace,production,dev,local,STAGING, and a request carryingHost: staging-<slug>orX-FourIA-Role: rootall refuse.migrationRefusal(env)resolves the canonical tenant withtenantSlugFromEnv(neverctx.containerId, neverresolveTenantSlug’s context fallback) and refusesnulland the reservedclientslug.- The
/controlentry runs the gate in a middleware before the sandbox stub is acquired, so a refused migrate performs no R2 write and no DO/container addressing. It returns403 MIGRATION_PLATFORM_FORBIDDEN,409 MIGRATION_SLUG_UNRESOLVED, or409 MIGRATION_RESERVED_CLIENT. An allowed staging request is served by the queue-only entry (S5): it enqueues the durable conversion on theSandboxV1DO and answers202(queued/running),200(committed +recordId) or502(SANDBOX_V1_CONVERSION_FAILED); the read-onlymigration-statusaction reportsabsent/queued/running/committed/failedand can never start work. See § “Sandbox v1 conversion orchestration” below. - The gate is migrate-only:
freeze,unfreeze,backup,checkpoint,checkpoint-park, andresumekeep their existing behavior on every platform and on the reservedclienttenant. sanitizeCustomerSlugrejects thedefaultsentinel in every resolver branch (C10), andSANDBOX_V1_CUTOVERdefaults tofalse— unselected tenants stay byte-identically on the legacy runtime.
Lerma’s provisioning and upgrade paths now both add the platform
bindings (WORKER_PLATFORM from cf_environment, SANDBOX_V1_CUTOVER=false)
and, on the staging platform only, the SandboxV1 DO binding
(Lerma.Cloudflare.SandboxV1). The per-tenant v1 container application is
created by Containers.deploy_script_and_container/1 from the same
platform/image predicate, with the v1 image supplied as a digest-pinned named
image (ctx.container.images.sandbox) because the durable_object policy takes
no application-level image. The upgrade path also re-asserts the tenant’s full
binding set, so it can rebuild a script that no longer exists, and an upgrade
rollback never deletes the live tenant’s script. Verified on staging
(2026-11-05); the staging Lerma was then restored to :latest-pre, so a
promotion from develop Lerma would drop the v1 bindings until the branch
lands. See
sandbox-v1-staging-observations.md.
Sandbox v1 conversion (S2)
SandboxV1.convertToV1() (src/sandbox-v1.ts) owns the data movement and the
conversion barrier; the Worker never runs the primitives in a request context.
- Intent (C5, C6).
src/sandbox-migration/intent.tswritessandbox-migration/<tenant>/intent.jsonbefore the first converting step. Reads are tri-state (absent/present/unknown); only an explicitabsentpermits legacy.presentorunknownmeans v1, permanently. - Lease (C4).
src/sandbox-migration/lease.tsis a per-tenant R2 lease: create-only acquire (etagDoesNotMatch: '*'), etag-CAS takeover of an expired/released tombstone, owner-scoped renew/release, and fail-closed parsing. The TTL is a liveness bound only. - Conversion (C7).
convertToV1()acquires the lease, writes/validates the intent, opens the legacy recovery point (stable-backup.json, F4E1-decryptingdata.sqsh.encwhen needed), streams the plaintext squashfs into the v1 container, extracts it withunsquashfs -f -d /home/openclaw, takes a v1DirectoryBackup(tar+zstd/1), writessandbox-migration/<tenant>/conversion.json, and publishessandbox-migration/<tenant>/stable.json.Dockerfile.v1installssquashfs-tools. - Barrier (C8).
verifyConversionRecord()completes only when the stable pointer names a shape-valid committed record. Readiness is a separate input (S3/S4);startGateway(env)is the DO-owned owner restart and waits for a real/health, but S2 only calls it when the caller supplies the launch env. - No fallback (C5). Once the intent exists, the Worker acquires the v1
runtime (S3 below); it never silently falls back to the legacy gateway.
ensureGatewayon a legacy stub still throwsSandboxV1RuntimePendingErroras a defense-in-depth check.
Sandbox v1 serving (S3 / OCM-1)
Runtime selection is the durable conversion intent, resolved once per request in
the global middleware (resolveRuntimeIntent). acquireSandbox(env, id, options, runtime) then returns the matching runtime:
legacy→ the 0.12getSandbox()stub (unchanged, byte-compatible).v1→V1SandboxClient(src/sandbox-v1-client.ts) wrapped around theSandboxV1DO.unknown→503 SANDBOX_V1_RUNTIME_UNKNOWNbefore any acquisition. Control actions on a converted tenant fail closed with503 SANDBOX_V1_CONTROL_UNSUPPORTED(S4 owns them) rather than acting on the wrong container.
Transport. The DO exposes a fetch() entrypoint: the Worker client sends the
request to the DO stub with X-FourIA-Sandbox-Port, and the DO rewrites it to
http://localhost:<port> before calling
ctx.container.getTcpPort(port).fetch(...). The entrypoint (not an RPC method)
is what carries a WebSocket upgrade Response back to the Worker. It never starts
the container: a stopped container answers 503 SANDBOX_V1_SLEEPING, so a probe
cannot wake an idle tenant.
Process RPC. ExecProcess handles cannot cross RPC, so the DO keeps them and
exposes v1Exec / v1ProcessOutput / v1ProcessWait / v1ProcessSignal /
v1ProcessState / v1ListProcesses (output is decoded to strings inside the
DO). V1SandboxProcess presents that as the SandboxProcess shape, so
startShell / runShell / waitForProcess and the CLI-backed admin routes keep
working.
Lifecycle. ensureGateway detects the v1 client and delegates to
ensureV1Gateway → DO v1EnsureGateway, which (1) adopts a gateway already
answering /health, (2) restores the committed v1 DirectoryBackup when the
container holds no tenant state (OPENCLAW_RESTORED=true for the launch), and
(3) runs start-openclaw.sh start as the single supervised owner, polling a real
/health before returning. The DO is the serialization point, so there is no
second owner even across Worker isolates. killGateway delegates to
v1KillGateway.
Health. getHealthSnapshot branches on the v1 client: v1Status() is the
non-waking ctx.container.running read (a passive probe on a stopped runtime is
sleeping, HTTP 200 at the route), readiness is a real /health response, 503
until ready, and /healthz stays the sub-second non-waking liveness read.
Not yet implemented on the v1 line (each fails with
SANDBOX_V1_UNSUPPORTED_OPERATION instead of an undefined-method crash):
createBackup / restoreBackup / directory backups (PER-2), file tools such as
writeFile / readFile / mkdir (OCM-3), and the lifecycle/eviction hooks
stop / stopIfIdle / onActivityExpired (OCM-2). Durable boot jobs and
scheduling still acquire the legacy stub (PER-3).
Sandbox v1 gateway proxy attribution
OpenClaw attributes proxy-shaped traffic (any Forwarded, X-Forwarded-*
or X-Real-IP header) against gateway.trustedProxies before Gateway auth.
It accepts the request only when the source address is trusted and the
forwarded client headers resolve to a non-loopback client; otherwise the
Gateway-authenticated routes answer 403
{"error":{"type":"proxy_attribution_required", …}}.
The Worker always rebuilds X-Forwarded-For / X-Real-IP from
CF-Connecting-IP (see rebuildForwardedClientHeaders in src/index.ts), so the
second condition holds. The first one depends on the ingress path, which a tenant
cannot configure:
| Runtime | Gateway sees the request from | In trustedProxies? |
|---|---|---|
legacy | the Cloudflare Sandbox container bridge (10.1.0.x) | yes — moltlazy unions 10.0.0.0/8 |
v1 | the ctx.container.getTcpPort() ingress owned by the platform | no (unknown to any tenant config) |
That made every proxied request on a converted/born-v1 tenant fail (the
reported symptom: proxy_attribution_required on all dev instances deployed
with the v1 runtime), including WebSocket upgrades and the admin-RPC reads,
because gateway.trustedProxies can only list addresses the tenant can know.
src/gateway/proxy-attribution.ts owns the fix. A request the Gateway refuses
cannot be attributed as a proxy, but a direct-remote request is always
attributed (OpenClaw only rejects proxy-shaped traffic it cannot resolve), and
it is served with the same token/device auth. SandboxV1.fetch() therefore:
- probes attribution once per gateway incarnation with a
GET /carrying the same rebuilt forwarded headers (X-Forwarded-For: 203.0.113.1)./is the Gateway-authenticated route that fails with the rejection, so its 403 is the definitive signal. A probe that cannot reach a still-booting Gateway leaves the modeunknown(traffic keeps its headers) and is retried by the next forward; - when the Gateway refuses attribution, strips every forwarded header from the
request before forwarding it, so OpenClaw attributes it as
direct-remote; - caches the verdict for the Durable Object’s lifetime and re-probes after a
gateway restart / container destroy (
v1KillGateway,v1Destroy,startGatewayOwner), so a healedgateway.trustedProxiesrestores client-IP fidelity without a Worker deploy.
Requests that carry no forwarded headers (container health probes, CLI reads)
skip the probe entirely, and the probe result is single-flighted so concurrent
requests at boot cannot fan out probes. Structured event:
sandbox.v1.proxy_attribution.direct (warn) — the direct fallback is active for
this gateway.
Trade-off: in direct mode the Gateway attributes the container-internal source instead of the real client IP (rate-limit subject and audit client IP). That is acceptable for a single-tenant gateway whose credential is the shared token, and it is strictly better than a 403. When attribution is accepted, nothing changes.
Provisioning an instance directly on Sandbox v1
A tenant can skip the legacy conversion entirely: Lerma provisions it on v1
(“born v1”), the Worker serves it on SandboxV1 from the first request, and
the tenant onboards inside the v1 container. The decision is durable in R2 and
written once, at provisioning time.
Marker contract. sandbox-migration/<canonical-slug>/runtime.json in the
tenant’s BACKUP_BUCKET:
{
"version": 1,
"tenant": "<canonical-slug>",
"runtime": "v1",
"mode": "fresh",
"provisionedAt": "<ISO8601>"
}
- Lerma writes it create-only, before the script deploy; the Worker only
ever reads it (
src/sandbox-migration/runtime-marker.ts). There is no env var, no endpoint and no Worker-side write path for it. - The read is tri-state like every other migration record (
readJsonObject): a transport error, malformed JSON, a non-matchingtenant, aversionother than1, anotherruntime, aconvertedmode, a missing/empty tenant or aprovisionedAtthat is not ISO 8601 with an explicit zone (Zor±hh:mm) isunknown, neverabsent. readFreshRuntime(env)is the route-facing convenience: it resolves the canonical tenant withtenantSlugFromEnvand reads the marker; an unresolved identity isunknown, and noBACKUP_BUCKETisabsent(a marker cannot exist without the bucket, so local/dev instances without R2 keep the historical behavior).
Resolution matrix (resolveRuntimeIntent,
src/sandbox-migration/intent-gate.ts):
| Platform | Conversion intent | Marker | SANDBOX_RUNTIME | Runtime |
|---|---|---|---|---|
not staging | any | any | any | legacy (unchanged; the platform is the only migration authority) |
staging | present/unknown + C8 record+pointer complete | any | any | v1 |
staging | present/unknown + barrier incomplete | any | any | pending |
staging | absent | present (valid fresh marker) | any | v1 |
staging | absent | unknown | any | unknown (fail closed) |
staging | absent | absent | v1 | pending (fail closed) |
staging | absent | absent | unset/legacy | legacy |
The conversion intent wins over a marker: a tenant that was converted (and historically also carried a marker) stays behind the C8 record+pointer barrier.
Fail-closed pending. SANDBOX_RUNTIME=v1 with no marker means the
instance asked for v1 but was never provisioned for it (the marker write and the
binding disagree). The Worker answers the existing
503 SANDBOX_V1_CONVERSION_PENDING and never falls back to legacy — a legacy
boot would create the exact legacy state the tenant is not supposed to have.
There is no “assume the marker” path and no automatic recovery: re-provision or
remove the binding.
Control plane. Neither migration action applies to a fresh tenant:
POST /controlX-FourIA-Control: migrate→409{ ok: false, action: 'migrate', code: 'MIGRATION_ALREADY_V1', error: 'tenant is provisioned on SandboxV1; there is nothing to convert' }, without addressing the DO or enqueuing (an unreadable marker →503 SANDBOX_V1_RUNTIME_UNKNOWN, since converting a tenant that may already be v1 would be destructive). The platform/slug gate still runs first, and a converted tenant has no marker, so it keeps its idempotent enqueue behavior.POST /controlX-FourIA-Control: migration-status→200{ ok: true, action: 'migration-status', state: 'absent', mode: 'fresh' }without touching the DO (never enqueues, never wakes). A converted tenant answersmode: 'converted'alongside its durablestate.backup/checkpoint/freeze/unfreeze/checkpoint-park/resumerun the existing v1 branches (snapshot through the DO, park/resume), so a fresh tenant is a first-class v1 tenant for the rollout machinery.
Boot and persistence. SandboxV1.restoreTenantStateIfNeeded() returns
false when the container holds no config and the recovery point is
pointer-absent with no conversion record, so a fresh tenant onboards
(start-openclaw.sh start runs the setup body) with OPENCLAW_RESTORED unset.
The first recovery point is taken right after that first successful boot.
v1EnsureGateway dispatches runInitialSnapshot() detached (registered with
ctx.waitUntil, never awaited), so the tenant no longer waits for the first
idle-cadence tick to become durable:
- When. Only for a tenant with no recovery point:
verifyRecoveryPointmust reportpointer-absentandconversion.jsonmust be absent. A converted tenant, a tenant that already snapshotted (cadence,backup,checkpoint-park), an unreadable pointer and any inconsistent state are left alone — the boot never snapshots blindly over an existing point. A parked tenant is skipped too (its park already committed the rollout record) and a container that is not running is never woken. - Guards. A single-flight promise (
#initialSnapshotRun) makes concurrent boots (the owner run and the adopted-gateway path both schedule it) fold into one attempt, a settled flag short-circuits the check once there is nothing to do (at most one R2 read per DO incarnation), andtakeOwnedSnapshot()gives the idle cadence and this attempt one shared owner so they cannot snapshot the directory concurrently — the loser skips and the cadence keeps its alarm armed. - Best effort. Any failure is caught and logged as
sandbox.v1.initial_snapshot_failed(bounded message); the boot staysreadyand the container keeps running. A failure is retried on a later boot: a new DO incarnation retries immediately, the same incarnation afterINITIAL_SNAPSHOT_RETRY_MS(10 min), so a broken backup cannot be hammered on every request. A success logssandbox.v1.initial_snapshotwithrecordId/size/took.
From then on the normal v1 persistence path applies: v1CreateSnapshot writes
the DirectoryBackup, catalogs it, advances
sandbox-migration/<slug>/stable.json (pruning by retention), and that pointer
is the recovery point everything restores (rollout park, restore by record id,
cold boot). A conversion record without a pointer still refuses to onboard over
it.
Residual window. The snapshot is detached, so a tenant that dies between
/health becoming ready and the archive committing (seconds to minutes on a
large directory, and longer on a failed + retried attempt) still has no durable
copy; the cadence remains the backstop for that sliver, and readiness is never
delayed by the backup.
Policy. Provisioning a fresh v1 tenant is limited to the staging platform
(including dev instances on it), like every other v1 lane: the resolution
matrix returns legacy outside WORKER_PLATFORM === "staging", so a marker
written on a production tenant is inert. The reserved client slug never
migrates and Lerma refuses to write a marker for it; the Worker deliberately
does not special-case client here — a marker is authoritative for serving.
Sandbox v1 idle snapshot cadence (PER-2)
A converted tenant’s container belongs to SandboxV1 (ctx.container), and raw
ctx.container has no eviction hook: the legacy onActivityExpired →
archive → stop path does not exist on SDK 1.0. The platform stops the container
after its inactivity timeout (applyInactivityTimeout, derived from
resolveSleepAfter), so without a cadence everything written since the last
committed record would be lost. The DO therefore owns a periodic snapshot:
- Arming.
startContainer()arms the cadence after the display/outbound setup succeeds (best-effort: a storage error is logged assandbox.v1.snapshot.cadence_arm_failedand never fails the boot). Arming writes a durable purpose marker (snapshot:cadence-pending) plusctx.storage.setAlarm(now + interval). The DO has one alarm slot and a queued conversion armsnow + 1s, so arming never postpones an earlier alarm — the marker, not the alarm time, tells the handler what to do. A restarted DO keeps the alarm (it is durable) andstartContainer()re-arms on the next boot. - Interval.
resolveSnapshotIntervalMs(env, inactivityTimeoutMs):SANDBOX_V1_SNAPSHOT_INTERVAL_MS(ms,0disables) → else the configuredBACKUP_INTERVAL_MINUTES→ else 1h. The result is capped at half the instance’s inactivity timeout when one is applied, because a snapshot due after the container has already idle-stopped is pointless (and the cadence must never wake it): staging (15m) → 7.5m, dev (2h) → 1h. The legacy 5-minute default is deliberately not inherited — it was tuned for the interval-gated cron path, and a 5-minute v1 cadence would snapshot back-to-back. - The alarm turn (
alarm()→runConversionAlarm()thenrunSnapshotCadence(); neither throws out of the handler):- disabled → clear the marker and stop;
#restoring(a manual restore owns the directory) → skip this turn, keep the cadence armed, so one overlap cannot silently switch the loss bound off;- container stopped → never wake it: clear the marker (the next
startContainer()re-arms); - the recovery point (
sandbox-migration/<tenant>/stable.json) is newer than the interval → skip (this dedupes with theActivityScheduler’s own interval-gated v1 snapshot); - otherwise
v1CreateSnapshot({ trigger: 'backup' }), loggingsandbox.v1.snapshot.cadencewithrecordId/took; - re-arm for the next interval.
- Teardown.
v1Destroy()clears the marker and the alarm — unless a conversion is queued on the same slot (deleting that alarm would strand it). - Parking. The cadence is deliberately park-agnostic: a parked tenant is drained, so a fresh snapshot is unnecessary but harmless (the catalog write is idempotent and touches R2 only).
Failures log sandbox.v1.snapshot.cadence_failed and re-arm; the previous
recovery point stays intact and the container keeps running.
Sandbox v1 restore by record id (PER-2)
POST /api/admin/storage/restore/:id?setCurrent=0 on a converted tenant applies
a catalogued v1 record instead of the legacy .sqsh archive:
- the route opens the durable maintenance window (
restore) before the DO touches the container, so a request arriving mid-restore cannot restart the gateway under the swap (ensureGateway’s v1 branch honors the window); SandboxV1.v1RestoreRecordstops the single gateway owner, restores the record’s directory intoHOME_DIR(the SDK swaps it atomically), and — by default — advances the stable pointer so a cold boot restores what the operator chose (?setCurrent=0applies the record without promoting it);- an unknown or unreadable record fails closed (
404) and never touches the container; a restore already in progress is refused by an in-DO flag; - the Worker then boots the restored state (
OPENCLAW_RESTOREDsemantics) and reportsgatewayReady. A boot failure does not undo the restore: the next request retries the boot, and the caller must not retry the destructive step.
v1 restores are synchronous: there is no legacy backup-job queue for the record format, and the SDK restore is bounded by the archive size.
Sandbox v1 outbound virtual hosts (OCM-1 / BIL-1)
The container reaches the Worker-owned services through virtual hosts instead of
the public internet: releases.r2.internal, docs.r2.internal,
leads.r2.internal, software.r2.internal (R2 namespaces resolved with the
Worker’s own binding), otel.internal (OTLP ingest from the tenant’s exporter),
cdp.internal (the container’s local CDP proxy forwarding Browser Rendering
commands to the tenant’s Sandbox DO through src/sandbox-cdp-bridge.ts →
src/cdp-session.ts — the browser credential never enters the container), and
ai.internal (the inference proxy in src/sandbox-ai-bridge.ts). The map lives
in src/sandbox-outbound.ts
(SANDBOX_OUTBOUND_ROUTES), the single source of truth for both runtimes:
- the legacy
Sandboxclass installs it as itsoutboundByHoststatics; SandboxV1registers each host withctx.container.interceptOutboundHttp(host, ctx.exports.SandboxOutboundGateway({ props: { host } }))instartContainer(). SDK 1.0 has no class statics — an intercept belongs to the container instance and disappears when it is replaced — so the routes are installed on every start, including a restarted DO that finds its container still running. A Worker image without the export logs a warning and leaves the container on direct egress instead of failing the boot.
Without this, a converted tenant silently loses OTLP export (AI cost
attribution), the R2 bridges and the agent browser (cdp.internal); the
container has no other way to resolve those names.
AI Gateway inference proxy (ai.internal)
AI Gateway capacity is account-scoped (20 gateways on the paid plan), so tenants share pools rather than owning a gateway. A gateway total is therefore never a client’s spend, and the account-scoped inference credential must not sit in a tenant container.
The route is opt-in (AI_GATEWAY_PROXY=true). It only takes effect when the
tenant’s primary model resolves under cloudflare-unified-billing/*; the
official cloudflare-ai-gateway provider ignores
CF_AI_GATEWAY_CUSTOM_DOMAIN and would send the derived token to Cloudflare,
which rejects it. Confirm the active model ref (Agent Studio’s model pane, or
openclaw models status) before enabling it on a tenant. When enabled, tenant
inference is routed through the Worker:
buildEnvVarssetsCF_AI_GATEWAY_CUSTOM_DOMAIN=ai.internaland hands the containerCLOUDFLARE_AI_GATEWAY_API_KEY/CLOUDFLARE_API_TOKENset toHMAC-SHA256(MOLTBOT_GATEWAY_TOKEN, "fouria-ai-proxy/v1")(src/gateway/ai-proxy-token.ts). The unified-billing plugin then callshttps://ai.internal/compat/....- The sandbox intercepts that host and runs
handleAiRequest(src/sandbox-ai-bridge.ts) in the Worker. - The handler verifies the derived token, resolves the tenant from the Worker
env (
resolveTenantSlug, never the request), maps/compat/*→/ai/v1/*, and forwards tohttps://api.cloudflare.com/client/v4/accounts/{account}/ai/v1/...with the real credential. - It always sets
cf-aig-metadatato the trustedtenant_id,environmentandplatform(plus the plugin’sopenclaw_run_id), and forcescf-aig-gateway-idfrom the trusted binding. A tenant cannot override either by editing its config.
Consequences:
- Attribution. Lerma’s development-expense report sums only log entries
whose
metadata.tenant_idmatches the client and whoseenvironmentis one of that client’s non-billable environments; legacy entries without metadata are counted asai_gateway_unattributed_requestsinstead of being assigned to a client. Cloudflare’s Analytics API can group the same metadata withmetadataValue(key: "tenant_id"), and spend-limit rules can split budgets by it. - No Access credential pair.
CF_ACCESS_TOKEN,CF_ACCESS_CLIENT_IDandCF_ACCESS_CLIENT_SECRETare no longer forwarded to containers. An Access-protected custom domain needs a per-tenant service token and service tokens do not producecf.user_id, so the pair could not deliver the identity attribution it was meant for. - Rollout to deployed instances. Lerma persists the opt-in on
instances.ai_gateway_proxyand re-deploys an already-running instance’s current bundle from the instance page (AI proxy toolkit action →Instances.set_ai_gateway_proxy/2). Only the binding changes; the image, gateway and container application are untouched, and the instance status is preserved. Upgrades carry the flag forward. - Custom-domain opt-out. Setting
CF_AI_GATEWAY_CUSTOM_DOMAINexplicitly keeps the direct path with the real credential; that domain must not be Access-protected. - Tests.
src/sandbox-ai-bridge.test.tspins the identity, metadata, gateway-forcing and path-mapping contract;src/gateway/env.test.tspins the derived-token wiring and the removal of the Access credentials.
Sandbox v1 rollout park (SEC-3)
A v1 tenant can now be parked for a container replacement, with the same guarantees as the legacy lane and one deliberate difference in ordering:
POST /control { X-FourIA-Control: checkpoint-park }on a converted tenant:- switches live sessions to
read-onlyand waits for their turns to end (drainLiveSessionsover the v1 gateway WS bridge; the previous modes are recorded durably withrecordPendingSessionModes), - commits the recovery point with
v1CreateSnapshot— fail-closed: a failed snapshot returns502 SANDBOX_V1_PARK_SNAPSHOT_FAILEDand the rollout never starts, because the replacement container has no other copy, - only then opens the durable park (
maintenance/replacement.json{ parked: true }) and arms the exec lock. A stopped container is already parked (skipped: 'not-running').
- switches live sessions to
- The park blocks writes through the existing layers:
tenantParkGuardrefuses mutating/api/admin/*and/debug/*requests with409 TENANT_PARKED, the/controlentry refuses every action exceptresumeandcheckpoint-park, andassertTenantExecAllowedstops Worker-initiated exec. Reads stay available. resumeclears the park record, unlocks exec, requests the boot through the v1 DO (which restores the committed record) and applies the recorded session modes once the gateway answers a real/healthprobe; if it is not ready yet the response sayssessionModes: "pending"and the durable map stays for the next boot. Resuming an unparked tenant is a no-op that never wakes it.
killGateway remains process-scoped on v1 (v1KillGateway signals the gateway
process); nothing in the park path destroys or recreates the container.
Sandbox v1 boot, owner and repair (OCM-2)
The durable lanes that run outside a request must resolve the tenant’s runtime themselves — they have no request-scoped sandbox to inherit:
- The
ActivitySchedulerresolves the runtime on every operation. Itssandbox()helper callsresolveRuntimeIntentand thenacquireSandbox(..., runtime). Before this, the scheduler always acquired the legacy client: after a conversion, an alarm-driven wake or a scheduled snapshot would have booted the old container, archived the legacy filesystem and advanced the legacystable-backup.json— silent divergence from the v1 recovery point.pending/unknowntenants are skipped (logged asactivity.sandbox.skipped), never guessed. - Scheduled snapshots on v1 write a v1 record. A
snapshotbackup job on a converted tenant callsSandboxV1.createSnapshot({ trigger: 'backup' })(one call: DirectoryBackup + catalog + pointer + retention) instead of the legacystage → archive → finalizepipeline.restore/deprovisionjobs on v1 are reported asskippedwith an explicit reason until those paths exist. restoreIfNeededno-ops on v1. The DO restores the committedDirectoryBackup(or onboards a fresh tenant) insidev1EnsureGateway/launchRestoredState. The legacy state probe (exists) is intentionally unsupported on the v1 client, so running the legacy restore step would readunknownand wedge the durable boot job. The boot job’srestorestep therefore returns immediately for v1 and thelaunchstep performs the restore; this is what lets the automation wake reuse the durable boot job instead of blocking the alarm onensureGateway.- The DO’s launch supervisor mirrors the legacy boot guards.
v1Execreturns a process handle, sostartGatewayOwnerraces the supervised launch’s exit against the/healthpoll: a non-zero exit before the port opens is a terminal launch failure reported in seconds (a zero exit stays ignored — the “lost the startup-lock race” path is judged by the health probe). On failure it tails/tmp/gateway-stdout.log; when the text matches a SQLite corruption signature it queues the durabledoctor-repairjob (gateway-launch-state-database) and fails with a retryable error, exactly like the legacy path. Relaunching cannot fix a malformed page, and the job re-queues the boot when the repair succeeds.
Sandbox v1 snapshots and the recovery point (PER-2)
On the legacy lane a snapshot is a .sqsh archive catalogued by
backup-handle.json; on v1 it is a DirectoryBackup record. The tenant needs
the same guarantees, so the v1 lane has its own catalog under the tenant’s
migration scope:
| Object | Purpose |
|---|---|
records/<id>.json | One catalog entry per committed record (size, sha256, format, trigger) |
stable.json | The stable pointer naming the committed recovery point (shared with the conversion) |
conversion.json | The migration artifact; doubles as the first catalog entry’s fallback |
v1CreateSnapshot()(DO).backupDirectory({ dir: '/home/openclaw' })writes the record and post-encrypts it in place with F4E1 (failing closed whenBACKUP_ENCRYPTION_REQUIRED=trueand no key is set). The catalog entry is written immediately after the archive, so a killed invocation leaves a findable record instead of an unlisted orphan, then the stable pointer advances and retention prunes oldest-first beyondBACKUP_MAX_VERSIONS(default 3) while never removing the record just taken. A failed snapshot leaves the container running and the previous recovery point intact. A stopped container is skipped, never started: a 1.0 instance loses its disk on an inactivity stop, soensureRunning()here would snapshot an empty/home/openclawand advance the recovery point over real data.v1CreateSnapshotreturns{ state: 'skipped', reason: 'not-running' };/controlfreeze/backup/checkpointreport the skip, and the scheduler’s snapshot job skips the cycle./controlrouting.backup,checkpointandfreezeon a v1 tenant are served by the DO (the legacy.sqshorchestration never runs for a converted tenant).freezetakes a snapshot and only then destroys the container — a failed snapshot keeps the tenant running (SANDBOX_V1_FREEZE_SNAPSHOT_FAILED) because the container is the only live copy until the record is committed.unfreezeis a no-op (the next request cold-boots from the recovery point). Other control actions still answer503 SANDBOX_V1_CONTROL_UNSUPPORTED.- Cold boot restores the pointer, not the conversion artifact. A snapshot
advances
stable.json, sorestoreTenantStateIfNeededrestores whatever the pointer names (records/<id>.json, falling back toconversion.jsonfor pre-catalog records) and then launches withOPENCLAW_RESTORED=true. A conversion record without a pointer is an inconsistent C8 state and is refused — onboarding over it would silently discard a converted tenant. - The C8 barrier is the pointer→record pair.
verifyRecoveryPointis the single read used by the runtime gate and the boot path: no pointer ispointer-absent, a pointer without a readable record isrecord-absent, an unreadable record isrecord-unknown, and a pointer naming a different record ispointer-record-mismatch. Only a matching, shape-valid pair iscomplete.
Sandbox v1 conversion orchestration (S5)
POST /control with X-FourIA-Control: migrate (forwarded by the Dispatch
Worker’s /_control/:slug/migrate) starts the legacy → v1 conversion. The work
streams the legacy squashfs, extracts it in the v1 container and uploads a v1
DirectoryBackup — minutes of I/O, far longer than a Worker request lives (its
background continuations are cancelled shortly after the response). The control
entry is therefore queue-only: it enqueues a durable job on the SandboxV1
DO and answers in milliseconds; the conversion runs on the DO alarm.
This is deliberate rather than incidental: the idle-snapshot cadence alarm
(§ Sandbox v1 idle snapshot cadence) fired on the live tenant lane and committed
record 4437c278-… with no operator action, which is the proof that alarms fire
on the Workers-for-Platforms tenant script. Queueing removes the request’s
wall-clock, CPU and disconnect exposure from a destructive data migration.
Contract
| Outcome | Status | Body |
|---|---|---|
| queued or in flight | 202 | { ok: true, action: 'migrate', state: 'queued' | 'running' } |
| already committed | 200 | { ok: true, action: 'migrate', state: 'committed', recordId } |
durable job recorded failed | 502 | { ok: false, action: 'migrate', state: 'failed', error, code: 'SANDBOX_V1_CONVERSION_FAILED' } |
SandboxV1 binding missing, or the DO unreachable | 503 | { ok: false, action: 'migrate', error, code: 'SANDBOX_V1_UNAVAILABLE' } |
The staging/reserved-slug gate (403/409) is unchanged and still runs in
middleware before any DO access, so a refused migrate performs no R2 write
and no container addressing. The route logs a structured
sandbox.v1.conversion.requested event (state, attempt, recordId; no
secrets). A transport failure is reported as 503 and not as failed: nothing
was recorded, so the same call is safe to retry, whereas failed is a durable,
terminal state an operator must act on.
Reading the status without starting a conversion (migration-status)
migrate is an idempotent enqueue (it starts work from absent/failed), so
a poll that reused it would convert a tenant just by asking. The read-only
action migration-status (X-FourIA-Control: migration-status, forwarded by the
Dispatch Worker as /_control/:slug/migration-status) exists for polling:
| Tenant runtime | Outcome |
|---|---|
legacy (no conversion intent) | 200 { ok: true, action: 'migration-status', state: 'absent' }, no v1 DO addressed |
pending / v1 | 200 { state: 'queued' | 'running' | 'committed' | 'failed', recordId?, error? } |
| v1 binding missing, or the read fails | 503 { ok: false, action: 'migration-status', error, code: 'SANDBOX_V1_UNAVAILABLE' } |
It calls only v1ConversionState() — never v1RequestConversion(), so it
cannot enqueue, never starts work and never wakes the container. The one write it
may perform is the lost-alarm re-arm (ensureConversionArmed() on a
non-terminal job), which arms the alarm slot but launches nothing. error (the
job’s bounded failure message) is included only for state: 'failed'. The action
is allowed through the runtime middleware for pending and v1 tenants and is
not part of the staging-only gate: reading a conversion state is not an action.
Durable job
- The job is the DO storage marker
migration:convert-request({ requestedAt, attempt }); the state the control plane polls ismigration:convert-status(queued→running→committed|failed, plus the 1-basedattempt). Both survive eviction and code updates. v1RequestConversion()is idempotent:committedreturns as-is (including when only the R2 conversion record survives a DO storage reset), an in-flight job returnsqueued/running, and onlyabsent/failedenqueues new work —failedstarts the nextattempt.- Arming never postpones an earlier alarm: the conversion and the cadence share
the DO’s single alarm slot, so
armConversionAlarm()only writes when there is no pending alarm or the pending one is later. Conversion arming wins. alarm()runs the conversion first, then the cadence, each in its own try/catch so one failing never starves the other, and finally re-arms a job that is still non-terminal (a backstop for a crash inside a turn).- A lost alarm is healed on the read paths:
v1ConversionState()(the poll),v1RequestConversion(),v1Status()andv1EnsureGateway()all callensureConversionArmed(). It re-arms only when the durable marker exists and the state is non-terminal, so a queued job can never sit forever; a terminal state with a leftover marker is cleared instead. The read paths themselves never wake the container — only the conversion turn does, by design. - Concurrency:
#conversionRunis the single owner, mirroring#gatewayStart. A second alarm turn (or a read path that armed one) folds into the in-flight run instead of starting a second destructive conversion. The conversion’s own lease/intent rules keep a crashed attempt idempotent on the next turn. - Ordering is the crash contract:
runningis durable before the conversion starts,committedis durable before the marker is dropped and before the gateway launch is attempted, and a failure recordsfailed(bounded message: 500 chars) without throwing. A launch failure therefore can never retract a commit — the nextv1EnsureGatewayretries the boot, whereasfailedwould re-run the destructive conversion. The container environment is rebuilt inside the DO from its own bindings (no secrets in DO storage). - There is no synchronous conversion entry point: it was removed from the DO, the RPC surface and the client, so a request cannot await the migration.
The conversion still depends on a committed legacy recovery point
(stable-backup.json); a tenant without one fails closed with a recorded
reason and must be checkpointed first. Because the intent is a point of no
return, the runtime gate requires the C8 barrier (conversion record + stable
pointer) before a tenant serves on v1: an intent without it is pending and the
Worker answers 503 SANDBOX_V1_CONVERSION_PENDING (only the idempotent
migrate entry stays reachable), so a failed conversion can never wake a fresh
empty v1 container over the legacy recovery point.
Key differences from the 0.12 stable line, all funneled through
src/gateway/sandbox-exec.ts:
- No
startProcess/execStream/getLogs().sandbox.exec(argv)starts a process and resolves once it is launched, not once it finishes; it never implies a shell.startShell(sandbox, script)wraps a shell command line in['/bin/bash', '-lc', script]and returns the handle;runShell(sandbox, script)waits and buffers{stdout, stderr, exitCode};outputOf(proc)/collectLogs(proc)read an existing handle. listProcesses()returnsProcessStatus[](.state, argv.command,.startedAt) with no methods.findExistingGatewayProcessfetches a live handle viagetProcess(id)and returns aGatewayProcesswrapper (.state,.commandstring,.refresh(),.waitForPort(),.kill(),.getLogs()).waitForProcessdelegates towaitForExit({ timeout })and kills the process on a local wait timeout (the SDK timeout cancels the wait only).- Busy-DO hardening for gateway reads. A contended container-control channel
can hold an
execlaunch for the SDK default (120s) and stall the Sandbox DO. Read routes therefore bound their sandbox work and degrade instead of hanging past the client timeout:ensureGateway(waitForReady:false)(every API read route) is bounded byENSURE_GATEWAY_READ_TIMEOUT_MS(30s) and returnsnullon expiry; the caller serves its normal “not ready” payload. Cold-start (waitForReady:true) is exempt.- A short-TTL positive readiness result (
GATEWAY_READINESS_CACHE_TTL_MS, 5s) collapses concurrent read-path probes onto one; it is cleared on kill/restart/recovery. - The gateway port probe (
isGatewayPortOpen) is bounded byPORT_PROBE_TIMEOUT_MS(10s) with a container-sidecurl -m, and marks the expected closed-port exit (curl7 = connection refused) withreportNonZeroExit: false(operationgateway.port-probe) so the normal “gateway down, start it” outcome is not logged as asandbox.process.failederror on every cold start — matching thegateway.health-probebehavior. - Expected CLI non-zero exits are not logged as failures either:
runShell(viareportNonZeroExit) andwaitForProcess(same option) let a call site mark an exit it handles as normal. The knowledgewiki statusprobe uses it (operation: 'knowledge.wiki-status') because an uninitialized vault exits non-zero and the handler reportsinitialized:false. Timeouts always log. - Route flows that poll the container wrap their whole flow in a budget
(
/api/admin/devices25s,/debug/container-config20s).
- No transport selection.
SANDBOX_TRANSPORTwas removed; the preview uses RPC automatically. SandboxOptions.labelswas removed upstream (worker identity is no longer attached at container start).
Deploy cutover: because the protocols are incompatible, a production deploy
must roll containers in one step — wrangler deploy --containers-rollout=immediate
— with rollout_active_grace_period at 0. In-flight processes/terminals stop
and pre-deploy process IDs become invalid.
Debug tools on Sandbox v1 (ROOT diagnostics)
The ROOT /debug/* surface (DEBUG_ROUTES=true, ROOT only) is the operator’s
escape hatch when a tenant misbehaves. A converted tenant is served by
V1SandboxClient (SandboxV1 DO), not the legacy Sandbox, so each route was
audited against the v1 surface (exec, getProcess, listProcesses,
containerFetch, wsConnect, killGateway, destroy, status,
isRuntimeActive) and adapted where the legacy call had no v1 equivalent.
| Route | What it calls | v1 | Notes |
|---|---|---|---|
GET /debug/version | runShell('openclaw --version' / 'node --version') | ✅ works | exec → waitForExit → output() are all implemented on the v1 client. |
GET /debug/processes | listProcesses() | ✅ works | v1 returns serializable ProcessStatus-shaped entries (.state, argv .command, no methods). The list is the DO’s tracked map, so it is empty after a DO eviction — not a container inventory. |
GET /debug/processes?logs=true | getProcess() + logs | ✅ works | A finished process is read with output() (the v1 handle has no replayable logs() stream). A running process is never read (logs_error: 'process running — live log stream disabled'). |
GET /debug/gateway-api?path= | containerFetch on the gateway port | ✅ works | v1 fetch forwards to ctx.container.getTcpPort(); a stopped container answers 503 SANDBOX_V1_SLEEPING instead of waking. |
GET /debug/cli?cmd= | startShell + waitForProcess + outputOf + status() + exitCode | ✅ works | Same CLI semantics as legacy; DEV_MODE instances still need OPENCLAW_GATEWAY_TOKEN forwarded (already done). |
GET /debug/logs[?id=] | gateway tail / finished-process output | ✅ works | No id → findExistingGatewayProcess → getLogs() tails /tmp/gateway-stdout.log. With id → output() fallback when logs() is unsupported. |
GET /debug/gateway-logs | findExistingGatewayProcess + getLogs() | ✅ works | Always the file tail (a live SDK log stream never settles). The response now carries path and runtime. |
GET /debug/env | Worker bindings only | ✅ works | Adds sandbox_runtime_actual (the durable per-tenant lane this request was served by) next to the env-level sandbox_runtime switch. |
GET /debug/container-config | exec cat …/openclaw.json + outputOf | ✅ works | 20s end-to-end budget unchanged. |
POST /debug/sqlite-integrity | runShellResilient (inline) / ActivityScheduler (durable) | ✅ works | The durable path resolves the tenant’s runtime itself (scheduler.sandbox()), so the repair job runs on v1 too. |
GET /debug/sqlite-integrity/jobs/:id | ActivityScheduler read | ✅ works | DO-only. .../jobs/:id/ws is the streaming variant. |
GET /debug/gateway-token | secret binding | ✅ works | No container access. |
POST /debug/stop-gateway | findExistingGatewayProcess + killGateway | ✅ works | killGateway delegates to v1KillGateway (process-scoped, with a best-effort pkill sweep). |
POST /debug/restart-openclaw | killGateway + cache resets | ✅ works | v1 branch: maintenance window → kill → resetStartupCache + clearPersistenceCache + resetV1GatewayStart → close window; the next request relaunches. No checkpoint and no restore-needed: a v1 restart destroys nothing and the marker is legacy-only. Same response fields + runtime: 'v1'. |
POST /debug/destroy-container | destroy() | ✅ works | v1 destroy tears down the container and clears the DO’s tracked processes/alarm; the route also clears the persistence/startup caches and the isolate start dedupe. |
POST /debug/reprovision | v1 snapshot + C8 recovery-point check + destroy() | ✅ works | The checkpoint is v1CreateSnapshot({ trigger: 'backup' }) (bounded by REPROVISION_CHECKPOINT_TIMEOUT_MS); “restorable” is verifyRecoveryPoint (pointer→record), not backups/<id>/meta.json. Response carries recordId and runtime: 'v1'; still fails closed (409) without a restorable copy. |
POST /debug/backup/:mode | v1CreateSnapshot (v1) / scheduleBackupIfNeeded (legacy) | ✅ works | On v1 the snapshot is one DO call (DirectoryBackup + catalog + pointer + retention) and needs no gateway, so the route never boots one. sync returns recordId. |
POST /debug/rescue-container | containerFetch probe + Containers API + destroy() | ⚠️ partial | Diagnosis and destroy work, but v1 containerFetch never wakes a stopped container: a sleeping tenant reads as unreachable. Use it for an orphaned/Error container, not to inspect a sleeping one. |
POST /debug/r2-put?key= | R2 binding | ✅ works | No container access. |
POST /debug/sleep | durable deprovision job (legacy) / stop() | ❌ not supported | v1 has no stop primitive (the tenant sleeps on the platform inactivity timer). Answers 503 { code: 'SANDBOX_V1_SLEEP_UNSUPPORTED' } instead of a 202 for work that never happens. |
Operational caveats:
- Access blocks headless
/debug/*calls on tenant hostnames (the per-tenant Access app answers302to the login page). Drive these routes from a ROOT browser session onhttps://<tenant>/_admin/(the Debug Tools cards), or from Lerma’s service token — not from a container-sidecurl. DEBUG_ROUTES=trueis Worker-local and/debug/*is ROOT-only; neither is provisioned by Lerma for a client instance, so this is a staging/dev/operator surface.- Process ids are per DO incarnation. After an eviction/redeploy the v1
process map is empty (
/debug/processesshows nothing,/debug/logs?id=404s) even though the gateway is healthy — use/debug/gateway-logs(the file tail) instead. - Nothing here is destructive beyond what it says:
restart/destroy/reprovisionnever touch R2 on v1 exceptreprovision’s own snapshot (which advances the recovery point) andrescue’s Containers-API image repair.
Container Image Build Profiles (dev vs production)
The Dockerfile takes a BUILD_PROFILE build arg (default production):
- production (default) installs only
xz-utils— the slim image (~1.36 GB) used by production and by E2E. - dev additionally installs the optional PDF/document tooling
(
pandoc+weasyprint— a large Python/cairo/pango stack, ~760 MB) that the dev-only Fouria Builderfouria-leadsskill needs (apps/fouria/skills/agent-only/fouria-leads/SKILL.md; the Builder is injected only whenOPENCLAW_DEV_MODE=true).
The profile is selected per wrangler environment via image_vars (Cloudflare’s
build-time variables, equivalent to docker build --build-arg):
| Environment | BUILD_PROFILE | Image |
|---|---|---|
env.staging (dev/test) | dev | with pandoc/weasyprint |
env.production | default (production) | slim |
| default / local Vite dev (local + E2E) | default (production) | slim |
There is a single image definition (apps/fouria/Dockerfile); CI
(test.yml) builds it with no build args when it pushes unstable-<sha>, and
the release CD (deploy-fouria.yml) promotes that same image by retagging it, so
released images are the slim production profile. The E2E basic (PR) and
production (develop) profiles both provision the slim production image (only
instance type/env differ). To build a dev image locally:
docker build --build-arg BUILD_PROFILE=dev -f apps/fouria/Dockerfile .Sandbox Recovery (stale + orphaned)
When the platform terminates a container behind a live Sandbox DO (idle
recycle, maintenance, SIGTERM), every RPC into it fails with the SDK’s
HTTP error! status: 500 wrapper — and retrying against the same instance
fails forever, bricking the tenant. gateway/sandbox-recovery.ts classifies
these dead-container errors (isStaleContainerError) and recovers
(recoverStaleSandbox): destroy the instance, reset the startup-probe and
persistence caches. The next RPC auto-provisions a fresh container and the
normal cold-start flow runs (R2 restore → start-openclaw.sh → gateway).
A container destroyed out-of-band (Cloudflare Containers dashboard “Destroy”,
failed operator force-stop) is a second failure class: the DO is alive but points
at a placement that no longer exists, so RPCs fail with the SDK’s retryable
admission errors (no container instance available, maximum number of running container instances exceeded) instead of the 500 signature. Those are legitimate
during a cold start, so they are not treated as stale on their own.
isContainerUnavailableError + trackProvisioningFailure escalate them to a
recovery once the streak reaches ORPHANED_INSTANCE_MIN_FAILURES and spans
ORPHANED_INSTANCE_GRACE_MS (120s — longer than the SDK’s ≈110s provisioning
budget, so a slow-but-healthy cold start is never cancelled mid-provision). The
streak is cleared whenever a sandbox RPC succeeds. maybeRecoverSandbox unifies
both classes for callers.
Recovery is wired into three failure points in gateway/process.ts:
findExistingGatewayProcess (listProcesses failures), checkGatewayHealth
(exec failures), and the ensureGateway startShell path (recover + retry
once). It is cooldown-guarded (STALE_RECOVERY_COOLDOWN_MS, 60s) and never
fires for deploy-window OperationInterruptedError. Permanent image errors still
require the rescue tool (rescueContainer, see OPERATOR-GUIDE §5.1). Logs use the
[SandboxRecovery] prefix.
Transient transport drops (retry, don’t reprovision)
A third failure signature is not a dead container: Network connection lost.
(and similar socket/WebSocket transport messages). It appears when the DO↔container
connection drops while the container is alive — most visibly while the Gateway is
applying a config/plugin reload (e.g. adding a model provider in the Control UI),
which restarts the Gateway and briefly tears down the container RPC/WS transports.
isContainerTransportError (gateway/sandbox-recovery.ts) classifies these, and
isStaleContainerError explicitly excludes them (even when the SDK wraps the
message in its sandbox.<op> error prefix) so a transient drop never destroys a
live instance. ensureGateway instead retries with a short backoff: the port
probe (isGatewayPortOpenResilient, up to TRANSPORT_RETRY_ATTEMPTS) and the
startShell spawn (once). Before this, a reload that dropped the transport mid
probe fell straight through to start-openclaw.sh → Failed to start process →
a 502 to the user, even though the Gateway came back seconds later. The retry
keeps the request alive across the reload window.
A Durable Object code update (a normal Worker/DO redeploy, or a Lerma
redeploy of the tenant script) resets the control session between the DO and its
container even though the container itself is not replaced. The cold start then
loses the transport on the supervised port wait: the atomic
start-openclaw.sh start process (config setup + gateway under one lock) keeps
running, but waitForPort() on the stale handle rejected and previously aborted
the cold start — killing a gateway that would have completed (observed as
Network connection lost.). The atomic start path has no separate
setup-process wait, so only the gateway port wait reattaches via
getProcess(id) and resumes (waitForPortResilient,
gateway/sandbox-exec.ts), bounded by the shared STARTUP_TIMEOUT_MS deadline.
The generic waitForExitResilient helper remains in sandbox-exec.ts for the
legacy/boot-smoke supervised waits. Only a non-transport error, an
un-reattachable process, or an exhausted deadline still fails the cold start.
The openclaw doctor diagnostics run through the same transport-resilient path:
runShellResilient, which retries the exec launch (startShellResilient) and
reattaches the wait (waitForExitResilient). Previously a transient
The operation was aborted drop on the doctor startShell surfaced as a hard
500 { ok: false, error: "The operation was aborted" }; it now reattaches to the
process and returns the structured diagnostic instead. GET /api/doctor still
runs inline this way; POST /api/admin/config/fix and POST /debug/sqlite-integrity
now enqueue a durable maintenance job on the ActivityScheduler DO
(activity/maintenance-jobs.ts + activity/maintenance-executor.ts) and return
202 { accepted, jobId }, because a large integrity check or doctor repair can
exceed a Worker request’s lifetime (the old 30s/90s shell timeouts surfaced as
“timeout in under 2 minutes”). The DO alarm runs the command under a 14-minute
lease with bounded retries (runMaintenanceOperation, still runShellResilient
internally) and stores the bounded result on the job record, polled via
GET /debug/sqlite-integrity/jobs/:id / GET /api/admin/config/fix/jobs/:id.
Both endpoints fall back to an inline run when the ACTIVITY binding is absent
(local dev / tests). The doctor repair runs openclaw doctor --fix --non-interactive
(the supported command for index-only integrity_check corruption — missing or
wrong-count index entries — which verifies the table data, preserves a backup,
and REINDEXes) and quiesces the gateway first: runMaintenanceOperation
opens the cross-isolate maintenance window (beginMaintenance(..., 'doctor-repair')),
kills the gateway so the repair can take exclusive state ownership, then closes
the window so the next request re-launches the gateway.
Restore → repair ordering (restore-config-pending.json). A restore replaces
the container’s writable layer but launches nothing, so a config repair applied
around a restore is silently overwritten by the next re-apply of the recovery
point (observed on tenant-tarjetero-dev: a config fix was lost on the next
gateway restore). Every successful restore (restoreIfNeeded and
restoreBackupById) now writes a durable restore-config-pending.json marker.
The durable boot job and the durable restore job launch the gateway on the
restored state through the shared launchRestoredState helper
(src/post-restore.ts); the admin sync/async-fallback restore route ends there
too, so every restore path has the same ordering guarantee. The helper calls
ensureGateway({ waitForReady: true }) and then
captureRestoredConfigVersion — writing configs/<ts>.json and
stable/config.json, so the restored config becomes the newest and stable
config version and the dashboard’s diff/history reflects what is actually live.
The marker is cleared only after that capture. A gateway-launch failure never
re-runs the destructive restore: restore jobs stay succeeded with
detail: "gateway-launch-pending" and the sync route returns
gatewayReady: false; the pending marker drives the next boot / maintenance
tick. A doctor-repair job checks,
before running: an active restore job, an open maintenance window / in-flight
restore (getPersistenceBusyReason), or a pending restored-config marker. In any
of those cases it defers (MaintenanceJobDeferredError): the job keeps its
attempts and queue position (60s re-check) instead of failing its three attempts
while the restore runs, and executes only after the gateway is up on the restored
state.
Automatic repair when the launch is blocked by a corrupt state database.
A warm R2 restore can bring back a damaged state/openclaw.sqlite (or an agent
openclaw-agent.sqlite). OpenClaw’s schema preflight reads PRAGMA user_version before anything else, so the Gateway dies within seconds with
Error: database disk image is malformed (readSqliteUserVersion →
inspectStateDatabaseSchema) and exits — no amount of relaunching helps. Two
things made that a multi-hour loop where the loading page sat on “Starting
OpenClaw Gateway”:
- The crash is now detected immediately. The readiness wait races a third
signal,
watchGatewayEarlyExit: the atomic cold startexecs the Gateway, so a non-zero exit of the supervised process before the port opens is a terminal launch failure. Previously the Worker waited out the whole sharedSTARTUP_TIMEOUT_MSbudget (≈5 min) before even reading the diagnostics. A zero exit is deliberately ignored — that is the “lost the startup-lock race to a healthy owner” path, decided by the existing port/lock adoption check — and transport/plumbing errors fromwaitForExit(a DO code update resets the control session) park the watcher instead of failing a healthy boot. - The documented repair is queued. When the launch diagnostics match a
corruption signature (
gateway/state-db.ts:database disk image is malformed,file is not a database,SQLITE_CORRUPT, …),ensureGatewayenqueues the durabledoctor-repairmaintenance job and throwsGatewayStateDatabaseRepairError(retryable). The job carriesreason: 'gateway-launch-state-database', which makes it (a) skip the post-restore gateway gate inpreflight— waiting for a gateway the corruption prevents is a permanent deferral; the restore-safety checks stay — and (b) re-queue the durable boot when the repair succeeds, since the launch itself is what failed and no request would retry it. Re-requesting is deduped (MAINTENANCE_JOB_BUSY→ keep the active job), so repeated failed launches cannot pile up repairs. Without theACTIVITYbinding (local dev) the launch reportsno-schedulerand the operator runs Repair Environment.
The loading page follows it: /api/startup/ws pushes repair (the DO’s active
maintenance job) and the page renders “Repairing the OpenClaw state database
(openclaw doctor –fix, attempt N)…”, suppressing the slow-step notice — the
container’s own startup-progress.json still names the launch step the repair is
recovering from, which is exactly the stale “Starting OpenClaw Gateway (311s)”
message.
Job status is pushed over a WebSocket — GET /api/admin/config/fix/jobs/:id/ws(and the /debug/sqlite-integrity/jobs/:id/ws variant) — instead of the
dashboard polling the JSON endpoints above every 2s (one request + Worker log
line per tick for the whole multi-minute repair). The streamer
(routes/api/maintenance-ws.ts) reads the durable record from the
ActivityScheduler DO, sends a frame only when it changes, and closes 1000
once the job is terminal; the dashboard’s subscribeMaintenanceJob falls back to
a slow HTTP poll only while the socket is down. While a job is active the
streamer also calls the DO’s rearm() self-heal, so a job whose alarm was
lost is scheduled again immediately.
The shared alarm is written through an optimistic-concurrency guard
(armWithGenerationGuard): the ActivityScheduler owns one alarm, so three job
enqueues, the warm-path schedule refresh, and the alarm handler’s own trailing
re-arm can interleave at await points. A stale arm that read “no pending job”
could execute setAlarm(later)/deleteAlarm() after a just-enqueued job’s
setAlarm(now) and strand the job until the later time — observed as a
doctor-repair job enqueued at 12:57:30 that only began at 13:02:11 (exactly the
previous boot job’s 14-minute lease). The guard bumps a stored generation and
abandons a superseded arm’s write, so the newest arm (fresh read) owns the
alarm.
While the control session reconnects, the loading page’s GET /api/startup/ws
stream treats a transport-drop snapshot as transient: it re-sends the last known
state (or a synthetic starting view) and backs off exponentially
(STARTUP_WS_POLL_INTERVAL_MS → STARTUP_WS_MAX_BACKOFF_MS, 30s) instead of
re-probing — and re-logging the same error — every 5s. A successful probe resets
the cadence.
The manual lever is POST /debug/reprovision (DEBUG_ROUTES, ROOT): it
checkpoints the live container first and fails closed — unless it just
committed a snapshot or a restorable backup already exists it refuses with
409 rather than silently destroying tenant state; {"force":true} is the
explicit override for an already-unrecoverable container. The checkpoint is only
attempted when isRuntimeActive() reports a live runtime (so an idle container
is never woken) and is time-bounded (REPROVISION_CHECKPOINT_TIMEOUT_MS); the
restorability check HEADs the newest catalog entries’ meta.json rather than
trusting the entry count. On success it destroys the instance, clears the
startup/persistence caches, and resets the orphan streak without ever contacting
the gateway — so it works when the container is entirely gone. Compare
POST /debug/restart-openclaw (gateway-aware, checkpointed) and
POST /debug/destroy-container (teardown only, used by E2E).
Activity-Driven Wake (ActivityScheduler)
Automations (“jobs”) run inside the Gateway, so the Gateway must be running
for a schedule to fire. Historically a Worker cron trigger woke every tenant on
every tick (expensive and since removed). src/activity/ replaces it with a
per-tenant activity clock:
ActivitySchedulerDurable Object (SQLite-backed, bindingACTIVITY) — stores a mirror of the tenant’s automation schedule and owns a single alarm. Deliberately a separate DO from the SDKSandboxDO, which already uses its own alarm forsleepAfter.- Mirror — warm paths (
GET /api/health, only whenreadiness.ok) readopenclaw automations list --jsonand push the earliestnextRunAtMsinto the DO. The DO throttles this to once per 5 min, and it never wakes a sleeping container. - Mirror bootstrap — the first mirror must not wait for a dashboard poll: the
durable boot job mirrors the schedule right after
launchRestoredState(activity.schedule.mirrored). A converted tenant that never opens the dashboard therefore still has a wake plan before it sleeps. - Alarm (
activity/alarm.ts) — firesCRON_WAKE_AHEAD_MINUTES(default 10) before the next job. On legacy it callsensureGateway({ waitForReady: false })(fast; the schedule read retries until the gateway serves). On v1 it enqueues the durable boot job (deduped) and returns{ ready: false }instead of blocking the alarm turn on a restore + launch; the boot job’s launch step mirrors the schedule when the gateway is ready, and a 60s fallback check covers a lost mirror. No CLI read runs against a booting container, and the read is gated on the non-wakingisRuntimeActive()read. No enabled jobs → alarm cleared → the container sleeps. A due-but-pending job is woken once per occurrence:wokenForsurvives mirror writes and resets only when the gateway reports a different next run, so a wake-ahead cannot turn into a 60s wake/CLI loop while the job is pending. A woken occurrence that does not advance backs off 5m → 30m; a failed wake/read keeps the schedule and backs off (bounded), it never deletes the only trigger a sleeping tenant has. Every activity re-arm goes through the shared combined alarm, so it cannot postpone a boot/maintenance/backup job. - Non-waking diagnostics — the alarm’s memory watch checks
isRuntimeActive()before its/procexec. On v1 that exec would start a stopped instance (the disk is gone), so a skipped scheduled snapshot or a failed wake can never warm a sleeping tenant by accident.
Net: healing/wakeup is independent of user traffic, gated on real job activity,
and free when idle. Traffic may cold-start the container; jobs get a pre-warmed,
healed one. Requires the ACTIVITY binding in wrangler.jsonc and in the lerma
provisioner (tenant_provisioner.ex resource_bindings) + the
new_sqlite_classes migration (dispatch_namespace.ex @do_sqlite_classes).
Change-Aware Backups
Scheduled snapshots are gated on whether tenant state changed. Detection is on by default; the watch loop is opt-in:
| Flag | Effect |
|---|---|
BACKUP_CHANGE_DETECTION (true default) | Phase 1 (cost). The scheduled snapshot path (scheduleBackupIfNeeded) runs a container-side scan first. Unchanged state ⇒ no squashfs, no legacy staging, no lease, no upload. Only an explicit false/0/off disables it. |
BACKUP_CHANGE_WATCH=true + BACKUP_WATCH binding | Phase 2 (RPO). The per-tenant BackupWatch DO checks every BACKUP_CHANGE_CHECK_MINUTES (2) and requests a durable snapshot from ActivityScheduler once a change has been quiet for BACKUP_CHANGE_DEBOUNCE_MS (60 s) and the floor (BACKUP_INTERVAL_MINUTES) has elapsed. |
Detection is default-on because a snapshot is not a cheap probe: it runs
mksquashfs over the whole tree, uploads the archive, and re-streams it through
the Worker for F4E1 encryption. Measured on fouria-backup-tarjetero-dev, two
snapshots 37 minutes apart differed by 0.08 % (348 KB of 428 MB) — every tick
that re-uploaded it paid for the full archive. Only an explicit opt-out turns the
gate off, so a tenant that omits the var still gets the saving.
The scan itself costs one find, so the trade is deliberately lopsided: an
unnecessary scan is negligible, an unnecessary snapshot is not.
Eviction compensates. Because a quiet tenant now takes no scheduled
snapshot, its newest archive legitimately ages past the idle window — which would
make TenantSandbox.onActivityExpired() re-archive the whole tenant once per
sleep cycle. The hook therefore runs the same scan and treats unchanged as
coverage, stopping inline (see Backup before
deprovisioning).
Scan (src/change-detection.ts). Runs find over the real state roots (/home/openclaw, /root/openclaw/.openclaw, /root/clawd; the home directory is symlink-only for state, and find does not follow symlinks). It prunes the transient legacy stage and .npm. It compares against the baseline marker /tmp/.fouria-last-backup and prints unchanged, changed, or active (changed within the debounce window). Only entries inside the roots count (-mindepth 1), so a root’s own mtime, which moves when the legacy stage is created, cannot read as a change. Known limitation: a pure deletion with no accompanying write is not detected until the next write in the same tree, because find -newer sees mtimes, not removals. Every ambiguous outcome is changed: a missing baseline, no roots, a non-zero exit, a timeout, or a thrown error.
Baseline. /tmp/.fouria-pending-backup is stamped before a snapshot captures state and promoted to the baseline only after the catalog commits. A failed snapshot leaves the baseline untouched. This runs on every snapshot, whether or not a flag is set, so enabling a flag later has a valid baseline.
A restore forgets the baseline. Restored files keep their archived (older) mtimes, so a -newer <baseline> scan would call the replaced tree unchanged and skip the first post-restore snapshot. Both successful restore paths (restoreIfNeeded, restoreBackupById) call resetChangeBaseline, which removes both markers so the next gate takes a snapshot unconditionally.
Archive size and restorable TTL
Two createBackup options in src/persistence.ts decide how large an archive is
and whether it stays restorable. Both matter more than they look: the archive is
uploaded by the container, re-read and re-encrypted by the Worker, downloaded
again on restore, and stored BACKUP_MAX_VERSIONS times, so every byte is paid
for maxVersions + 2 times.
SNAPSHOT_EXCLUDES— passed tomksquashfsas wildcard patterns (the SDK normalizes**away and warns). Excludes.npm,*.log,*.tmp,.cache, andworkspace/*node_modules*. It deliberately does not exclude pluginextensions/<id>/node_modules(a restore has no reinstall step of its own —start-openclaw.shonly installs when the extension directory is missing, so stripping the deps leaves a plugin that loads and then fails at import), SQLite databases, or session transcripts. Opt-in (BACKUP_ARCHIVE_EXCLUDES, defaultfalse). This is a fidelity trade, not a pure win: the archive is the recovery tier, and an archive restored without those paths no longer matches the filesystem a Tier 1 workspace snapshot restores (or the pre-eviction tree). The default keeps the archive a complete image; enable it per instance once you know the tenant regenerates those paths.SNAPSHOT_ARCHIVE_TTL_SECONDS(10 years) — the SDK enforces the TTL at restore time (doRestoreBackuprefuses an archive pastcreatedAt + ttlwithBackupExpiredError), so it is a restorability gate, not a storage-lifecycle hint. The previous 7-day value silently made a retained recovery point unrestorable —cleanupOldBackupsprunes by count, so an idle or frozen tenant could keepmaxVersionsarchives that no boot could apply, andisUnrecoverableArchiveErrorthen dropped the stable pointer and failed the boot closed. The durablerepairstep heals existing tenants:extendArchiveTtlrewritesmeta.jsonfor every archive below the minimum, preservingdir/createdAt/encrypted/encryptionVersion, and reports the ids asttlExtended/ttlFailedfromPOST /api/admin/storage/reconcile. The restore path heals its own archive (ensureArchiveRestorable, called right afterrestore-target.jsonis published in bothrestoreIfNeededandrestoreBackupById): therepairstep only runs inside a snapshot job, so a tenant that cannot boot could never reach the repair that would make it bootable — the wedge observed onstaging-client-dev, where all three catalogued archives (7-day TTL) expired on 2026-10-29 and every boot then failed in ~250 ms on the expiry check after its container was replaced. The SDK’s expiry message (Backup <id> has expired (created: …, TTL: …s)) is now also classified as an unrecoverable archive, so a metadata rewrite that cannot take effect drops the dead entry instead of retrying it forever.
Compression is still the SDK default (lz4, chosen for speed, not ratio); moving
to zstd trades container CPU for bytes and is tracked separately.
Watch (src/backup-watch/). watch.ts is the pure state machine. durable-object.ts is the adapter, and arm.ts holds gating and policy with no DO imports, so warm paths can call it without an import cycle.
- Never wakes a container: liveness is the non-waking
isRuntimeActive(). A not-running tenant is disarmed and not scanned. - Keepalive guard. Every scan is an
execthat renews the container’s sleep timer. A tenant unchanged forBACKUP_CHANGE_IDLE_DISARM_MS(10 min, below the staging 15 min sleep window) is disarmed, and re-arming never resets the idle clock. The watch can therefore delay a quiet container’s sleep by at most that window. - No starvation. An
activetenant defers snapshots for at mostBACKUP_CHANGE_MAX_DEFER_MS(10 min), so constant writes still get snapshotted. - No hot-path work. Snapshots are never taken from the watch directly. It enqueues the existing
ActivitySchedulerdurablesnapshotjob, which keeps that job’s gateway start, lease, retention, and restore-pending guards. - Gated before probing. Each tick first asks the scheduler for
getSnapshotGate()— one DO storage read, no container RPC — and re-arms without any probe or scan when a snapshot job owns the durable slot or the floor has not elapsed since the newest committed snapshot (whichever trigger produced it: dispatch cron, operator, or this watch). A gate read failure is fail-open (logged, then the normal tick). Previously a busy tick still ran a liveness probe plus afindscan every two minutes (~1,440 container operations per tenant per day) to discover there was nothing to do (durable-agents audit finding F7). A changed tenant inside the floor is no longer scanned, sodirtySincestarts at the first scan after the floor;maxDeferMsstill bounds starvation. - Arming happens on a running tenant only: the dispatch
backupcontrol action (/control) and the warmrefreshActivitySchedulepath (gateway healthy).
Wiring. wrangler.jsonc declares the BackupWatch class (migration v8) and the BACKUP_WATCH binding in every environment. In production the tenant’s bindings come from lerma, which binds BACKUP_WATCH only when the backup_change_watch provisioning flag is on (see apps/lerma). Both flags are plain-text vars that lerma always sets to true or false, so turning a flag off explicitly reverts it on redeploy.
Persistence policy telemetry (Phase 0)
Before changing the backup policy (see docs/plans/persistent-workspace-phase0.md)
the current behaviour is measured, not guessed. Three Analytics Engine events are
the input; they are observability only and never gate a snapshot, restore, or
stop:
| Event | Emitted from | double1 | double2 | double3 |
|---|---|---|---|---|
persistence-snapshot | finalizeSnapshot (success), ActivityScheduler (fail/skip) | durationMs | archiveDurationMs | bytes |
persistence-restore | restoreIfNeeded (recovery point), restoreBackupById | durationMs | bytes | — |
backup-change-window | BackupWatch tick (runBackupWatchTick observation) | dirtyWindowMs | scanMs | — |
bytes is the captured workspace size from the archive’s meta.json (one small
metadata read at archive), so “the artifact is large” and “the pipeline is slow”
are separable. trigger (scheduled / manual / deprovision) and the skip
reason (interval-not-elapsed, unchanged, not-running, backup-lock-busy)
are blobs, so a quiet series has a visible denominator. backup-change-window
records the continuous dirty time the watch accumulated (dirtySince), which is
the RPO distribution the checkpoint cadence will be chosen from.
Queries and the phase gates are in docs/plans/persistent-workspace-phase0.md.
Workspace snapshots on eviction (Phase 1, flag-gated)
WORKSPACE_SNAPSHOT=true makes TenantSandbox.onActivityExpired() capture a
platform filesystem snapshot immediately before eviction and commit the handle at
workspace/snapshot.json in BACKUP_BUCKET (one head handle; image-pinned;
30-day TTL). A new workspace-snapshot Analytics Engine row records
durationMs, bytes, trigger=evict and the outcome.
Capture is observability-grade in Phase 1: boot still uses the archive, so a
capture failure (or an unsupported runtime) never blocks the archive/stop path —
unsupported and failed are warnings, not errors.
The fallback must restore the same tree. Phase 2 may boot from the handle and
fall back to the archive, so the two tiers have to hold the same content. The
archive is a complete image today because SNAPSHOT_EXCLUDES is behind
BACKUP_ARCHIVE_EXCLUDES (off by default); an instance that opts into the
excludes makes an archive restore diverge from a snapshot restore by exactly
those paths. Do not enable the excludes on an instance whose snapshot boot must
be able to miss into the archive without regeneration.
Runtime prerequisite. Cloudflare only supports snapshots on a container
application with scheduling_policy: "durable_object". The tenant container is
on the default policy, and that cannot be changed in place: the policy is
immutable, a new Durable Object class/namespace is required, and the legacy
Container class is unsupported under it. Tier 1 therefore depends on the
Sandbox 1.0 workload adapter, tracked in
#380 (S2–S4), not on a config
change. SandboxV1 already runs on durable_object and implements the same
snapshotWorkspace() method, so the capability is real where the runtime allows
it. Keep the flag off until an instance is opted into v1; see
docs/plans/persistent-workspace-phase1.md.
Backup before deprovisioning
A container is never deprovisioned without a committed snapshot. The durable
backup-job queue (src/activity/backup-jobs.ts) gains a deprovision operation
(mode: 'sleep' | 'freeze'); the ActivityScheduler alarm runs it as
snapshot → stopIfIdle() (sleep) or snapshot → destroy() (freeze), and a
failed snapshot retries rather than tearing the container down (fail closed).
Reuse a snapshot that already covers the state. onActivityExpired() stops
inline instead of enqueueing a snapshot when the container has already been idle
for a whole window and an existing archive already captures the current
filesystem — a second archive is pure cost. Coverage is proved two ways, cheapest
first:
- Age.
hasRecentSnapshot()(a pure catalog read insrc/persistence.ts) answers whether the newest committed archive is newer than the instance’s idle timeout (resolveSleepAfterMs(): staging 15m, dev 2h, production 6h) plusIDLE_SNAPSHOT_REUSE_SLACK_MS(5m, so the archive finalized just before the previous stop attempt still counts). - No change since the last commit. With
BACKUP_CHANGE_DETECTIONon — the default — a quiet tenant takes no scheduled snapshot at all, so its newest archive legitimately ages past the idle window and the age test alone would re-archive it on every sleep cycle. When the gate is enabled, the hook runs the samescanTenantChanges()the scheduled path uses and treatsunchangedas coverage: the baseline is promoted only after a catalog commit, sounchangedmeans that archive already holds this state.changed/active, a missing baseline, or detection being off fall through to the archive.
A read failure is treated as “no recent snapshot” (fail closed). This also breaks
the loop where the deprovision snapshot’s own RPCs renewed the activity window,
stopIfIdle() refused, and the next expiry snapshotted again (observed on
tenant-tarjetero-dev: three 422 MB archives per idle expiry while the container
never stopped).
Deprovision archives stamp the schedule. finalizeSnapshot receives
stampScheduled: true for deprovision jobs as well as trigger: 'scheduled'
ones, so a committed deprovision archive resets lastScheduledBackup and the
next cron tick does not archive the same state again.
Stop only on inactivity (stop-time re-check). The snapshot can take minutes
on a large tenant, so the container may be used again between enqueue and stop.
The sleep branch therefore does not call sandbox.stop() directly: it calls
TenantSandbox.stopIfIdle(), an RPC on the Sandbox DO (the activity authority).
That re-reads the DO’s own idle window — renewed by every request, including a
proxy request for an in-flight chat — and the container process list, and stops
only when the window has genuinely elapsed and no transient work is running. If
the tenant became active during the snapshot the job returns skipped
(detail: active:<reason>) and leaves the container running; the next idle
expiry enqueues a new job. Refusals: activity (window renewed),
transient-activity, probe-failed (unknown is not idle),
billable-production (never auto-sleeps), not-running.
- Idle sleep.
TenantSandbox.onActivityExpired()stops inline when an existing snapshot covers the state (a recent archive, orunchangedfrom the change scan when detection is on), and otherwise enqueues adeprovision(sleep) job, so the Sandbox DO leaving memory never loses an in-flight multi-minute snapshot. With noACTIVITYbinding it falls back to the previous stop-now path. - Freeze.
POST /controlfreeze enqueues adeprovision(freeze) job and acknowledges immediately (202 { ok, accepted: true, frozen: false, jobId }), failing closed at enqueue (409when a durable job is already active). The caller does not hold a multi-minute request open: it polls the tenant’sGET /api/health/backup/:jobIduntil the job reaches a terminal status. A snapshot failure surfaces as afailedjob and the container stays running, so lerma never marks a still-running instance frozen. - Platform stops (an inactivity timeout firing after the DO left memory, a container crash, or main-process exit) have no Worker hook; the periodic + change-watch snapshots bound that residual RPO.
The pollable endpoint (GET /api/health/backup/:jobId) carries no in-Worker
authentication — the same posture as /api/health — and is reachable by the
lerma control plane’s health-monitor service token through the health Access
tunnel (the *.fouria.io/api/health path application in
apps/fouria/iac/access.tf). Cloudflare Access path inheritance covers the
/backup/* sub-path, and it is also listed explicitly so the tunnel allowlist
documents the endpoint.
Health & Status Endpoints
GET /api/health is the unified health endpoint: it returns the startup/readiness/liveness
probes plus the canonical status (running|starting|stopped), processId and processType.
It requires no authentication inside the Worker, but at the Cloudflare edge it is
Access-protected by path-scoped applications (*.fouria.io/api/health) that carry
no role requirement: they admit any authenticated user (browser) and the lerma
health-monitor service token (headless).
GET /api/status, GET /api/admin/gateway/status/polling and GET /api/admin/gateway/status/ws are thin
aliases over the same snapshot. Browser clients use the sibling paths outside /api/health/* —
GET /api/health-ws (WebSocket stream every ~2s), GET /api/health-snapshot (HTTP snapshot), and
GET /api/health-components (Cloudflare component states) — while /api/health, /api/health/ws,
and /api/health/components remain mounted for headless service-token consumers (lerma). The
browser-facing sibling paths exist because /api/health and /api/health/* are covered by the
path-scoped FourIA Health - <slug> Access app whose immutable AUD differs from the host app’s: the
browser’s host-app session is invalid there, so Access answers with a 302 login redirect to
*.cloudflareaccess.com — which WebSocket handshakes cannot follow and browser fetch() calls
reject on cross-origin CORS.
By default /api/health warms the container (lerma keep-alive). ?passive=1 probes without warming
— the ROOT /_admin/diagnostics surface uses it so an operator can debug an instance whose container is
down. The admin dashboard subscribes to the health WebSocket (with an HTTP polling fallback) and never
blocks the UI on gateway status.
GET /healthz is a Kubernetes-style liveness probe: one non-waking
isRuntimeActive() read, always 200 { ok, ts, status } (status = active |
idle), never warming the container or probing the gateway. It exists because
/healthz otherwise fell through to the gateway proxy (app.all('*')), which
warmed the container and waited for the port — ~108 s on a cold instance, so the
platform killed the request before it answered. Readiness stays on /api/health.
A single /api/health request issues one gateway /health curl exec, shared by the
readiness and liveness probes. They previously each ran their own, so every poll spawned a
pair of identical curl processes (visible in /debug/processes) — a duplicate on the
frequent lerma probe. The process scan and the /health probe are now both fetched once, in
parallel, and reused across the snapshot.
Gateway memory (gateway: { pid, rssBytes, threads })
The gateway is one Node process in a 4 GiB / no-swap container whose RSS is dominated by
per-worker_thread V8 heaps that an in-process restart never reclaims
(docs/plans/sandbox-memory-usage.md). A dev tenant “rebooted” once and the only way to answer
“was it memory?” was to SSH in and read /proc, dmesg and the gateway log by hand.
src/gateway/memory.ts makes that a first-class signal:
| Piece | Behaviour |
|---|---|
buildGatewayResourceScript() | One shell exec: reads the cgroup memory limit (mem:<bytes>; v2 memory.max, v1 fallback) and pgrep -x 'openclaw-gatewa' → gw:<pid>:<rssKb>:<threads> per live gateway server (same exact comm as the soft reload, so it never measures the openclaw gateway run launcher; zombies skipped). |
readGatewayMemory() | Runs it (gateway.memory-read, bounded) and returns { resources, containerLimitBytes }; readGatewayResources() is the resources-only view. Never throws: a failed read is nulls, not a broken health probe. |
resolveMemoryRestartBytes() | Derives the watchdog threshold: 75% of containerLimitBytes, or null below 3 GiB (too small for a healthy gateway; the gate is below the 4 GiB standard-1 class to tolerate runtime accounting). Unknown limits fall back to 3 GiB (the 4 GiB standard-1 value). |
sampleGatewayResources() | Throttles the read to GATEWAY_RESOURCE_SAMPLE_INTERVAL_MS (60s), so the frequent health poll issues at most one extra exec per interval and only a fresh sample writes a metrics row. A cached null is served inside the window too, so a broken probe is not retried every poll. |
maybeRestartGatewayForMemory() | The dev-gated watchdog (below). |
attachGatewayResources() in routes/public.ts attaches the sample to the snapshot as
gateway and writes a gateway-resources Analytics Engine row (double1 = RSS bytes,
double2 = threads, blob1 = gateway-resources, blob2 = customer, trailing
platform/environment blobs like every other cost row). It is applied by both /api/health
(and /api/health-snapshot) and the /api/health-ws stream, and only when the snapshot is
running — a sleeping/stopped container is never woken for a diagnostic.
Dev-gated memory watchdog. When the sample reaches the instance-aware threshold —
resolveMemoryRestartBytes(): 75% of the container’s cgroup memory limit (3 GiB on a 4 GiB
standard-1, 4.5 GiB on a 6 GiB standard-2, 6 GiB on an 8 GiB standard-3, 9 GiB on a 12 GiB
standard-4) — on a dev instance (isDevAgentsEnabled: -dev hostname,
OPENCLAW_DEV_MODE, or local DEV_MODE), the gateway is replaced — killGateway +
resetStartupCache + ensureGateway({ skipRestore: true, bypassMaintenance: true }). That is the
only mechanism that actually returns RSS to baseline.
The limit is read from the container on every sample (one extra local read in the same exec), so an
instance resized by an instance-type rollout picks up the new threshold without a Worker
deploy. Instances smaller than 3 GiB (i.e. basic/lite) are skipped (instance-too-small): a
restart cannot make an undersized box viable, and a 3 GiB fixed threshold on an 8 GiB
standard-3 would be only 37% of the limit and restart a healthy gateway.
Design constraints worth keeping:
- The threshold guards the container limit, not OpenClaw’s warning. OpenClaw’s own
rss_thresholdis 1.5 GiB and fires permanently on a healthy-but-large dev gateway (~2.1 GiB observed, ~2.6 GiB peak). Using it as the action threshold would restart the gateway in a loop. - Scale with the instance, never hardcode. A fixed byte threshold means a different headroom on every instance type; the ratio keeps ~25% of the limit free for the kernel and the next allocation burst.
- No soft-reload step. A measured in-process
SIGUSR2restart does not reclaim RSS (1.14 → 1.38 → 1.18 GiB), so reloading first would only add a failed attempt. skipRestore. A kill leaves the filesystem untouched; re-entering the R2 restore barrier would deadlock on the backup lock for no benefit.- Cooldown claimed before acting.
GATEWAY_MEMORY_WATCHDOG_COOLDOWN_MS(30 min) is recorded before the replacement, so two concurrent health polls cannot start two restarts. - Detached. The watchdog is dispatched through
executionCtx.waitUntil(or left detached when there is no context), so a multi-minute replacement never delays or fails a health response.
Every decision returns a stable machine reason (not-dev, no-resources, instance-too-small,
below-limit, cooldown, rss-above-limit), and an actual restart logs a structured
gateway.memory.pressure event with pid, rssBytes, rssMib, threads, the resolved
limitBytes/limitMib, and the measured containerLimitBytes/containerLimitMib.
Two-stage policy and continuous sampling (2026-11-03). The watchdog above only runs when
something calls attachGatewayResources, and nothing polls /api/health while the gateway is
healthy — the Control UI talks to the gateway directly. During the tenant-tarjetero-dev outage
that blind spot was total: gateway-resources had no rows for ~17 h, so the crash had no memory
evidence at all.
checkGatewayMemoryHealth()(gateway/memory.ts) is now the single sampler used by both/api/health*and the durableActivityScheduler.alarm()(runMemoryWatch()), so a tenant is observed whether or not a browser is polling. The throttle and thegateway-resourcesrow are unchanged.- Every fresh sample also writes
gateway-memory-sample(blob1,blob2= customer,blob3=running | absent,double1= RSS,double2= threads,double3= container limit). The row exists even when no gateway process is running, so a gap in the series means “nobody sampled” instead of “nothing happened”. decideMemoryAction()is the pure policy behind both callers: ≥ 60 % of the cgroup limit (resolveMemoryCheckpointBytes,GATEWAY_MEMORY_CHECKPOINT_RATIO) →checkpoint; ≥ 75 % (resolveMemoryRestartBytes) →restart, escalating tocheckpoint-and-restartwhen the newest archive is older thanGATEWAY_MEMORY_LOSS_BUDGET_MS(15 min).maybeRestartGatewayForMemorydelegates to the same function (mapping its reasons back to the vocabulary above), so the thresholds cannot drift between the request path and the durable alarm.- The durable alarm performs the checkpoint by enqueuing a snapshot job with
trigger: 'memory-pressure'before restarting, so the work is on disk regardless of what the kernel does next. The checkpoint has its own cooldown (EMERGENCY_SNAPSHOT_COOLDOWN_MS, 10 min) and is surfaced as agateway.memory.policyevent. - The request path may still only restart: it cannot own a durable job, and blocking a health response on a 450 MB snapshot is worse than restarting.
healthWsHandlernow passes itsexecutionCtxtoattachGatewayResources. Without it the detached watchdog promise was cancelled as soon as the frame was sent, so the watchdog was a no-op on the one path a browser actually keeps open.- Sampling never wakes the container, and it yields to every durable job.
sandbox.execauto-starts a stopped container, so an unbounded/procread from the alarm racedcontainer.start()during a local restore (gateway.memory-read failed (launch-failed)/Container failed to start, and a workerdjsgInternalErrorthat aborted the start).checkGatewayMemoryHealthnow takes the non-wakingisRuntimeActive()read first (bounded byPASSIVE_PROBE_TIMEOUT_MS, like the passive health probe) and returns{ action: 'skipped', reason: 'runtime-inactive' }with afresh: falsesample when the runtime is off — so a sleeping tenant is never billed for a diagnostic and never woken by an alarm tick.runMemoryWatchadditionally returns before sampling while a boot, backup, or maintenance job is active (queued/running/retrying): those own the container, and the watchdog is observability, never a participant in a boot.
Design and remaining work: docs/plans/graceful-shutdown-and-emergency-checkpoints.md.
A destructive restore is never triggered by an unreadable container. probeContainerState()
returns present | absent | unknown, and restoreIfNeeded only restores on absent. On unknown
it throws the retryable ContainerStateUnknownError and writes no restore-needed latch, so
the durable boot job retries instead of replacing the filesystem of a container it cannot read.
This is what turned the 2026-11-03 wedged container (dead control channel) into a destructive
450 MB restore that timed out and pinned every later boot behind
RestoreInFlightError (restore-outcome-unknown).
Cloudflare component states
The dedicated GET /api/health/components endpoint returns a quick view of the
Worker-bound platform pieces rendered by the ROOT diagnostics horizontal
Cloudflare Components widget. GET /api/health also carries the same
(redacted) components entry — but only on the non-passive path. Lerma’s
HealthMonitor polls /api/health?passive=1 frequently and never reads
components, so the passive probe skips the component probe entirely (no
isRuntimeActive read, no R2 HEAD) to keep the frequent poll slim:
| Component | Probe |
|---|---|
| Container | Non-waking sandbox.isRuntimeActive() (up / sleeping / unknown) |
| Durable Object | Sandbox + ACTIVITY bindings present and the control channel answered |
| R2 | Bindings-shaped BACKUP_BUCKET plus a HEAD of backup-handle.json |
| AI Gateway | Present only when all three gateway credentials are configured (up = configured) |
Every probe is non-waking and individually bounded (3s), so the endpoint is
safe on a sleeping instance and can never hang the health response; on failure
the components entry is omitted instead of blocking. The AI Gateway is
inclusion-gated on credentials and reported as up (“configured”) — the probe
deliberately does not issue an authenticated Cloudflare API call on every poll,
and a missing account-scoped token must not read as a false outage.
Each component also carries binding-derived facts (a { label, value }[]
shown in the widget’s hover panel) and a Cloudflare dashboard logsUrl (a
per-product logs/metrics deep link opened on click). Both are derived from
Worker bindings/env only — never an authenticated Cloudflare API call — so
the surface stays non-waking and free (src/gateway/cloudflare-links.ts builds
the links: Container → Workers Logs/observability, Durable Object → DO
metrics/logs, R2 → the BACKUP_BUCKET_NAME bucket page, AI Gateway → the
gateway page). When the account id is not bound the links fall back to the
dashboard’s documented ?to=/:account/… form; a component without a resolvable
link renders as a plain (non-clickable) card. The unified GET /api/health
snapshot redacts facts/logsUrl (redactComponentDetails) — it is a
broadly-consumed public probe — and serves only the state summary; the enriched
view is exclusive to GET /api/health/components.
On phones the widget collapses into a compact status widget that shares a
row with the gateway and persistence-lock status widgets at the top of the
diagnostics page (not floated or overlaid): a single pill carrying the Cloudflare
icon, a worst-case rollup status dot and an up/total summary. Tapping it
expands the same component strip as a compact panel. To build that parallel row,
AdminLayout omits the topbar gateway + persistence indicators on the
diagnostics page on phones (App.tsx) and DiagnosticDashboard renders
GatewayStatusLed + PersistenceLockIndicator beside the Cloudflare widget. The
gateway pill gains an identifying icon and the lock an
Idle/Backing up/Restoring label (its color still encodes the state); both
use compact phone sizing so the three widgets stay on one line on normal phones
and wrap gracefully on very narrow ones. The desktop layout is unchanged — the
topbar keeps its indicators and the Cloudflare card renders full-width.
The ROOT /_admin/diagnostics page renders its operational tools as separate
clickable Fouria DevOps widget cards (backup connectivity + configuration
editor + paired devices). Each card’s title row toggles its body while the
action buttons stay live, and the cards load their data asynchronously and
render their chrome immediately — there is no blocking loading animation for the
data that populates them. On phones each option collapses into a vertical
icon + name menu row (collapsed by default; tap to open) with the option’s
actions revealed inside the opened body. The Configuration Editor reflows to the
viewport: action buttons stack full-width, the section tabs scroll horizontally,
long JSON wraps, and the raw textarea shrinks. The Debug Tools grid drops to a
single column so nothing forces sideways scrolling, and the admin bottom nav
uses flexible items (with safe-area insets) so every nav entry fits. The
floating dev role switcher (DevUserSelectorCompact) is lifted above the bottom
nav on phones so it neither covers the last nav item nor overflows the viewport.
The gateway status indicator lives in the admin topbar beside the
persistence-lock indicator and the notification bell (GatewayStatusLed, mounted
in App.tsx’s AdminLayout). Its color-coded LED reflects the gateway status
(running/starting/sleeping/stopped/unknown); clicking it reveals the
complete status value (status, process, process type, the startup/readiness/
liveness probes, restore state and last update) and a reboot icon that
performs the same soft-reload/restart as the removed Gateway Controls banner. The
reboot action is hidden behind an Admin badge when /debug/* is disabled,
matching the previous gating.
Live gateway detection
On the Worker-driven cold start the gateway is a supervised sandbox.exec
process (/bin/bash -lc 'exec openclaw gateway run …'), so it is visible to
listProcesses() for the life of the container. The legacy full mode (manual
runs, test:gateway:boot) still launches it as a detached background child of
start-openclaw.sh (… &), and a container provisioned by an older image may
still be serving a detached gateway during a rolling deploy — for those, after
the script exits the gateway binary is invisible to listProcesses().
To avoid false no_process reports on log/status surfaces, those endpoints
(GET /api/admin/logs, GET /debug/logs, GET /debug/stop-gateway,
POST /api/admin/gateway/restart, GET /api/admin/audit-logs) call
findExistingGatewayProcess(sandbox, { probeLive: true }), which falls back to
a bounded /health probe and — when the gateway responds — returns a synthetic
gateway-live process whose logs are read from /tmp/gateway-stdout.log. The
passive health snapshot path deliberately does not probe so a down container
fails fast.
Log Levels
| Level | When to use |
|---|---|
debug | Local development |
info | Production default |
warn | High-traffic production |
error | Security-sensitive production |
TypeScript & Tooling
Type Generation
Worker types are auto-generated:
bun run types # wrangler types → worker-configuration.d.ts
Never manually edit worker-configuration.d.ts — it’s regenerated from wrangler.jsonc.
Testing
bun run test # Unit tests (Vitest)
bun run test:coverage # With coverage report
bun run test:client # SPA component/page tests (jsdom)
bun run test:ui # Playwright UI suite (Vite dev server + mocked API)
bun run test:ui:headed # Same, with a visible browser
bun run test:e2e # E2E tests (cctr + Playwright)
bun run test:e2e:cli # CLI-only E2E (cctr)
The Playwright UI suite (test/ui/, config playwright.config.ts) drives the
real SPA served by bun run dev with per-test /api/* mocks — no Worker,
container, or Cloudflare credentials. It covers the app shell, role-based
navigation and redirects, every admin page’s render, and the notification bell.
See docs/TESTING.md for the
full description and the Nix PLAYWRIGHT_BROWSERS_PATH notes.
Formatting & Linting
bun run format # oxfmt (auto-format)
bun run format:check # Check formatting
bun run lint # oxlint
bun run typecheck # tsc --noEmit
Common Tasks
Adding a New API Endpoint
- Create route handler in
src/routes/api/<domain>.ts - Register in
src/routes/api.ts - Add client method in
src/client/api.ts - Write tests (unit + integration)
- If new dependencies: update
package.json
Adding a New Environment Variable
- Add to
OpenClawEnvinterface insrc/types.ts - If passed to container: add to
buildEnvVars()insrc/gateway/env.ts - Update
.dev.vars.example - Document in ../../docs/ENV-VARIABLES.md