F FourIA latest 2026.10.29 GitHub ↗

Fouria Operator Guide

On this page

1. Welcome

Fouria is a Cloudflare Worker that runs OpenClaw (the AI agent gateway) inside a Cloudflare Sandbox container. Each customer gets an isolated container; the Worker starts it, proxies chat traffic to it, snapshots its state to R2, and keeps it healthy.

Most of the time the system runs itself. This guide is for the moments when it does not: when a container is stuck, a backup will not restore, the AI gateway rejects requests, or the gateway will not start.

The three surfaces you operate

SurfaceEntry pointWhen to use
Health snapshotGET /api/health (and ?passive=1)Is the gateway up? What is its status?
Debug routes/debug/* (only when DEBUG_ROUTES=true)Deep inspection + emergency actions (stop, destroy, rescue, restart)
Diagnostics modulesrc/diagnostics.tsThe code that powers container rescue; gateway restart lives in src/gateway/process.ts + src/routes/debug.ts

Mental model: the lifecycle

Request / cron / health probe
        │
        ▼
 ensureGateway()  ── finds or starts the container
        │
        ▼
 start-openclaw.sh  (onboard → moltlazy patch → plugins → gateway run)
        │
        ▼
 OpenClaw gateway  (port 18789, Control UI + WebSocket)
        │
        ▼
 createSnapshot() → R2  (state preserved between restarts)

When any link in this chain breaks, the tools in this guide are your recovery path.


2. Getting Started

2.1 Prerequisites

You should be comfortable with:

  • Reading a JSON health response (curl … | jq).
  • Cloudflare concepts: Workers, Containers, R2, AI Gateway.
  • That the gateway runs inside a container — most “server is down” issues are really “container is down or the gateway process inside it is down”.

2.2 Enabling diagnostics — how to turn on debug options

Debug tooling is off by default in production for safety. There are two layers:

Layer A — DEBUG_ROUTES (the /debug/* HTTP surface)

The /debug/* routes are gated by the DEBUG_ROUTES environment variable (apps/fouria/src/index.ts:379, src/types.ts:34).

# In .dev.vars (local) or as a Worker var (production)
DEBUG_ROUTES=true
  • DEBUG_ROUTES !== 'true' → every /debug/* call returns 404 immediately (index.ts:292).
  • Setting it is non-destructive: it only exposes the inspection/action endpoints; it does not change runtime behavior.

⚠️ Never leave DEBUG_ROUTES=true on a tenant-facing deployment. It exposes /debug/gateway-token, /debug/stop-gateway, and /debug/destroy-container. Use it on operator/ROOT deployments only.

Layer B — passive health probes (no extra config)

The public health endpoint GET /api/health is always on. Append ?passive=1 to inspect a down instance without warming/cold-starting it (routes/public.ts:163):

curl "https://<worker>/api/health?passive=1" | jq .

Passive probes use a short 3-second bound (PASSIVE_PROBE_TIMEOUT_MS, config.ts:24) and never trigger ensureGateway, so a dead container fails fast instead of spinning up for minutes.

2.3 Your first diagnostic call

# 1. Is the gateway healthy? (fast, no warm-up)
curl "https://<worker>/api/health?passive=1" | jq '.status, .degraded, .containerError'

# 2. Full active snapshot (warms the container, schedules a backup)
curl "https://<worker>/api/health" | jq .

# 3. Deep dive (requires DEBUG_ROUTES=true)
curl "https://<worker>/debug/version" | jq .
curl "https://<worker>/debug/processes?logs=true" | jq .

Read the status field — it is one of running | starting | stopped (health-probes.ts:67). Read degraded — when true, a permanent container error (e.g. deleted image) requires a rescue (see §6.1).

2.4 Manual snapshots

POST /api/admin/storage/sync queues a durable snapshot by default when the ACTIVITY Durable Object is configured. It returns 202 with a jobId (or 409 when a backup/restore job already owns the slot). Use ?inline=1 only when the caller needs the completed snapshot response immediately, such as an E2E checkpoint; ?async=1 retains the accepted-then-background behavior. Without ACTIVITY, the endpoint falls back to inline work.

The shared R2 lease is valid for 12 minutes (STEP_DEADLINE_MS), renewed every 3 minutes and at slow snapshot/restore phase boundaries. A dead lease can consequently take longer to recover than under the former 60-second TTL; this favors preventing false expiry while a durable archive step is active. Watch for structured persistence.lock.renewal_late and persistence.lock.renewal_failed events.


3. Operation Guidelines — Debug Options & How to Activate Them

3.1 The debug endpoint catalog

All under /debug/*, active only when DEBUG_ROUTES=true. Each maps to logic in src/diagnostics.ts or gateway/process.ts.

Method & PathWhat it doesBacked by
GET /debug/versionFourIA Worker version + git commit, plus OpenClaw + Node versions in the container__FOURIA_VERSION__ / __FOURIA_COMMIT__ + runShell (sandbox.exec)
GET /debug/processes?logs=trueList container processes (+ logs); returns 503 { status: "platform_updating" } while the sandbox runtime is being updated (deploy)sandbox.listProcesses
GET /debug/gateway-apiProbe the OpenClaw HTTP APIGATEWAY_API_TIMEOUT_MS
GET /debug/cliRun an OpenClaw CLI commandstartShell (sandbox.exec)
GET /debug/logsContainer stdout logssandbox
GET /debug/envSanitized environment (secrets masked)maskEnvValues()
GET /debug/container-configOpenClaw config from inside containersandbox.readFile
GET /debug/gateway-logsGateway stdout (/tmp/gateway-stdout.log)sandbox.exec
POST /debug/sqlite-integrityRead-only PRAGMA integrity_check over every *.sqlite under /root/openclaw/.openclaw (e.g. state/openclaw.sqlite after a restore). Opens each file readOnly; never checkpoints, repairs, or removes WAL/SHM sidecars. Returns ok, per-file messages, and sidecars. Results are a snapshot while the gateway is writing — stop the gateway first for an authoritative check. Runs as a durable maintenance job (ActivityScheduler DO) because a large check outlives a Worker request; returns 202 { accepted, jobId } and is polled via GET /debug/sqlite-integrity/jobs/:id or streamed over GET /debug/sqlite-integrity/jobs/:id/ws (runs inline when the ACTIVITY binding is absent)sandbox.exec (node:sqlite, readOnly)
GET /debug/gateway-tokenThe gateway auth tokenresolveSecret
GET /debug/gateway-lockOpenClaw native lock stategateway/lock.ts
POST /debug/stop-gatewayKill gateway without restartingkillGateway()
POST /debug/restart-openclawHard gateway restart (kill → clear startup cache → flag R2 restore → start-openclaw.sh reprovision)killGateway() + resetStartupCache() + signalRestoreNeeded()
POST /debug/reprovisionForce a container reprovision (checkpoint → 409 unless restorable backup or force → destroy → clear startup/persistence caches → reset orphan streak); never touches the gatewaycreateSnapshot() + getBackupHistory() + sandbox.destroy()
POST /debug/destroy-containerDestroy the container entirelysandbox.destroy()
POST /debug/sleepForce the idle-sleep deprovision path (E2E seam): enqueue the durable backup-then-stop job the idle hook enqueues, so a container is never slept without a committed snapshot. 202 {jobId}getActivityScheduler() → enqueueBackupJob({operation:'deprovision',mode:'sleep'})
POST /debug/rescue-containerDiagnose + rescue a stuck containerrescueContainer()
POST /debug/backup/:modeTrigger a checkpoint on demand (mode = sync | async): sync blocks and returns the result (502 on failure); async acknowledges 202 and runs the snapshot via waitUntil. Idle container is skipped, never wokenensureGateway() + scheduleBackupIfNeeded()
POST /debug/r2-put?key=Write raw content to R2 (tests)BACKUP_BUCKET.put

Secret safety: GET /debug/env runs every value through maskEnvValues() (diagnostics.ts:31). Any key matching token, api_key, password, secret, credential, auth, or private_key is replaced with ••••••••. Never paste raw /debug/env output into tickets.

3.2 Reading the gateway lock (the most common “ghost” outage)

OpenClaw enforces single-instance ownership via three lock layers — a state lock, a config lock, and an exclusive TCP bind on port 18789 (gateway/lock.ts:1, docs.openclaw.ai/gateway/gateway-lock). A stale lock is the #1 cause of "gateway already running (pid …)" / EADDRINUSE on restart.

curl "https://<worker>/debug/gateway-lock" | jq '.state'

state is one of (lock.ts:27):

StateMeaningOperator action
freeNo lock held — safe to startNone
held-liveA live gateway owns the portAdopt it; do not kill
stale-reclaimableLock files present but owner PID goneAuto-reclaimed on next start; restart is safe
conflictLock error seen in the logWait/restart; avoid parallel starts
unknownProbe failed (container off)Check container health first

The hard restart path checkpoints state before killing the gateway, then waits for the next activation to restore that committed snapshot. OpenClaw’s native lock protects gateway ownership, but does not serialize filesystem mutations performed by onboarding, plugins, configuration, or backup operations.

3.3 Emergency actions — when to use each

ActionUse whenRisk
POST /debug/stop-gatewayGateway wedged but config is fine; you want to manually start freshLow — does not destroy data
POST /debug/restart-openclawConfig valid but gateway process dead; you want a clean in-place restartLow
POST /debug/reprovisionContainer lost/orphaned (e.g. destroyed from the dashboard) but the DO is alive; gateway RPCs are timing outLow–Medium — checkpoints first, 409 if no restorable backup (override force); R2 restore keeps state
POST /debug/destroy-containerYou want the platform to provision a brand-new container (fresh state)High — loses in-memory + unsnapshotted state
POST /debug/rescue-containerContainer is in Error mode / image deleted (see §6.1)Medium — destroys stuck instance, keeps R2 backups

3.4 The two diagnostic operations you will script most

Both live in src/diagnostics.ts and accept a dryRun option — always run dry first.

POST /debug/restart-openclaw — checkpointed hard gateway restart

Exposed as the diagnostic dashboard button (ROOT, DEBUG_ROUTES=true). It is the hard restart path (the opposite of the Admin API’s soft reload):

  1. Find the live gateway process via findExistingGatewayProcess(sandbox, { probeLive: true }).
  2. Refuse with 409 if the tenant is busy: a restore is in flight (or its outcome is still unknown after a timeout), or a maintenance window is open. Nothing is stopped.
  3. Under the R2 lease, open a restart maintenance window and killGateway(sandbox) — SIGTERM with GATEWAY_KILL_GRACE_MS (5s) grace, SIGKILL only if still active; uses the [o]penclaw pkill bracket trick (no blanket "openclaw" pkill, no manual lock-file deletion — respects OpenClaw’s own single-instance lock). The gateway is stopped before the archive is cut, so the checkpoint is consistent.
  4. Create and commit a snapshot of /home/openclaw. If a restore is pending over a committed recovery point, the checkpoint is refused (409) before the gateway is stopped.
  5. resetStartupCache() — clears the startup health cache.
  6. signalRestoreNeeded(BACKUP_BUCKET) — flags R2 restore so the next request reprovisions a fresh gateway via start-openclaw.sh. The maintenance window closes only after this marker is set, so no request can relaunch the gateway on pre-restart state in between.

If the checkpoint fails before the gateway is stopped, the gateway is not touched. If it fails after the stop, the gateway is restarted in place on the untouched filesystem and no restore is armed. There is no direct binary launch; reprovisioning always goes through start-openclaw.sh.

The Admin API POST /api/admin/gateway/restart is the soft reload: it sends SIGUSR2 to the live openclaw gateway run process (softReloadGateway), then resetStartupCache(). When no live gateway is found it starts the gateway through the normal cold start (ensureGateway) and does not arm restore-needed: the live filesystem is kept, and the restore barrier still restores a container that has no state. It returns 409 while a maintenance window is open.

SIGUSR1 is not a restart signal: in current OpenClaw it starts Node’s inspector on 127.0.0.1:9229 and leaves the gateway running. The signal must reach the gateway server, not the openclaw gateway run launcher: the launcher merely supervises the server, so SIGUSR2 to the launcher kills it and orphans the server (PPID 1) without restarting anything, while the endpoint still answers reloaded: true. buildSoftReloadScript therefore matches the server’s exact comm (openclaw-gatewa, Linux’s 15-char truncation). Inside the container OpenClaw always takes the in-process restart path — isContainerEnvironment() reports disabled / unmanaged even without OPENCLAW_NO_RESPAWN=1 — so soft reload never spawns an orphan either way. See ../../docs/plans/sandbox-memory-usage.md for the #463 decision record.

POST /debug/reprovision — force a container reprovision

The escape hatch for an orphaned/dead container whose Sandbox Durable Object is still alive — most commonly after the instance was destroyed from the Cloudflare Containers dashboard, which leaves the DO pointing at a placement that no longer exists.

  1. Busy guard — refused with 409 while a restore is in flight (or its outcome is still unknown) or a maintenance window is open. Unlike force, this cannot be overridden: destroying the container mid-apply leaves a half-replaced filesystem. Re-checked after the checkpoint, in case a restore started while it ran.
  2. Checkpoint first (createSnapshot) — only when isRuntimeActive() reports a live runtime (an idle container is never woken just to checkpoint it), and bounded by REPROVISION_CHECKPOINT_TIMEOUT_MS so an orphaned container’s hanging RPC cannot block the route. If the container is gone or the snapshot times out, the route falls back to an existing restorable backup (verified via HEAD on the newest catalog entries’ meta.json, not just the entry count). A snapshot abandoned by the timeout can finish after the destroy, but it only archives state that already existed, so it cannot corrupt anything.
  3. Fail closed: if neither the checkpoint nor an existing backup is restorable, the route returns 409 and does not destroy the container. Destroying it would permanently lose tenant state.
  4. clearPersistenceCache() / resetStartupCache() — drop the per-isolate “already restored / startup ok” facts about the old container.
  5. clearProvisioningFailures(sandbox) — reset the orphaned-instance failure streak.
  6. sandbox.destroy() — tear down the instance so the next RPC provisions a fresh one.

Unlike restart-openclaw, this never talks to the gateway, so it still works when the container is completely gone and gateway RPCs are timing out.

curl -X POST "https://<worker>/debug/reprovision" | jq .

When the container is already gone and no backup exists (state is unrecoverable), pass the explicit override:

curl -X POST "https://<worker>/debug/reprovision" \
  -H 'content-type: application/json' -d '{"force":true}' | jq .

rescueContainer(sandbox, env, options) — diagnostics.ts:296

Recovers a container stuck in Error mode (e.g. the fouria:unstable-<sha> image it references was deleted by the stale-image cleanup cron). Sequence:

  1. Probe container health (probeContainerHealth, short PASSIVE_PROBE_TIMEOUT_MS bound). Healthy → { action: 'none', ok: true }.
  2. Repair image via the Cloudflare Containers API (createContainersApiClient → rolloutImage with strategy: 'full_auto') if credentials are present and the current image ≠ expected (diagnostics.ts:339).
  3. Destroy the stuck container (sandbox.destroy()) so the platform provisions a fresh instance against the repaired image.
# Diagnose first (dry run — does nothing)
curl -X POST "https://<worker>/debug/rescue-container" \
  -H 'content-type: application/json' -d '{"dryRun":true}' | jq .

# Actually rescue (requires CLOUDFLARE_API_TOKEN + CONTAINER_APPLICATION_ID)
curl -X POST "https://<worker>/debug/rescue-container" \
  -H 'content-type: application/json' -d '{"dryRun":false}' | jq .

The result includes action ∈ none | destroy | repair-image | repair-image-and-destroy | error, a diagnosis (health + errorClass), and the image report (current, expected, repaired, rolloutId).

Error classification (what “permanent” vs “transient” means)

classifyContainerError() (diagnostics.ts:166) decides whether retrying helps:

  • Permanent (no point retrying): no such image, no matching app, no namespace configured, did not call start, out of memory, resource exhaustion, pid limit. → a rescue (image repair + destroy) is required.
  • Transient (recovers on its own): connection refused, econnrefused, etimedout, timed out, not mapped, no container instance available. → just wait / warm up.
  • unknown → treat as transient unless health stays red.

This classification drives the degraded flag on the health snapshot (health-probes.ts:281): a permanent error is what flips degraded:true and tells you a rescue is needed.


4. Working with Backups

State lives in the container at /home/openclaw and is preserved as squashfs snapshots in R2 (persistence.ts). The gateway restarts from the latest snapshot on boot.

4.1 The persistence model

ConceptWhereNotes
Backup store handlebackup-handle.json in R2Tracks up to maxVersions (default 3)
Backup objectsbackups/<id>/data.sqsh[.enc], meta.json.enc when encrypted at rest
Config versionsconfigs/<timestamp>.jsonKept 30 days (CONFIG_RETENTION_DAYS)
Restore markerrestore-needed keyDurable restore-in-progress/retry latch; absent while idle or after success
Lockbackup-lock (60s TTL, 20s heartbeat)Shared cross-isolate lease: serializes manual + scheduled snapshots

4.2 Encryption at rest (F4E1)

When BACKUP_ENCRYPTION_KEY is set, every snapshot is post-encrypted with AES-256-GCM before it lands in R2 (persistence.ts:405, createSnapshot). Config versions under configs/ are encrypted with customMetadata.encrypted='true'.

# Generate a key (store it as a secret_text binding, NEVER in .dev.vars for prod)
openssl rand -base64 32
  • BACKUP_ENCRYPTION_KEY unset → backups are stored UNENCRYPTED (warned in logs).
  • BACKUP_ENCRYPTION_REQUIRED=true → backups fail closed if the key is missing (persistence.ts:419). Recommended in production.

Legacy plaintext backups remain restorable; decryption is transient and deleted in a finally block (persistence.ts:223). See docs/plans/r2-encryption-at-rest.md.

4.3 How backups happen

  • Dispatch cron (running instances only) — the Dispatch Worker’s cron (every 15 min production / hourly staging) fans a backup control action out to every non-frozen tenant. The tenant asserts isRuntimeActive() (a non-waking bridge read) and skips a sleeping container — an idle instance has no workload and no filesystem changes to capture, and waking it would restart billing. This is the backstop for tenants the lerma HealthMonitor does not poll. Before snapshotting, the persistence layer refuses to run while a restore-needed marker is present, rechecked under the shared lease, so a wake that lands mid-restore cannot snapshot and promote a partial filesystem over the committed recovery point.
  • On hard restart — POST /debug/restart-openclaw first creates a checkpoint, then clears the startup cache and flags R2 restore.
  • Manual — POST /api/admin/storage/sync calls createSnapshot() (explicit admin action).
  • Async admin — POST /api/admin/storage/sync?async=1 and POST /api/admin/storage/restore/:id?async=1 commit a durable job to the ActivityScheduler Durable Object and return 202 { accepted: true, jobId }. A DO alarm runs the job and retries it with a bounded lease, so the workflow survives the request ending, isolate eviction, and Worker redeploys (the old waitUntil() path was only guaranteed ~30s after the response). Poll GET /api/admin/storage/jobs/:id for queued | running | retrying | succeeded | failed | skipped.
  • Health probes are observability-only: GET /api/health NEVER writes the backup catalog.
  • POST /debug/r2-put only writes test objects; it does not create a snapshot.

Stable recovery point. A snapshot becomes the recovery point only once its archive, optional F4E1 encryption, and catalog entry are all committed (stable-backup.json). A cold restore prefers that pointer and refuses to restore a newer archive whose promotion was interrupted, and retention always keeps it — so a snapshot that failed partway can never evict the last known-good recovery point. GET /api/admin/storage/backups reports it as stableBackupId. This is a checkpoint guarantee, not zero-RPO replication: writes acknowledged after the last committed snapshot are outside it (see issue #377). The pointer only advances forward in time by default, so a stale snapshot can never demote a newer committed one; a targeted restore is the one exception (see §4.4).

Two guards keep the pointer safe in the other direction:

  • Restore pending. While a restore-needed marker is present and a committed recovery point exists, a snapshot is still archived but does not move the pointer, and a pre-replacement checkpoint is refused (409). The live filesystem is not authoritative until the pending restore has applied.
  • Restore-only hold. A targeted restore with setCurrent=0 (§4.4) writes a recovery-hold.json marker. Scheduled snapshots keep archiving but do not move the pointer onto the restored state, so the restore is not silently undone within one backup interval. Any explicit snapshot (manual, checkpoint, hard restart) or a restore that sets the recovery point clears the hold. GET /api/admin/storage/backups reports it as recoveryHeld.

All three snapshot paths share one tenant-scoped R2 backup-lock lease, so only one isolate ever stages the fixed legacy stage directory (.fouria-legacy-stage) or calls sandbox.createBackup at a time. The lease has a 60s TTL and is renewed every 20s while a snapshot is in flight, and it is released by compare-and-swap to a tombstone (never by deleting the lock object, which would race a concurrent takeover). A live lease left behind when an isolate is killed is reclaimed after the 60s TTL; the tombstone ({acquired:0}) is the normal released state, not a stuck lock.

Crash-consistency (issue #344). A Worker isolate can be killed by its CPU budget after sandbox.createBackup() has uploaded backups/<id>/… but before the handle is written — leaving an orphaned, apparently-unrecoverable snapshot and a missing backup-handle.json. Two guards prevent/repair this:

  1. The catalog entry is committed immediately after the archive is created (after encryption, before the slower config-version capture and retention prune), so an interrupted run still leaves a catalogued, restorable archive.
  2. The catalog is reconciled explicitly from backups/*/meta.json by reconcileBackupCatalog (the durable repair step, POST /api/admin/storage/reconcile, restoreIfNeeded, scheduleBackupIfNeeded, and the reprovision guard) whenever an archive is missing from the handle, so pre-existing orphans self-heal. getBackupHistory stays a pure read: a dashboard poll never lists, prunes, sweeps, or writes. Retention is enforced on that reconcile path too, so an orphan adopted after a run of failed snapshots cannot push the catalog past maxVersions.

The scheduled backup control action additionally runs under a dedicated CPU budget (the Dispatch Worker’s BACKUP_CONTROL_CPU_MS) so the full orchestration can finish instead of being killed mid-flight. Freeze now enqueues a durable backup-then-destroy job and acknowledges immediately (202 { jobId }) under the default budget; the caller polls GET /api/health/backup/:jobId (through the health Access tunnel) until the job reaches a terminal status instead of holding the request open. Unfreeze keeps the default.

Legacy-stage timeout (issue #344 follow-up). The pre-snapshot legacy-state staging shell can exceed its bound on a large tenant (observed killing the tarjetero stage at exactly 300s). The bound is now LEGACY_STAGE_TIMEOUT_MS (default 600000, i.e. 10 min), the idempotent stage is retried once on a transient timeout/transport drop, and a stage failure logs its exit code, timeout flag and bounded stderr. The Sandbox process is also reported to Worker Logs with its processId so the failure can be joined to the Containers process.complete event.

The stage wait itself is transport-resilient (runShellResilient): a Sandbox DO↔container drop mid-stage reattaches to the process by id and resumes (bounded by the stage deadline) instead of ending the request. The Dispatch backup CPU budget (BACKUP_CONTROL_CPU_MS) is aligned to the same 10 minute bound.

Asynchronous control acknowledgment (issue #344). Even with the larger budget, holding the POST /control request open for the whole snapshot outlived the dispatch/Worker request lifecycle, which cut the tenant request right after createBackup uploaded the archive and before the catalog write. The scheduled backup action now acknowledges immediately (202 { ok: true, accepted: true, jobId }) and enqueues the snapshot as a durable job on the per-tenant ActivityScheduler DO, so the dispatch cron and operators never block on it. A waitUntil task is cancelled ~30s after the response (platform cap) and was observed dropping a snapshot that was still cold-booting, so the backup must not rely on it. The crash-consistent catalog commit + reconciliation above keep the archive durable if the isolate is still replaced mid-snapshot. To trigger either mode on demand (ROOT, DEBUG_ROUTES=true), use POST /debug/backup/async or POST /debug/backup/sync.

The two paths behave differently when the lease is busy:

  • Scheduled acquisition is non-blocking: if another isolate holds the lease, this tick writes no backup and returns (it does not wait).
  • Manual (POST /api/admin/storage/sync) and the hard-restart checkpoint wait for the lease with bounded exponential backoff (up to 120s), then fail with an error rather than silently reporting success. A contended manual sync can therefore take up to 120s and then return an error; it never bypasses the lock.

A lease that cannot be proven free — held live, malformed, or unreadable — is treated fail-closed: manual callers keep waiting and then time out, and scheduled callers skip the tick. (Contention is bounded, not instant.)

Automatic backup trigger state machine

There is one automatic snapshot trigger: the Dispatch Worker cron, which fans a backup control action out to every non-frozen tenant. The tenant only proceeds when isRuntimeActive() reports a live runtime — an idle container is never woken — after which the interval/lock gates below decide whether an object lands in R2. Health probes play no part: GET /api/health is observability-only and must not alter the catalog.

stateDiagram-v2
    direction TB

    [*] --> Idle

    state "Idle (no snapshot scheduled)" as Idle
    state "Runtime active? (isRuntimeActive)" as RuntimeGate
    state "Interval elapsed?" as IntervalGate
    state "backup-lock free?" as LockGate
    state "Acquire shared lease (wait ≤120s)" as Acquire
    state "Snapshotting (sandbox.createBackup)" as Snapshot
    state "Encrypting at rest (F4E1)" as Encrypt
    state "Committing catalog + pruning" as Commit
    state "No backup written" as NoBackup
    state "Reprovision guard" as ReproGuard
    state "Dispatch cron (backup control)" as DispatchCron

    Idle --> Acquire : hard restart checkpoint (POST /debug/restart-openclaw)
    Idle --> Acquire : manual sync (POST /api/admin/storage/sync)
    Idle --> ReproGuard : reprovision (POST /debug/reprovision)
    Idle --> DispatchCron : Dispatch Worker cron (15 min)

    DispatchCron --> RuntimeGate : backup control action
    RuntimeGate --> Idle : not running (never wake)
    RuntimeGate --> IntervalGate : running
    ReproGuard --> Snapshot : checkpoint (container alive)
    ReproGuard --> Idle : 409 — no checkpoint and no restorable backup
    IntervalGate --> Idle : < effective interval since last
    note right of IntervalGate : max(BACKUP_INTERVAL_MINUTES, 2 × last snapshot duration)
    IntervalGate --> LockGate : interval elapsed and BACKUP_BUCKET bound
    LockGate --> Idle : lease held (skip this tick)
    LockGate --> Snapshot : lease acquired (60s TTL, 20s heartbeat)

    Acquire --> Snapshot : lease acquired (60s TTL, 20s heartbeat)
    Acquire --> NoBackup : lease held > 120s (bounded error)

    Snapshot --> NoBackup : InvalidBackupConfigError / R2 error
    Snapshot --> Encrypt : archive created
    Encrypt --> NoBackup : BACKUP_ENCRYPTION_REQUIRED and key missing
    Encrypt --> Commit : encrypted (.enc) or plaintext (warned)
    Commit --> Idle : backup-handle.json written, oldest pruned
    NoBackup --> Idle : lease released (CAS tombstone)

The same state machine as an Eraser flowchart. Render it by POSTing the DSL to the Eraser render API (see the eraser-diagrams skill); it is kept inline so the diagram can be regenerated without an API key.

direction down

Start [shape: oval]
Idle [shape: rectangle, label: "Idle - no snapshot scheduled"]
RuntimeGate [shape: diamond, label: "Runtime active? (isRuntimeActive)"]
HardRestart [shape: rectangle, label: "Hard restart checkpoint"]
ManualSync [shape: rectangle, label: "Manual sync"]
ReproGuard [shape: diamond, label: "Reprovision guard"]
DispatchCron [shape: rectangle, label: "Dispatch cron (backup control)"]
NotRunning [shape: oval, color: gray, label: "Skip (never wake)"]
IntervalGate [shape: diamond, label: "Interval elapsed?"]
LockGate [shape: diamond, label: "backup-lock free?"]
Acquire [shape: rectangle, label: "Acquire shared lease (wait ≤120s)"]
Snapshot [shape: rectangle, label: "Snapshotting (sandbox.createBackup)"]
Encrypt [shape: rectangle, label: "Encrypting at rest (F4E1)"]
Commit [shape: rectangle, label: "Commit catalog + prune"]
NoBackup [shape: oval, color: red, label: "No backup written"]

Start > Idle
Idle > HardRestart
Idle > ManualSync
Idle > ReproGuard : "POST /debug/reprovision"
Idle > DispatchCron : "Dispatch Worker cron (15 min)"
DispatchCron > RuntimeGate : "backup control action"
RuntimeGate > NotRunning : "not running"
RuntimeGate > IntervalGate : "running"
ReproGuard > Snapshot : "checkpoint (container alive)"
ReproGuard > Idle : "409 - no checkpoint, no backup"
IntervalGate > Idle : "too soon"
IntervalGate > LockGate : "yes + BACKUP_BUCKET bound"
LockGate > Idle : "lease held (skip tick)"
LockGate > Snapshot : "acquired (60s TTL, 20s heartbeat)"
HardRestart > Acquire
ManualSync > Acquire
Acquire > Snapshot : "acquired (60s TTL, 20s heartbeat)"
Acquire > NoBackup : "lease held >120s (bounded error)"
Snapshot > NoBackup : "InvalidBackupConfigError / R2 error"
Snapshot > Encrypt : "archive created"
Encrypt > NoBackup : "encryption required, no key"
Encrypt > Commit : "encrypted or plaintext warned"
Commit > Idle : "catalog written, oldest pruned"
NoBackup > Idle : "lease released (CAS tombstone)"
Automatic triggerEntry pointGates that must pass
Dispatch cron checkpointPOST /control (X-FourIA-Control: backup) via the Dispatch Worker crontenant not frozen; runtime active (isRuntimeActive(), never wakes an idle container); BACKUP_BUCKET bound; effective interval elapsed (max(BACKUP_INTERVAL_MINUTES, 2 × last snapshot duration)); 60s lock free; SDK presigned creds present
Hard-restart checkpointPOST /debug/restart-openclawnone (unconditional checkpoint before kill)
Reprovision guardPOST /debug/reprovisionruntime-active checkpoint (time-bounded) or a restorable backup exists; otherwise 409 (override with {"force":true})
Manual snapshotPOST /api/admin/storage/synccaller authorized (ADMIN/ROOT) — explicit admin action

Operator traps that leave the bucket empty even though the worker is healthy.

First, the Dispatch cron is the only automatic trigger. A tenant missing from TENANT_INDEX (or marked frozen) is skipped. The lerma HealthMonitor is observability-only and never backs up.

Second, the Sandbox SDK (>= 0.12.9) createBackup() on the remote path calls requirePresignedURLSupport(), which throws InvalidBackupConfigError unless R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, BACKUP_BUCKET_NAME, and CLOUDFLARE_ACCOUNT_ID are all set on the tenant worker. The failure is caught by scheduleBackupIfNeeded() and logged as [persistence] Scheduled backup failed: — the R2 bucket then contains only backup-handle.json (or nothing at all).

Third, snapshots only run against an active runtime. A sleeping container is skipped (never woken — it has no workload and no filesystem changes to capture), and POST /debug/reprovision returns 409 when the runtime is idle and no restorable backup exists.

Verify all three before assuming backups are running.

To verify the R2 backup bucket is reachable without triggering a snapshot, use POST /api/admin/storage/test-connectivity (ADMIN or ROOT). It HEADs the backup catalog handle; a 404 (no backup written yet) still counts as connected. ROOT users can run it from the Diagnostic Dashboard’s Test Backup Connectivity button even though the Backups page is ADMIN-gated.

The number of retained versions is BACKUP_MAX_VERSIONS (default 3); the oldest beyond that are pruned. Retention is enforced on the snapshot path and by the explicit catalog reconcile (reconcileBackupCatalog, run by the durable repair step before every scheduled snapshot and by restore), so a run of failed snapshots cannot leave the catalog or R2 over the cap. GET /api/admin/storage/backups is a pure read; run POST /api/admin/storage/reconcile (ADMIN or ROOT) to adopt orphaned archives, sweep dead ones, re-encrypt plaintext ones, and prune on demand.

4.3b Change-aware backups (opt-in)

Scheduled snapshots can skip tenants whose state has not changed, and a per-tenant watch can snapshot within minutes of a change instead of waiting for the next 15-minute dispatch tick. Both are off by default; see docs/WORKER-DEVELOPMENT.md (Change-Aware Backups) for the mechanism.

  • Enable: set the lerma provisioning flags BACKUP_CHANGE_DETECTION and/or BACKUP_CHANGE_WATCH to true for that lerma environment, then redeploy the affected tenants. BACKUP_CHANGE_WATCH also binds the BACKUP_WATCH Durable Object.
  • Roll back: set the flag back to false and redeploy. Both flags are always emitted explicitly, so the rollback takes effect on redeploy.
  • Observe: [persistence] Scheduled backup skipped: tenant state unchanged means the gate saved a snapshot. [backup-watch] scan failed (or request failed) means the watch fell back to a snapshot or retry. A scan failure is never a skipped backup.
  • Watch for cost: each armed tenant runs one short find per check. The watch disarms after 10 idle minutes, so a quiet tenant is not kept awake.

4.4 Restoring

  • Automatic — restoreIfNeeded() runs at container startup and restores the committed recovery point when the container has no tenant state. The restore-needed marker is not required to trigger a cold restore: its absence is normal while idle and after a successful restore. A cold restore writes the marker before applying the archive and clears it after restore plus legacy state materialization succeed. If candidate scanning finds no restorable archive and no stable recovery point exists, it clears the temporary marker for the permitted clean start. A recorded stable recovery point with no restorable archive instead fails closed and keeps the marker.
  • Targeted — restoreBackupById(sandbox, bucket, backupId, env, ctx, options) restores a specific snapshot (persistence.ts). Use the backup history + config version list to pick one. A successful targeted restore force-promotes the restored archive to the recovery point (stable-backup.json) even when it predates the current one, so the next cold boot restores the same archive. Without this, restoreIfNeeded keeps preferring the newer pointer — which advances on every committed snapshot — and silently re-restores the wrong state (e.g. a post-deploy snapshot taken from a not-yet-materialized config, observed on tenant-tarjetero-dev).
  • Restore without moving the recovery point — POST /api/admin/storage/restore/:id?setCurrent=0 (also ?async=1&setCurrent=0) passes { setRecoveryPoint: false } through to restoreBackupById, which applies the archive but leaves stable-backup.json unchanged. Use this to inspect or recover an older snapshot with the next fresh container reverting to the current recovery point. The flag defaults to set-current (?setCurrent=1 / omitted), matching the targeted-restore behavior above. The Backups page exposes it as the Set as recovery point checkbox (checked by default); unchecking maps to setCurrent=0. The restored state is then protected by a restore-only hold (§4.3), so scheduled snapshots cannot move the recovery point onto it.
  • After a restore the gateway is launched and the config is re-captured — every restore path (the durable boot job, the durable restore job, and the synchronous admin route) finishes through launchRestoredState (src/post-restore.ts): it brings the restored state up (ensureGateway({ waitForReady: true })) and captures the live config as a new configs/<ts>.json plus stable/config.json, so the restored configuration becomes the latest and stable config version. A launch failure does not fail the restore (the restore already succeeded): the sync route returns gatewayReady: false and the durable job succeeds with detail: "gateway-launch-pending", leaving the durable restore-config-pending.json marker for the next boot/maintenance tick. A doctor --fix repair is deferred (without consuming attempts) while a restore is queued/applying or the restored gateway is not up yet — otherwise its fix would be silently overwritten by the next re-apply of the recovery point (observed on tenant-tarjetero-dev).
  • Timeouts never start a second restore — sandbox.restoreBackup is bounded by RESTORE_BACKUP_TIMEOUT_MS. The bound only stops the wait: the container call keeps running. A timeout therefore records the restore as outcome unknown, and every other restore is refused (409, RestoreInFlightError) until that record is older than the restore phase budget (GATEWAY_RESTORE_PHASE_BUDGET_MS, 11 min). A restore that fails definitively does not block a retry. The restore-needed marker stays set, so the next restore after the window re-applies the archive.
  • Writer quiesce + maintenance window — restoreBackupById opens a cross-isolate maintenance window (maintenance/replacement.json, operation restore) and stops the gateway immediately after it takes the shared restore lease, then closes the window in a finally. A live gateway (or automations/channels) therefore cannot keep writing while the archive replaces the writable layer, and a concurrent health/proxy refetch cannot restart it mid-restore. restoreIfNeeded refuses to run while any maintenance window is active, which breaks the restore → stop-gateway → refetch → remount-the-latest loop; the window self-heals on its TTL if the controller dies.
  • Recovery visibility — the Backups page marks the committed recovery point in the snapshot timeline (GET /api/admin/storage/backups → stableBackupId) with a recovery point badge and a header line (A cold boot restores <id> — <date>). The loading page names the backup + date being loaded and shows a progress bar synchronized with the durable restore’s phase events: while a restore is in flight it reads the durable restore-target.json (the archive being applied) and otherwise the committed stable-backup.json, surfaced as recovery.restoring / recovery.committed, plus restoreProgress ({ phase, label, fraction }) from the ActivityScheduler DO, on /api/health (non-passive) and /api/startup/ws. So an operator can always tell which snapshot a cold start is restoring, when it was taken, and how far the restore has progressed, without opening logs. GET /api/admin/boot-state returns the same restoreProgress for inspection.

Restore and scheduled-backup safety rules:

  • The pre-restore umount is best-effort: umount … || true only guards the shell command, not the exec RPC. A cold-start/platform interruption on that exec is caught and logged instead of aborting the restore before restoreBackup runs — the previous fatal behaviour kept restore-needed and failed the startup restore barrier on every boot.
  • An empty catalog read does not clear restore-needed. The read can be transient (a reconcile list failure, or an archive not yet visible); a pending marker is kept so an operator-requested restore is never silently dropped. Restore failures keep the marker. If candidate scanning finds no restorable archive, a recorded stable recovery point fails closed and keeps the marker; with no stable pointer, the clean-start fallback removes it.
  • If restore fails after writing partial state, restore-needed remains in R2 and forces a retry even when a config file is already present. Scheduled snapshots skip while that marker exists, rechecked under the backup lease, so a cron tick cannot overwrite the stable snapshot during a cold boot. The gate is keyed on the durable R2 marker (not a container-state probe) so an inconclusive probe can never silently disable scheduled backups.

Therefore, an absent restore-needed object by itself is not evidence that R2 lost a restore request. Check stable-backup.json and its referenced backups/<id>/meta.json; those identify the committed recovery point. A fresh container restores it based on the state probe even when the marker was absent.

Each entry in backup-handle.json records whether it was created with the SDK’s localBucket mode (persistence.ts). That flag is forwarded to restoreBackup() so a local-binding backup restores through the local R2 binding instead of the remote presigned-URL path — a backup created locally only exists in the local bucket, so a remote restore 404s with Presigned URL download failed (exit code 22).

Local-bucket mode is the explicit BACKUP_LOCAL_BUCKET=true opt-in; when unset it falls back to DEV_MODE for local development. Do not rely on the DEV_MODE fallback for a real deployment (including the cloud E2E, which sets DEV_MODE=true only to bypass Access): set BACKUP_LOCAL_BUCKET=false there so backups land in the remote BACKUP_BUCKET the Worker restores from.

Entries created before the flag was persisted carry no localBucket; on restore it is defaulted from the same signal used at create time, so legacy local backups still restore via the local binding. A restore that 404s at the presigned URL (archive gone / unreachable) is treated like BACKUP_NOT_FOUND — the stale entry is dropped from the catalog so one dead backup cannot block every subsequent boot.

Operator workflow for “I need to roll back to a known-good config”:

  1. List config versions: GET /api/admin/storage/config (or inspect configs/ in R2).
  2. Confirm the snapshot you want exists in backup-handle.json’s backups[].
  3. If the container is up, stop the gateway (POST /debug/stop-gateway), then call a targeted restore, then restart. If the container is down, the next boot auto-restores the newest.

⚠️ The current implementation uses the Sandbox backup API and the BACKUP_BUCKET R2 binding. It does not mount R2 with s3fs. Never delete tenant backup objects outside the retention or decommission procedure.

4.5 Setting up R2 (one-time)

Scheduled backups run when the Worker has its BACKUP_BUCKET binding and the interval/lock guards pass (scheduleBackupIfNeeded() in persistence.ts). The bound bucket is provisioned per tenant, for example fouria-backup-{slug}{-environment}{-platform}.

The Sandbox SDK remote path requires R2 S3 credentials. In @cloudflare/sandbox >= 0.12.9, createBackup() (and restoreBackup()) on the non-localBucket path calls requirePresignedURLSupport(), which throws InvalidBackupConfigError unless all of R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, CLOUDFLARE_ACCOUNT_ID, and BACKUP_BUCKET_NAME are present on the tenant Worker. hasBackupCredentials() only gates on BACKUP_BUCKET, so a misconfigured worker still runs the scheduler and then fails at snapshot time (logged as [persistence] Scheduled backup failed:) with an empty bucket. Only local-bucket mode (BACKUP_LOCAL_BUCKET=true, or the DEV_MODE fallback for local development) avoids the credential requirement.

  1. Bind the tenant bucket — the Lerma provisioner creates and binds the tenant R2 bucket as BACKUP_BUCKET and sets BACKUP_BUCKET_NAME.

  2. Let lerma resolve it (it is now mandatory) — the provisioner resolves a credential per instance and fails provisioning (rolling back) when none can be produced; it no longer warns and deploys a tenant that silently cannot back up. Strategy is R2_TOKEN_STRATEGY (auto | shared | per_instance):

    • auto (default): uses R2_TOKEN_MAX_PER_ACCOUNT; a known limit of at most 250 selects shared, a larger or unknown limit selects per_instance. Cloudflare exposes no maximum-token endpoint, so the limit defaults to unknown → per-instance token minted at instance creation, scoped to the tenant bucket.
    • shared: set R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY in the lerma environment (generate with mix lerma.r2_token --all-buckets).

    Generated per-instance credentials are persisted on the instance record and reused across redeploys; a token minted for a failed brand-new provisioning is deleted on rollback. CLOUDFLARE_ACCOUNT_ID is a plain var and is always injected.

  3. Ensure the client is being probed — the lerma HealthMonitor is the only automatic trigger and only polls clients in production/error. A client in draft gets no scheduled snapshots regardless of credentials.

    # For a standalone/legacy Worker (not provisioned via lerma):
    npx wrangler secret put R2_ACCESS_KEY_ID
    npx wrangler secret put R2_SECRET_ACCESS_KEY

Sandbox snapshot behavior differs between deployed and local environments; local tests use the SDK’s localBucket mode when BACKUP_LOCAL_BUCKET=true (or the DEV_MODE fallback) in persistence.ts. The flag is persisted in the backup catalog so restores route to the same mode — a local backup restores via the local R2 binding, and a remote (presigned-URL) backup restores via R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY / CLOUDFLARE_ACCOUNT_ID / BACKUP_BUCKET_NAME. Pre-persist entries default to the current mode. There is no s3fs mount.


5. Troubleshooting — Critical Issues

This section maps symptoms → root cause → the exact recovery from this guide.

5.1 Container stuck in Error / image deleted → RESCUE

Symptom: /api/health?passive=1 returns degraded:true, containerError:"no such image" (or no matching app). The instance can never boot.

Why: the stale-image cleanup cron deleted the fouria:unstable-<sha> image the application references (diagnostics.ts:42). The cleanup script now protects every tag still referenced by a container application — including the versions of an in-progress rollout — and aborts (fail closed) when that in-use set cannot be resolved (.github/scripts/cleanup-stale-images.sh). Production deploys are also restricted to stable release tags, so a production application can no longer be pinned to the unstable-<sha> channel (Lerma.Cloudflare.ImageRegistry.stable?/1).

Fix: rescueContainer (§3.4):

curl -X POST "https://<worker>/debug/rescue-container" -d '{"dryRun":true}' | jq .
# confirm action would be repair-image-and-destroy, then:
curl -X POST "https://<worker>/debug/rescue-container" -d '{"dryRun":false}' | jq .

Credentials required: CLOUDFLARE_API_TOKEN (Containers:Edit), CONTAINER_APPLICATION_ID, and either CONTAINER_IMAGE_TAG or an explicit expectedImage (diagnostics.ts:304, .dev.vars.example:43). Without them, the image cannot be repaired — only the container is destroyed (you then must re-deploy the correct image tag).

NEVER set these on tenant workers — the rescue tool re-points the application image and is operator/ROOT-only (.dev.vars.example:40).

5.2 Gateway “already running” / EADDRINUSE on restart

Symptom: restart fails with gateway already running (pid …) or failed to bind gateway socket.

Why: a previous gateway still holds the native lock / TCP bind when the new one starts (gateway/process.ts).

Fix: for a clean state use killGateway() (the POST /debug/stop-gateway route), which SIGTERMs with the GATEWAY_KILL_GRACE_MS (5s) grace and only SIGKILLs if still active, then wait for the port to close before reprovisioning. POST /debug/restart-openclaw (hard restart) does this then reprovisions via start-openclaw.sh. Inspect lock state with GET /debug/gateway-lock.

5.3 Gateway process invisible but container live

Symptom: listProcesses() shows nothing, yet /health answers.

Why: the gateway runs as a detached child of start-openclaw.sh (process.ts:427). The SDK does not track it, but it is healthy.

Fix: the live-gateway fallback (findExistingGatewayProcess with probeLive:true) detects it via a bounded /health probe. Prefer restart/rescue APIs that use this path rather than destroy-container (which would throw away a healthy instance).

5.4 Backup restore loops / “hadBackup but not restored”

Symptom: health startup.ok=false permanently with Backup found but not restored.

Why: restoreIfNeeded found an entry but every backup expired/was missing (persistence.ts:212). The code explicitly reports a clean start in this case so it does not gate startup forever (persistence.ts:233).

Fix: confirm real backups exist (backup-handle.json → backups[]); if none, the instance boots fresh (expected). If backups exist but restore fails, check BACKUP_ENCRYPTION_KEY (encrypted backups cannot be read without it).

Manual-restore loop (restore → stop-gateway → refetch → remount the latest). A manual restore of an older archive kills the gateway so no writer mutates the layer being replaced. Without a guard, the next health/proxy refetch calls ensureGateway → restoreIfNeeded, which prefers the committed recovery point (the newer archive) and silently undoes the restore. Two guards prevent this: restoreBackupById holds a cross-isolate maintenance/replacement.json window (operation restore) for the whole restore, during which ensureGateway refuses to launch, and restoreIfNeeded itself refuses to run under any active window. A targeted restore that should survive cold boots also force-promotes the archive (the default); a setCurrent=0 restore deliberately does not, and is reverted on the next fresh container.

5.4b Stuck gateway start (“already in progress” forever)

Symptom: logs repeat [Gateway] Gateway start already in progress for openclaw , waiting... for tens of minutes while listProcesses returned N processes keeps climbing and the port stays closed (health probes curl exit 7 every ~5s). The gateway never binds 18789.

Why: the shared ensureGateway start promise spans restore → supervised start-openclaw.sh start → port wait. If the wrapper never execs the gateway (or the container control channel drops), the promise does not settle and every request waits on it. The raw listProcesses().length cannot distinguish this from a busy boot — it counts exited processes too and only grows.

Automatic recovery: the Worker now tracks the start phase and force-resets a phase that overran its budget, logging a structured gateway.start.stuck event and killing the stuck process; after repeated resets it reprovisions the container. Before resetting a port-wait, it makes a fresh /health check: a gateway that became healthy after the original readiness deadline is adopted (gateway.start.late-ready) instead of killed. If that recheck cannot reach the container, health is unknown and reset is deferred until a later request can probe again. A start-openclaw.sh start wrapper older than the cold-start budget with the port closed is also killed immediately.

Manual recovery (if needed): POST /debug/reprovision (ROOT, DEBUG_ROUTES=true) to destroy + reprovision the container, or POST /debug/restart-openclaw to hard-restart the gateway in place; a freeze/unfreeze from lerma also forces a fresh cold start.

Investigate: filter the cloudflare-workers dataset to event = "gateway.start.stuck", event = "gateway.start.phase", event = "gateway.start.late-ready", or event = "gateway.start.health-recheck-failed" — records carry the phase, attempt, elapsed time, and process id needed to distinguish a real stuck start from a late healthy gateway or an inconclusive probe.

5.4c Repeated same-size snapshots and a container that never sleeps

Symptom: R2 keeps receiving a full archive of the same size every few minutes/hours (e.g. 422 MB at 02:35/02:39/02:44 and 05:39/05:43/05:49 UTC), the container instance shows running with a started_at days old, and backup-handle.json has a stale lastScheduledBackup even though new entries appear. Often paired with a cost spike at a quiet hour.

Why: the idle-sleep deprovision job snapshots before stopping. On a large tenant that archive takes minutes, and its own RPCs renew the Sandbox DO activity window, so the stop-time stopIfIdle() re-check refuses (active:activity). The SDK re-arms, the next idle expiry enqueues another deprovision, and each attempt uploads the full archive again. Deprovision archives used to be unstamped, so the scheduled cadence did not advance either.

Fix (current bundle): TenantSandbox.onActivityExpired() first reads the catalog (hasRecentSnapshot); a committed snapshot inside the idle window (+5 min slack) already covers the unchanged filesystem, so it stops inline without a new archive, and a deprovision archive now stamps lastScheduledBackup. If an older tenant still loops, redeploy it to the current bundle. Diagnose with:

npx wrangler r2 object get fouria-backup-<slug>/backup-handle.json --remote --file -
npx wrangler r2 object get fouria-backup-<slug>/backups/<id>/meta.json --remote --file -

Related — 30-minute dev heartbeats. OpenClaw 2026.10.5 retired agents.entries.*.heartbeat: the heartbeat is an ordinary automation job and config reloads never rewrite it, so a migrated dev instance can keep the default 30-minute cadence (a full agent turn on the main session each time). The dev-only ensure_dev_heartbeat_cadence in start-openclaw.sh reconciles it on post-start; to fix a warm instance immediately:

openclaw automations edit heartbeat-fouria-builder \
  --every 12h --session isolated --light-context

5.4d “The gateway rebooted — was it memory?”

Symptom: the gateway cycled (2–5 boots in a day), the dashboard dipped to starting/503, and someone asks whether it hit the container memory limit.

Answer it from the Worker first — no SSH needed. /api/health (and /api/health-snapshot / /api/health-ws) carries a gateway block for a running gateway:

"gateway": { "pid": 4242, "rssBytes": 2234000000, "threads": 29 }

RSS is sampled from /proc/<pid>/status at most once a minute, so it is cheap to poll. The same numbers land in Analytics Engine as a gateway-resources row (double1 = RSS bytes, double2 = threads, blob1 = gateway-resources, blob2 = customer) — chart it to tell a monotonic leak from a transient peak:

SELECT toStartOfHour(timestamp) AS hour,
       max(double1) / 1048576 AS rss_mib, max(double2) AS threads
FROM fourai_cost_metrics
WHERE blob1 = 'gateway-resources' AND blob2 = 'tarjetero'
  AND timestamp > NOW() - INTERVAL 1 DAY
GROUP BY hour ORDER BY hour

Reference points on a standard-1 container (4 GiB, no swap):

ReadingMeaning
~1.1–1.4 GiBnormal baseline right after a hard restart
~2.1 GiB (peak ~2.6 GiB)observed steady state of a busy dev gateway; still healthy
≥ 1.5 GiBOpenClaw’s own rss_threshold warning — expected to fire at this level; it is not an action item
≥ 3 GiBthe dev watchdog threshold — the gateway gets replaced

Before blaming memory, rule it out properly. An OOM kill leaves fingerprints that a graceful restart does not:

# inside the container (wrangler containers ssh)
grep oom_kill /proc/vmstat            # must stay 0
dmesg | grep -i "killed process"      # empty
grep -ciE "heap out of memory|Allocation failed" /tmp/gateway-stdout.log
grep -c "restart-signal fence" /tmp/gateway-stdout.log   # graceful self-restarts
grep -c "host timing gap detected" /tmp/gateway-stdout.log

A non-zero restart-signal fence / received SIGTERM count means the gateway restarted itself on purpose (doctor repair, config reload, update) — not OOM. A host timing gap detected: process was frozen ~Ns line means the platform froze the microVM; a freeze long enough to blow past startup deadlines is followed by the supervisor replacing the process, which looks like a reboot but is not a memory event. Neither of those is an OOM.

Dev-only memory watchdog. On a dev instance (-dev hostname / OPENCLAW_DEV_MODE) the Worker replaces the gateway when RSS reaches GATEWAY_RSS_RESTART_BYTES (3 GiB of the 4 GiB container), at most once per 30 minutes, and logs:

{ "event": "gateway.memory.pressure", "rssMib": 3072, "threads": 31, "limitMib": 3072 }

If you see that event, the watchdog fired before the kernel OOM killer could — that is the intended behaviour, not an outage. It is deliberately gated to dev: a replacement interrupts live agent sessions. Production keeps today’s behaviour (no automatic replacement).

Fix the cause, not the symptom. RSS growth is driven by oversized persisted transcripts and accumulated worker_thread V8 heaps that an in-process restart never reclaims; see ../../docs/plans/sandbox-memory-usage.md for the mitigation set and what is still open.

5.5 Platform & setup troubleshooting

SymptomLikely causeAction
npm run dev / deploy: UnauthorizedCloudflare Containers not enabled for the accountEnable Containers in the Containers dashboard
[sandbox-do] backup.restore … Presigned URL download failed (exit code 22) … 404restore hit the remote presigned-URL path for an archive that only exists in the local R2 binding (a BACKUP_LOCAL_BUCKET/localBucket backup), or BACKUP_BUCKET_NAME doesn’t match the BACKUP_BUCKET binding bucketredeploy so the backup catalog persists localBucket (§4.4) — local backups restore via the local binding; for remote backups verify R2_ACCESS_KEY_ID/R2_SECRET_ACCESS_KEY/CLOUDFLARE_ACCOUNT_ID/BACKUP_BUCKET_NAME all point at the same bucket as the BACKUP_BUCKET binding (§4.5)
[sandbox-do] backup.restore error <id> — The operation was aborted (~30 s) repeated on every boot, with sandbox.exec … The operation was aborted for health probes/umountthe archive restore ran under a request execution context, which Cloudflare cancels shortly after the response, so sandbox.restoreBackup was aborted mid-flight (large archives, e.g. 350 MB, never finish)fixed: request paths defer the restore to the durable ActivityScheduler boot job (RestoreDeferredToBootJobError → GatewayBootPendingError). Verify with grep -c "await restoreIfNeeded" apps/fouria/src/index.ts (must be 0) and look for [Gateway] Restore deferred to the durable boot job; the restore then runs in the DO alarm and shows Durable boot job began/succeeded
Gateway won’t startmissing/incorrect secretsnpx wrangler secret list + npx wrangler tail
Config changes not taking effectstale Docker build cachebump the # Build cache bust: comment in Dockerfile, then redeploy
Slow first requestcold start (1–2 min)expected; later requests are faster
R2 not mountingan R2 secret missing, or running the local Vite dev serverset all four R2 secrets (§4.5); R2 mount is production-only
Access denied on /_admin or /apiCF_ACCESS_TEAM_DOMAIN / CF_ACCESS_AUD not setconfigure Cloudflare Access (§8.1)
Devices not appearing in admin UIdevice-list CLI takes 10–15 s (WS overhead)wait and refresh
WebSocket issues in local devlocal vite dev WS-proxy limitationdeploy for full WS functionality
Windows: script exits 126CRLF line endings in shell scriptsadd .gitattributes with * text=auto eol=lf

5.6 Deploy-window “Could not list processes” / platform interrupted

Symptom: logs show [Gateway] Could not list processes: followed by a stack trace ending in createPlatformInterruptedError / translatePlatformInterruption, and/or /debug/processes returns a 503 with { "status": "platform_updating" }.

Why: the Sandbox SDK throws OperationInterruptedError (“Sandbox operation listProcesses was interrupted while the platform was updating the sandbox runtime”) when the Durable Object isolate is replaced — exactly what happens during a deploy / code update. It is a transient platform lifecycle event, not an application failure, and it self-heals on the next request once the new isolate is live.

Fix: nothing to fix. In findExistingGatewayProcess this is logged as a single concise “platform was updating the sandbox runtime” line instead of a raw stack trace; /debug/processes returns 503 platform_updating so debug tooling doesn’t misreport a crash. If it persists beyond the deploy window, re-check wrangler tail and the container instance health in the Containers dashboard.

5.7 Container terminated/destroyed → every request 500s or burns a cold-start budget

Symptom A — platform-terminated: the platform SIGTERMed the container (idle recycle, maintenance, rescue) and after that all requests fail with [Gateway] Could not list processes: plus sandbox.exec error … HTTP error! status: 500 — the container is dead and nothing ever restarts it. The dashboard shows “Could not list processes” and the UI is fully bricked.

Symptom B — out-of-band destroyed (orphaned): the instance was destroyed from the Cloudflare Containers dashboard (or a failed operator force-stop). The Sandbox DO is still alive but points at a placement that no longer exists, so every request logs around no container instance available / maximum number of running container instances exceeded and each request burns the SDK’s full provisioning retry budget (≈20s instance + 90s port) before giving up — CPU is spent repeatedly trying to reach a container that can never be admitted, and the UI never recovers.

Symptom C — control-protocol mismatch (deploy cutover): during a deploy the new Worker bundle carries a new @cloudflare/sandbox package while the still-running container predates the new control handshake, so every request logs sandbox.exec error … — 'utils.getRuntimeMetadata' is not a function. (or RuntimeControlProtocolError / unsupported-protocol-version). This is the transient stable↔1.0-preview skew described in AGENTS.md; it clears once the container rollout replaces the instance.

Why: the Worker holds a Sandbox stub for a DO whose container instance is gone. Retrying any RPC against that instance fails forever, and the old code swallowed the error and returned null instead of reprovisioning — so no fresh container was ever created (gateway/sandbox-recovery.ts).

Fix (automatic): maybeRecoverSandbox classifies all three failure classes and calls recoverStaleSandbox (destroy the dead instance + reset the startup/persistence caches + reset the orphan streak). The next RPC auto-provisions a fresh container and the normal cold-start flow runs (R2 restore → start-openclaw.sh). Wired into findExistingGatewayProcess, checkGatewayHealth, and the ensureGateway startShell retry (gateway/process.ts).

  • Stale (HTTP error! status: 500) recovers immediately (isStaleContainerError).
  • Control-protocol mismatch ('utils.getRuntimeMetadata' is not a function / RuntimeControlProtocolError) also recovers immediately: the SDK logs the sandbox.exec error … line but re-throws the bare TypeError, so isStaleContainerError matches the message and reprovisions from the (now matching) image config instead of erroring until the platform rollout catches up.
  • Orphaned (no container instance available) is retryable on its own, so it is only treated as an orphan once the failures span ORPHANED_INSTANCE_GRACE_MS (120s) — deliberately longer than the SDK’s ≈110s provisioning budget so a legitimately slow cold start is never cancelled mid-provision. The streak is cleared whenever a sandbox RPC succeeds (the container is demonstrably alive), so a healthy instance never trips the threshold later.

Recoveries are cooldown-guarded (STALE_RECOVERY_COOLDOWN_MS, 60s) to prevent destroy-storms during request floods. Look for [SandboxRecovery] and repeated provisioning failures lines in wrangler tail. Manual fallback: POST /debug/reprovision (works even when the gateway is gone), or restart the container from the Containers dashboard.

NOT covered: permanent image errors (no such image) still require the rescue tool (§5.1), and deploy-window OperationInterruptedError is expected and self-heals (§5.6).

5.8 Trace a Worker failure into its Sandbox process

The Tenant Worker writes structured worker.request.failed records for unhandled request errors and sandbox.process.failed records when a Sandbox command fails, times out, or cannot be launched. Worker logs also include worker.request.started / worker.request.completed records with the method, path, status, duration, and cfRay when provided by Cloudflare. Cloudflare adds $metadata.requestId and $metadata.traceId to Worker log events automatically; error records also pass an Error to console.error so $metadata.error and exception details are populated.

In Workers Observability, filter the cloudflare-workers dataset by event = "sandbox.process.failed" (or worker.request.failed). For a process failure, copy its processId, then select the containers dataset and filter on the same processId plus $container.applicationId. On process.complete, use exitCode and timedOut as the outcome: the container’s generic message may read success even when the process exited non-zero. The structured Worker record intentionally excludes the command and process output to avoid leaking credentials; use the existing ROOT-only debug/process diagnostics when those details are needed. Persistence events (persistence.config-version.failed, persistence.restore.failed, and storage.restore.failed) include backupId when an archive exists; use it to correlate Worker errors with the container’s backup.create / backup.restore events. persistence.backup.failed records failures before the archive is necessarily created.

For CPU-limit errors during a backup or restore, filter Worker logs to event = "persistence.phase". The paired records share an operationId; use cfRayId to join request-triggered work to worker.request.*, or jobId for ActivityScheduler work. The last started phase without a terminal record is the stage to investigate first. See the Worker development guide for the phase fields and common stage names.


6. Subsystem Deep-Dives

6.1 Working with Cloudflare Containers

  • Image reference is built as registry.cloudflare.com/<accountId>/<name>:<tag> (buildImageReference, diagnostics.ts:213). name defaults to fouria (DEFAULT_CONTAINER_IMAGE_NAME).
  • Rescue needs the Containers API (/accounts/{account}/containers/applications/..., createContainersApiClient, diagnostics.ts:235). It uses Authorization: Bearer <CLOUDFLARE_API_TOKEN> and a full_auto rollout.
  • Timeouts that keep a degraded/off container from hanging: CONTAINER_INSTANCE_TIMEOUT_MS (20s), CONTAINER_PORT_TIMEOUT_MS (90s — kept generous for cold starts), CONTAINER_POLL_INTERVAL_MS (300ms) (config.ts:33). Override via SANDBOX_INSTANCE_TIMEOUT_MS / SANDBOX_PORT_TIMEOUT_MS / SANDBOX_POLL_INTERVAL_MS (.dev.vars.example:78).

Container cost & lifecycle

The tenant container is a standard-1 instance (½ vCPU, 4 GiB memory, 8 GB disk). Approximate 24/7 monthly cost (per Cloudflare Containers pricing):

ResourceProvisionedApprox. cost
Memory (4 GiB)billed 24/7~$26/mo
CPU (½ vCPU, ~10% used)usage-based~$2/mo
Disk (8 GB)billed 24/7~$1.50/mo
Workers Paid planflat$5/mo
Total~$34.50/mo

Notes:

  • CPU is billed on active usage, not provisioned capacity. Memory/disk are billed on provisioned capacity for the full uptime.
  • Cold starts take 1–2 minutes; the first request after a sleep/wake is slow.
  • Cost floor from the derived idle timeout — non-billable containers sleep when idle instead of running 24/7. The timeout is derived from the instance hostname (staging 15m, dev 2h). Before a non-billable container idle-sleeps, the Worker snapshots it (a durable backup-then-stop job), so a sleep never discards uncommitted state. The billable production environment does not auto-sleep; it is stopped only by an explicit lerma freeze (which also snapshots first) and wakes on traffic. There is no SANDBOX_SLEEP_AFTER secret to set. Waking a sleeping container is the ActivityScheduler DO alarm’s job.

6.2 R2 issues

SymptomLikely causeAction
Backup stored UNENCRYPTED warningBACKUP_ENCRYPTION_KEY unsetSet the key (secret_text binding)
Backup fails closedBACKUP_ENCRYPTION_REQUIRED=true, key missingProvide the key
Restore of encrypted backup failskey mismatch / missingEnsure BACKUP_ENCRYPTION_KEY matches the one used at write time
Scheduled backup never runsmissing R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY / BACKUP_BUCKET_NAME / CLOUDFLARE_ACCOUNT_ID, or the tenant is never probed while healthySandbox SDK presigned-URL path needs all four; hasBackupCredentials (persistence.ts) gates the scheduler, requirePresignedURLSupport throws InvalidBackupConfigError at snapshot time. Scheduled snapshots now fire on any healthy /api/health (passive included), so verify the lerma HealthMonitor actually polls the tenant
Reprovision returns 409no restorable backup and the checkpoint failedExpected fail-closed behaviour. Restore a checkpoint via POST /api/admin/storage/sync first; only pass {"force":true} when the container is already gone and its state is unrecoverable
Old config versions pile upretention is 30 days; normalcleanupOldConfigVersions prunes automatically
More backups than maxVersionsa run of snapshots failed before their retention prune, so reconciled orphans accumulatedself-heals via the explicit reconcile: the durable repair step (and POST /api/admin/storage/reconcile) enforces retention through reconcileBackupCatalog, while GET /storage/backups itself never mutates

R2 persistence uses the Sandbox SDK backup API; there is no s3fs mount to inspect. Verify backup state with GET /api/admin/storage/backups and never delete tenant backup objects outside retention or decommission procedures.

6.3 AI Gateway issues

The only provisioned AI provider path is Cloudflare AI Gateway, configured when all three are set (start-openclaw.sh:161, config.ts:97):

npx wrangler secret put CLOUDFLARE_AI_GATEWAY_API_KEY   # your gateway API key
npx wrangler secret put CF_AI_GATEWAY_ACCOUNT_ID        # Cloudflare account id
npx wrangler secret put CF_AI_GATEWAY_GATEWAY_ID        # gateway id within the account

Cloudflare AI Gateway gives caching, rate limiting, analytics, and cost tracking, and the cloudflare-unified-billing plugin extends the catalog to 150+ models (Google Gemini, Anthropic Claude, OpenAI, xAI Grok, DeepSeek, Alibaba Qwen, MiniMax, Moonshot Kimi, and more) via the AI Gateway REST API. The plugin — plus the cloudflare-ai-gateway provider and diagnostics-otel — is pre-installed into the image at build time from ClawHub (unpinned = latest release; published at https://clawhub.ai/paso4/plugins/cloudflare-unified-billing) so no network install runs on the latency-critical boot path. start-openclaw.sh only installs from ClawHub as a fallback when the extension state is missing. Lerma provisions the AI-scoped credential to tenant workers as both CLOUDFLARE_AI_GATEWAY_API_KEY and the REST alias CLOUDFLARE_API_TOKEN.

Premade agents:

Production deployments ship no premade agents — only the main agent written by openclaw onboard. The dev-only Fouria Builder and Fouria Q&A agent are injected by moltlazy only on dev instances (the Worker derives OPENCLAW_DEV_MODE from the instance’s own -dev hostname, or DEV_MODE locally; they can self-install plugins and read per-tenant client documentation materialized from R2). Neither agent is ever part of a production instance: their purpose is to model the production agent gateway and subagent squad (agents, skillsets, plugins, tools, automations) that satisfies the client’s business logic. Because OpenClaw has no per-agent run timeout, dev instances raise the global agents.defaults.timeoutSeconds to 1800 s (30 min) so the Builder is not aborted mid-build; production keeps the 300 s baseline.

AgentEnvironmentPrimary modelPurpose
fouria-builderdev onlyClaude Sonnet 4.6Read client proposal/phases, build a working POC
fouria-qadev onlyClaude Sonnet 4.6Run cctr corpus validation against the Builder’s POC
SymptomCauseAction
Onboard skips AI Gateway authone of the three vars missingset all three; mark NEED_ONBOARD=true
cloudflare-ai-gateway plugin missingbuild-time pre-install missing/removedit is pre-installed in the image; start-openclaw.sh reinstalls from ClawHub as a fallback (bounded, non-fatal)
cloudflare-unified-billing plugin missingbuild-time pre-install missing/removedit is pre-installed in the image; start-openclaw.sh reinstalls the latest @paso4/cloudflare-unified-billing from ClawHub as a fallback (unpinned, bounded, non-fatal)
diagnostics-otel install failspre-install missing/removedit is pre-installed in the image; the fallback installs the latest ClawHub build (unpinned). There are no version pins to bump
Model requests fail with auth errorswrong/expired gateway keyrotate CLOUDFLARE_AI_GATEWAY_API_KEY
No embedding / memory search errorsno embedding key (CF_AI_GATEWAY_OPENAI_COMPAT_KEY)memory search auto-disabled (start-openclaw.sh:422) — expected

The cloudflare-unified-billing plugin (40+ models via the REST API) is pre-installed in the image and enabled alongside the gateway provider; enablement is written by a single config patch (which also enables admin-http-rpc and disables diagnostics-otel unless its endpoint is set), not a plugins enable CLI call, to avoid a hanging ClawHub registry refresh.

When migrating existing tenants from the old provisioner, re-provision the tenant and perform a cold restart after the AI credential is configured. Older R2 snapshots may contain a stale native-provider auth profile and the old random AI binding; a warm restore alone does not replace those persisted values.

6.4 Startup process issues

The startup sequence is start-openclaw.sh (STEPS array, start-openclaw.sh:69): init → onboard → plugins → config-patch → config-validate → tunnel → gateway. Watch it live via:

curl "https://<worker>/api/health"        # status: starting while script runs
curl "https://<worker>/debug/gateway-logs" | tail -n 50
SymptomCauseAction
Stuck at config-validate → startup abortsinvalid generated configmoltlazy validate locally; fix feature flags
Stuck at onboard > 60sAI Gateway plugin not presentthe cloudflare-ai-gateway provider is pre-installed in the image (so its plugin phase is a no-op); if it still times out, check ClawHub/npm reachability — onboard is capped at 60s and continues, gateway starts --allow-unconfigured
Gateway never becomes ready (readiness gate times out)enabled-but-missing provider pluginthe provider plugins are pre-installed in the image, so this should be rare; start-openclaw.sh falls back to installing the latest ClawHub spec when the extension state is missing (look for Installing … from ClawHub in /tmp/startup.log). The enablement config patch must not carry plugins.bundledDiscovery (removed in 2026.9.x — including it fails validation and drops the whole patch)
Gateway dies before port opensgateway binary crashGET /debug/gateway-logs for the crash; check EADDRINUSE. If the log shows SQLite integrity_check failed for …/state/openclaw.sqlite, run POST /debug/sqlite-integrity (then poll its job id) to confirm which database is corrupt before restoring an older verified backup (do not run openclaw doctor --fix on the only copy)
STARTUP_TIMEOUT_MS (300s) exceededfallback plugin installs slowplugins are pre-installed in the image so the normal boot skips installs; if it stalls, the fallback is doing network installs — check ClawHub reachability
Loading page never advancesstartup-progress.json not writtencheck /tmp/startup.log via debug/gateway-logs

The loading page renders from /tmp/startup-progress.json + /tmp/startup.log (routes/public.ts:131, read bounded by STARTUP_PROGRESS_TIMEOUT_MS).


8. Cloudflare Platform Setup

One-time operator setup tasks for the current FourIA Worker deployment.

8.1 Cloudflare Access (protects /_admin, /api, /debug)

  1. Enable Access on the worker — Workers & Pages dashboard → your Worker → Settings → Domains & Routes → workers.dev → Enable Cloudflare Access. Copy the Application Audience (AUD) tag.

  2. Configure who can access — Zero Trust → Access → Applications → your worker app → add allowed identities (email, Google, GitHub, …).

  3. Set the secrets:

    npx wrangler secret put CF_ACCESS_TEAM_DOMAIN   # e.g. myteam.cloudflareaccess.com
    npx wrangler secret put CF_ACCESS_PLATFORM_AUD  # platform AUD (always required)
    npx wrangler secret put CF_ACCESS_CLIENT_AUD    # optional per-tenant AUD

    Alternatively create a Self-hosted Access application and protect paths /_admin/*, /api/*, /debug/*.

/debug/* must be Access-protected. Never expose it publicly — it can stop, destroy, and rescue containers.

8.2 Browser Automation (CDP)

A Chrome DevTools Protocol shim for browser automation. Set:

npx wrangler secret put CDP_SECRET   # shared secret for CDP auth
npx wrangler secret put WORKER_URL   # worker public URL
npm run deploy
EndpointDescription
GET /cdp/json/versionBrowser version information
GET /cdp/json/listList available browser targets
GET /cdp/json/newCreate a new browser target
WS /cdp/devtools/browser/{id}WebSocket connection for CDP commands

All endpoints require ?secret=<CDP_SECRET>.

8.3 Obsidian Sync via Cloudflare Tunnel

Exposes the Obsidian LiveSync-compatible service (port 18790). Requires an active Cloudflare zone for your domain.

cloudflared tunnel create obsidian-sync
cloudflared tunnel token <UUID>          # then:
npx wrangler secret put CLOUDFLARE_TUNNEL_TOKEN

Add a proxied CNAME: name obsidian-sync, target <UUID>.cfargotunnel.com. Ingress (~/.cloudflared/config.yml): hostname: obsidian-sync.yourdomain.com → service: http://localhost:18790. The gateway only starts the tunnel when CLOUDFLARE_TUNNEL_TOKEN is set (start-openclaw.sh:497); otherwise obsidian-sync is localhost-only.

8.4 Built-in skills

The container ships pre-installed skills in /root/clawd/skills/, including cloudflare-browser (browser automation via the CDP shim; needs CDP_SECRET + WORKER_URL). Scripts: screenshot.js, video.js, cdp-client.js.


9. Requirements & References

Requirements

  • Workers Paid plan ($5/mo) — required for Cloudflare Sandbox containers.
  • An AI provider key via Cloudflare AI Gateway (API key, account id, gateway id) — the only supported AI path. Unified Billing is available as an alternative (docs).

Free tiers exist for Cloudflare Access, Browser Rendering, AI Gateway, and R2 Storage.

See also


7. Quick Reference

Environment variables (operator-relevant)

VariablePurposeNotes
DEBUG_ROUTESEnables /debug/*Set true only on operator/ROOT deploys
CLOUDFLARE_API_TOKENAI Gateway REST auth; Containers:Edit for rescueAI-scoped tenant alias; rescue token on ROOT/ops
CONTAINER_APPLICATION_IDTarget app for image rolloutOperator-only
CONTAINER_IMAGE_TAG / CONTAINER_IMAGE_NAMEExpected image unstable-<sha> / fouriaOperator-only
CLOUDFLARE_AI_GATEWAY_API_KEY / CF_AI_GATEWAY_ACCOUNT_ID / CF_AI_GATEWAY_GATEWAY_IDAI Gateway authRequired for AI provider
BACKUP_ENCRYPTION_KEY / BACKUP_ENCRYPTION_REQUIREDR2 encryption at restRecommended true in prod
BACKUP_BUCKET_NAME / R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEYR2 persistenceAll four (incl. CLOUDFLARE_ACCOUNT_ID) required for scheduled backups; SDK signs R2 presigned URLs
BACKUP_INTERVAL_MINUTES / BACKUP_MAX_VERSIONSBackup cadence floor (auto-stretched to 2× the last snapshot duration) / retentionDefaults 5 / 3
“not encrypted” badge on a backupThe archive is not encrypted at rest (a legacy archive, or one whose encryption step never ran)The next backup repairs it in place; it is never promoted to the recovery point
SANDBOX_INSTANCE_TIMEOUT_MS / SANDBOX_PORT_TIMEOUT_MS / SANDBOX_POLL_INTERVAL_MSContainer fail-fast boundsSee §6.1
CLOUDFLARE_ACCOUNT_IDCloudflare account id (plain var)Read by the Sandbox SDK for R2 presigned URL signing; legacy CF_ACCOUNT_ID is deprecated
CF_ACCESS_TEAM_DOMAIN / CF_ACCESS_PLATFORM_AUD / CF_ACCESS_CLIENT_AUDCloudflare Access authProtects /_admin, /api, /debug
CDP_SECRET / WORKER_URLBrowser automation (CDP)See §8.2
CLOUDFLARE_TUNNEL_TOKENObsidian sync ingressSee §8.3
STARTUP_LOG_VERBOSITYsteps/detailed/verbose boot logSecret-free

One-line recovery cheatsheet

# 1. Is it up? (no warm-up)
curl "https://<worker>/api/health?passive=1" | jq '{status,degraded,containerError}'

# 2. What's the lock state?
curl "https://<worker>/debug/gateway-lock" | jq .state

# 3. Degraded (no such image)? Rescue (dry first):
curl -X POST "https://<worker>/debug/rescue-container" -d '{"dryRun":true}' | jq .
curl -X POST "https://<worker>/debug/rescue-container" -d '{"dryRun":false}' | jq .

# 4. Gateway wedged but config OK? Stop, wait 8s, restart (via Admin API).
# 5. Lost state? Inspect backups, then restore by id or let next boot auto-restore.

This guide is generated from the source in apps/fouria/src/diagnostics.ts and its dependencies (gateway/process.ts, gateway/health-probes.ts, gateway/lock.ts, persistence.ts). When the behavior of those modules changes, update this document in the same change.