FourIA — Worker Dashboard Architecture

On this page

Architecture Overview
Frontend Architecture (Vite + React SPA)
- Build & Dev Flow
- Vite Configuration
- Client Module Structure
- Frontend ↔ Backend Communication
Backend Architecture (Hono Worker)
- Request Pipeline
- API Route Structure
Container Startup Flow (start-openclaw.sh)
- Startup Progress (/api/startup-progress)
- moltlazy patch — Config Injection
- Why not packages/moltlazy/sdk?
RBAC & Access Control
- Roles
- Dashboard Pages by Role
- Auth Flow
Development Tips
- Debugging the Container
- Understanding Boot Logs
- Service Integrations (Cloudflare Tunnel)
- CRON and Scheduled Tasks
- Backup and Hard-Restart Policy
- Sandbox Recovery (stale + orphaned)
- Activity-Driven Wake (ActivityScheduler)
- Health & Status Endpoints
- Log Levels
TypeScript & Tooling
- Type Generation
- Testing
- Formatting & Linting
Common Tasks
- Adding a New API Endpoint
- Adding a New Environment Variable

Deep-dive into the apps/fouria component, the core Tenant Worker of FourIA.

See ../docs/DEVELOPMENT.md for project-level overview and ../docs/ENV-VARIABLES.md for environment variable reference.

Architecture Overview

Browser
   │
   ▼
┌─────────────────────────────────────────┐
│       Cloudflare Worker (index.ts)       │
│                                          │
│  Middleware: Logging → Auth → RBAC       │
│                                          │
│  Routes:                                 │
│    /api/*     → Hono API routes          │
│    /_admin/*  → React SPA (Vite build)   │
│    /debug/*   → Debug endpoints (ROOT)   │
│    /*         → Proxy to OpenClaw GW     │
│                                          │
│  Activation: Worker restore → onboard    │
│              → patch → launch gateway    │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│   Cloudflare Sandbox Container           │
│                                          │
│  start-openclaw.sh:                      │
│    1. openclaw onboard (auth setup)      │
│    2. Enable plugins (unified billing)   │
│    3. moltlazy patch (config injection)  │
│    4. Start Cloudflare Tunnel (optional) │
│    5. background gateway + health poll   │
│                                          │
│  ┌────────────────────────────────────┐  │
│  │     OpenClaw Gateway               │  │
│  │  - Control UI on port 18789        │  │
│  │  - WebSocket RPC protocol          │  │
│  │  - Agent runtime                   │  │
│  └────────────────────────────────────┘  │
└─────────────────────────────────────────┘

Frontend Architecture (Vite + React SPA)

The dashboard is a React 19 SPA built with Vite 6 and served from /_admin/.

Build & Dev Flow

bun run dev           → Vite dev server on :8787
    │                     Hot-reload for frontend changes
    │                     Proxies /api/*, /debug/* to :8788 (worker)
    │
bun run dev:worker    → wrangler dev on :8788
    │                     Runs actual Worker code
    │                     Backend changes require restart
    │
bun start             → bun run build:docker && wrangler dev
                          Full local stack (no hot-reload)

Key insight: bun run dev only handles the Vite dev server for the frontend. Backend changes in src/routes/, src/auth/, src/gateway/, etc., require re-running bun run dev:worker.

Vite Configuration

// vite.config.ts — key settings
base: '/_admin/'       // All assets served under /_admin/
server: { port: 8787 }  // Dev server port
proxy: {
  '/api':    'http://localhost:8788',
  '/debug':  'http://localhost:8788',
  '/cdp':    'http://localhost:8788',
}

Client Module Structure

src/client/
├── App.tsx              # Root component with React Router
├── main.tsx             # Entry point — mounts React app
├── api.ts               # Typed API client → /api/admin/*
├── pages/               # Route-level pages
│   ├── HomePage.tsx       # Dashboard overview
│   ├── AgentsPage.tsx     # Agent management
│   ├── ChannelsPage.tsx   # Connections/integrations
│   ├── KnowledgePage.tsx  # Knowledge base/wiki
│   ├── AccountPage.tsx    # User/team management
│   ├── IntegrationsPage.tsx   # Integration config
│   ├── SecretsPage.tsx    # Vault management
│   ├── BackupsPage.tsx    # Backup management
│   ├── DiagnosticDashboard.tsx  # Costs + diagnostics
│   ├── AuditLogsPage.tsx  # Audit log viewer
│   └── AutomationEditorPage.tsx # Automation workflows
├── components/          # Shared UI components
└── hooks/               # Custom React hooks

Frontend ↔ Backend Communication

The frontend (src/client/api.ts) calls /api/admin/* endpoints. These are served by the Worker’s Hono routes (src/routes/api.ts), which:

Verify RBAC (requireWriteRole() for mutations)
Execute OpenClaw CLI commands inside the Sandbox container
Parse CLI output and return JSON

Backend Architecture (Hono Worker)

Request Pipeline

Incoming Request
    │
    ├─► Logging middleware (all routes)
    ├─► Sandbox init (getSandbox() into context)
    │
    ├─► Public routes (no auth):
    │     /sandbox-health, /api/otel/*
    │
    ├─► CF Access Auth middleware:
    │     Extract JWT → validate → set accessUser
    │
    ├─► RBAC resolution:
    │     resolveCustomerRole → set userRole, customerSlug
    │
    ├─► Route dispatch:
    │     /api/*     → API routes (auth + RBAC)
    │     /_admin/*  → Admin UI (SPA assets)
    │     /debug/*   → Debug routes (ROOT only)
    │     /*         → Catch-all proxy to gateway
    │
    └─► Gateway proxy:
          HTTP proxy → containerFetch(port 18789)
          WS proxy   → WebSocket relay with token injection

API Route Structure

// src/routes/api.ts — route aggregation
api.route('/admin', adminApi)
  ├── /whoami, /version, /logs, /gateway-api, /debug-status, /gateway-token  (misc)
  ├── /devices/*      → devicesRoutes
  ├── /storage/*      → storageRoutes
  ├── /gateway/*      → gatewayRoutes
  ├── /config/*       → configRoutes
  ├── /cron/*         → cronRoutes
  ├── /dreaming/*     → dreamingRoutes
  ├── /users/*        → usersRoutes
  ├── /integrations/* → integrationsRoutes
  ├── /knowledge/*    → knowledgeRoutes
  ├── /secrets/*      → secretsRoutes
  ├── /agents/*       → agentsRoutes
  ├── /models/*       → modelsRoutes
  ├── /channels/*     → channelsRoutes
  ├── /costs/*        → costsRoutes
  ├── /vault/*        → vaultRoutes
  └── /audit-logs/*   → auditLogsRoutes

// Write gate: POST/PUT/DELETE/PATCH require ADMIN or ROOT
// Read gate: GET/HEAD available to all authenticated roles

Container Startup Flow (`start-openclaw.sh`)

The container entrypoint (start-openclaw.sh) runs sequentially:

1. Fast-path check: Is gateway already running? → exit 0

2. Worker-side lifecycle coordination
   - Restore, snapshot, restart, and destroy are coordinated by the Worker.
   - The script itself has no `flock`; OpenClaw's native lock only protects
     gateway ownership, not onboarding/config/plugin filesystem mutations.

3. Pre-backup optimization:
   - Move logs to /tmp (excluded from squashfs snapshots)
   - Clean stale backup files

4. Onboard (if needed):
    - Detect Cloudflare AI Gateway auth from env vars
    - If AI Gateway auth is configured, preinstall the `cloudflare-ai-gateway`
      provider plugin (pinned, `--accept-capabilities`) **before** onboard —
      non-interactive onboarding cannot approve plugin capabilities, so without
      the preinstall its plugin phase fails and burns the 60s onboard cap on
      every cold start. With the plugin present, the phase is a no-op and
      onboarding completes cleanly.
    - Run: openclaw onboard --non-interactive --accept-risk

5. Enable plugins:
   - cloudflare-unified-billing (always) — entry opts into
     `hooks.allowConversationAccess=true` so the `agent_end`/`message_received`
     hooks run (OpenClaw 2026.8.2 blocks non-bundled conversation hooks without it)
   - cloudflare-ai-gateway (idempotent ensure — skips if already installed)
   - diagnostics-otel (conditional on OTEL_EXPORTER_OTLP_METRICS_ENDPOINT)

6. Patch config via moltlazy:
   - Run: node /app/moltlazy/dist/cli.js patch
   - Generates per-section includes (moltlazy-session/tools/logging.json) injected
     via per-section $include; gateway/agents written inline (no root $include)
   - References the gateway token as an env SecretRef (kept in the container environment)
   - Run: node /app/moltlazy/dist/cli.js validate
   - **Validation is FATAL**: an invalid config would prevent the gateway from
     booting, so the script aborts (exit 1) instead of proceeding.

7. Harden file permissions:
   - chmod 600 moltlazy-*.json, chmod 700 .openclaw

8. Cloudflare Tunnel (optional):
   - Start cloudflared for obsidian-sync (port 18790)

9. Start gateway:
    - background `openclaw gateway run --port 18789 --verbose --allow-unconfigured --bind lan`
    - poll `/health`, then exit while the gateway child remains detached

Startup Progress (`/api/startup-progress`)

The loading page shows a transparent boot stepper + live log while the gateway starts. start-openclaw.sh declares its boot phases in a single STEPS array (single source of truth) and emits progress to two /tmp files on each phase (kept in /tmp so they stay out of squashfs backups and R2):

File	Content
`/tmp/startup-progress.json`	`{ steps:[{id,label}], currentStep, currentStepId, totalSteps, skipped, ts }`
`/tmp/startup.log`	Clean, secret-free, chronological boot detail (verbosity-gated)

mark_step <id> [skipped] resolves the label/index from STEPS, writes the progress JSON atomically (mktemp + mv), and echoes the transition to the container stdout. log_boot <level> <msg> writes clean boot detail to /tmp/startup.log. Step transitions are intentionally not written to the log — the loading page’s stepper shows step progress, so the log stays clean (no duplicated step/checked info). Adding a phase = one STEPS row + one mark_step call — the loading page renders the list dynamically, so no other code changes.

GET /api/startup-progress (public, no warm-up) reads both files and returns { available, steps, currentStep, currentStepId, totalSteps, skipped, ts, log }. It is bounded by STARTUP_PROGRESS_TIMEOUT_MS and never calls restoreIfNeeded/ensureGateway, so it adds no load to the health path.

The loading page consumes this data over the GET /api/startup/ws WebSocket (not by polling). Each push is a combined { ...HealthSnapshot, progress } payload — the health snapshot (which drives readiness/redirect) plus the boot steps + log. The endpoint is non-passive: it warms/starts the gateway on every push, so the loading page itself drives startup over one push channel instead of hammering /api/health + /api/startup-progress with short-interval HTTP polls.

The loading page shows the boot log panel only while the gateway is being (re-)provisioned — i.e. while start-openclaw.sh is running (or about to run). Once the gateway binary is already running and the page is merely waiting for it to become ready (processType === 'gateway' in the /api/health snapshot), it hides the boot log and shows the FourIA logo instead.

The loading page uses a single push channel — GET /api/startup/ws — and does not poll while the WebSocket is open. Flow:

Handshake — the first WS push reports whether the gateway is already running; if so the page redirects immediately, otherwise the stream keeps warming and pushing step updates.
Boot phase — while boot steps are in progress, the sandbox pushes the health snapshot + progress (steps + log) every STARTUP_WS_POLL_INTERVAL_MS (~5s). No client polling.
Fallback — if the WebSocket drops (e.g. local wrangler dev, a network blip), the page falls back to slow HTTP polling (/api/health every 5s) until the overall wall-clock budget (MAX_WAIT_MS, 8 minutes) is exhausted.

The page only declares failure on a permanent container error (degraded: true in the /api/health snapshot — deleted image, missing app, where a rescue is required), surfaced immediately via the container error. Every other state — still booting, gateway reachable but warming up, restore in progress — keeps waiting until the wall-clock budget is exhausted, because a cold start legitimately takes 3–5+ minutes (onboard + plugin installs + gateway warmup). Mixed probe states (e.g. startup.ok lagging behind readiness.ok while a backup restore is pending) are not treated as failures; only the budget timeout triggers the doctor diagnostics message.

The loading page treats the canonical gateway status (status === 'running', driven by the readiness probe) as “ready” rather than trusting startup.ok alone, because startup.ok is gated on backup-restore state that can lag a healthy, already-serving gateway.

Security invariant: the boot log only contains curated log_boot lines — the script’s raw stdout (which can contain the AI Gateway API key) must never be written to /tmp/startup.log.

Verbosity is controlled by STARTUP_LOG_VERBOSITY (steps | detailed (default) | verbose), passed through from the Worker via buildEnvVars().

`moltlazy patch` — Config Injection

moltlazy patch is the mandatory config system. It runs every startup and generates:

Config Section	Type	Content
`gateway.*`	Mandatory	Port, mode, bind, auth
`session.*`	Mandatory	Session scope, DM settings
`tools.toolSearch`	Mandatory	Tool search configuration
`logging.*`	Mandatory	Log levels, format, destinations
`agents.defaults`	Always-on	Default model, system prompt
`agents.entries`	Always-on	Premade agents keyed by id (dev-only `fouria-builder`, `fouria-qa`)
`skills.entries`	Always-on	Built-in skills
`memory.*`	Feature-gated	Memory backends (knowledge graph)
`channels.*`	Env-gated	Channel tokens (Telegram, Discord, etc.)
`plugins.*`	Feature-gated	Plugin management config

The moltlazy-owned sections (session/tools/logging) are generated into per-section include files and injected into openclaw.json via per-section $include directives — not a root $include (root includes fail closed for Control UI saves / config.apply). They are regenerated from scratch every startup — never edit manually. gateway.* and agents.* are written inline into openclaw.json; the gateway token is an env SecretRef (resolved from the container environment, which start-openclaw.sh keeps set).

Why not `packages/moltlazy/sdk`?

The SDK module (sdk/) provides a typed RPC client over the OpenClaw Admin HTTP API. It is not integrated in the Worker runtime for v1. The Worker instead uses CLI calls (sandbox.startProcess('openclaw ... )) to interact with the gateway. The SDK is planned for future integration when the Worker can directly call the Admin API via GATEWAY_RPC DO.

RBAC & Access Control

Roles

Role	Permissions	Access Scope
ROOT	`admin:platform`, `admin:customer`, `api:all`	All tenants, all debug endpoints
ADMIN	`admin:customer`, `api:all`	Own tenant only
BASE_USER	`api:own`	Own tenant, read-only

Dashboard Pages by Role

Page	ROOT	ADMIN	BASE_USER
Dashboard Home	Yes	Yes	Yes
Agents	Yes	Yes	Read-only
Connections	Yes	Yes	Read-only
Knowledge	Yes	Yes	Read-only
Account/Team	Yes	Yes	Read-only
Integrations	Yes	Yes	No
Secrets Vault	Yes	Yes	No
Backups	Yes	Yes	No
Diagnostics	Yes	No	No
Debug (`/debug/*`)	Yes	No	No
Audit Logs	Yes	No	No

ROOT is the maintainers’ intended role. It has full access to debugging tools, container inspection, and platform-level diagnostics at /_admin/diagnostic and /debug/*. ROOT is restricted to diagnostics-only API paths (DIAGNOSTICS_PATHS in src/routes/api.ts) — management pages (Agents, Connections, Backups, etc.) are ADMIN-gated. As a diagnostic exception, ROOT can still verify backup connectivity (Test Backup Connectivity on the Diagnostic Dashboard → POST /api/admin/storage/test-connectivity) without the Backups page.

In a harness/dev mode (DEV_MODE or E2E_TEST_MODE), /debug/* additionally permits ADMIN so the automated E2E suite can drive CLI/operator routes from a bare workers.dev worker that has no Cloudflare Access or Dispatch Worker in front of it. Real production (DEV_MODE=false, E2E_TEST_MODE=false) stays ROOT-only per the table above.

Auth Flow

Cloudflare Access intercepts request → user authenticates (email OTP, Google, GitHub)
Access injects Cf-Access-Jwt-Assertion header
Worker verifies JWT against Cloudflare Access JWKS endpoint
resolveCustomerRole() resolves the RBAC role — Paso4 ROOT users (platform Access AUD) are always ROOT; client users get their lerma-assigned role (ADMIN/BASE_USER) forwarded by the Dispatch Worker via X-FourIA-Role
userRole and customerSlug are set in Hono context
Route handlers check role before serving content

Development Tips

Debugging the Container

# Get container ID from Worker logs
docker ps | grep sandbox

# Enter container shell
docker exec -it <container_id> bash

# Inside container: check openclaw status
openclaw status
openclaw devices list --json
openclaw config get gateway.port

# Check logs
tail -f /tmp/openclaw-logs/*.log

Understanding Boot Logs

During cold start, these log patterns are normal:

[Gateway] listProcesses returned 0 processes
[Gateway] Starting new OpenClaw gateway...
[ERROR] Uncaught ProcessReadyTimeoutError: Waiting for: port 18789 (TCP)

The gateway takes up to 3 minutes to become ready. The Worker retries automatically. If errors persist beyond 3 minutes:

Check docker logs <container_id> for OpenClaw errors
Verify CF AI Gateway config (CLOUDFLARE_AI_GATEWAY_API_KEY + CLOUDFLARE_API_TOKEN + CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID)
Run bun run dev:worker to see full Worker output

Service Integrations (Cloudflare Tunnel)

The container can run Cloudflare Tunnels for exposing internal services:

# Obsidian sync on port 18790
# Requires CLOUDFLARE_TUNNEL_TOKEN secret
# start-openclaw.sh auto-starts cloudflared if token is set

CRON and Scheduled Tasks

The Worker no longer uses a cron trigger. Containers are kept awake for non-frozen tenants by the lerma HealthMonitor, which probes the GET /api/health endpoint every few minutes (authenticated at the Cloudflare edge with a service token — see the path-scoped public-health Access applications in apps/fouria/iac/access.tf). The endpoint cold-starts the gateway via ensureGateway when it is not running, and also schedules interval-gated R2 backups in the background. This replaced the old */15 * * * * wake/cron trigger.

The container runs scheduled tasks via OpenClaw’s cron system.

Backup and Hard-Restart Policy

Backups are snapshots of /home/openclaw, not a live filesystem journal. Agent workspaces are therefore kept under /home/openclaw/workspace; files outside that tree are not protected by the Sandbox snapshot.

The Sandbox SDK’s createBackup returns a handle marked with localBucket: true when DEV_MODE selects the local R2 binding (persistence.ts). That flag is persisted in backup-handle.json and forwarded to restoreBackup(), so a local backup is always restored through the local binding — never through the remote presigned-URL path, which 404s for archives that exist only in the local bucket. Legacy catalog entries without the flag default to the current DEV_MODE, and a presigned-URL 404 during restore is treated as a gone backup and pruned from the catalog.

Automatic snapshots are interval-gated from non-passive GET /api/health calls. The explicit manual checkpoint is POST /api/admin/storage/sync. Soft reload (POST /api/admin/gateway/restart) does not snapshot.

POST /api/admin/storage/test-connectivity (ADMIN/ROOT) is a read-only diagnostic that verifies the R2 backup bucket binding is reachable by HEADing the backup catalog handle — a 404 (no handle yet) still counts as connected. It is whitelisted for ROOT through the diagnostics gate (DIAGNOSTICS_PATHS in src/routes/api.ts) so platform operators can test backup connectivity from the Diagnostic Dashboard without explicit access to the ADMIN-gated Backups page.

Hard restart (POST /debug/restart-openclaw) checkpoints before killing the gateway. If the checkpoint fails, the gateway is left running. After a successful checkpoint, the next activation restores the committed snapshot.

flowchart TD
  HM["Lerma HealthMonitor<br/>active /api/health"] --> S["scheduleBackupIfNeeded"]
  MAN["POST /api/admin/storage/sync"] --> C["createSnapshot"]
  S --> G{"credentials + interval + lock"}
  G -- no --> N["no-op"]
  G -- yes --> C
  C --> A["Sandbox snapshot<br/>/home/openclaw"]
  A --> E["optional F4E1 encryption"]
  E --> P["commit backup-handle.json"]
  HR["POST /debug/restart-openclaw"] --> CP["checkpoint first"]
  CP --> K["graceful kill + restore marker"]
  K --> R["restore committed snapshot<br/>on next activation"]
  SR["POST /api/admin/gateway/restart"] --> SOFT["SIGUSR1 in-process reload<br/>no snapshot"]

Do not perform repeated hard restarts without verifying a successful checkpoint. The checkpoint protects only files inside /home/openclaw and only after the backup catalog has been committed.

Sandbox Recovery (stale + orphaned)

When the platform terminates a container behind a live Sandbox DO (idle recycle, maintenance, SIGTERM), every RPC into it fails with the SDK’s HTTP error! status: 500 wrapper — and retrying against the same instance fails forever, bricking the tenant. gateway/sandbox-recovery.ts classifies these dead-container errors (isStaleContainerError) and recovers (recoverStaleSandbox): destroy the instance, reset the startup-probe and persistence caches. The next RPC auto-provisions a fresh container and the normal cold-start flow runs (R2 restore → start-openclaw.sh → gateway).

A container destroyed out-of-band (Cloudflare Containers dashboard “Destroy”, failed operator force-stop) is a second failure class: the DO is alive but points at a placement that no longer exists, so RPCs fail with the SDK’s retryable admission errors (no container instance available, maximum number of running container instances exceeded) instead of the 500 signature. Those are legitimate during a cold start, so they are not treated as stale on their own. isContainerUnavailableError + trackProvisioningFailure escalate them to a recovery once the streak reaches ORPHANED_INSTANCE_MIN_FAILURES and spans ORPHANED_INSTANCE_GRACE_MS (120s — longer than the SDK’s ≈110s provisioning budget, so a slow-but-healthy cold start is never cancelled mid-provision). The streak is cleared whenever a sandbox RPC succeeds. maybeRecoverSandbox unifies both classes for callers.

Recovery is wired into three failure points in gateway/process.ts: findExistingGatewayProcess (listProcesses failures), checkGatewayHealth (exec failures), and the ensureGateway startProcess path (recover + retry once). It is cooldown-guarded (STALE_RECOVERY_COOLDOWN_MS, 60s) and never fires for deploy-window OperationInterruptedError. Permanent image errors still require the rescue tool (rescueContainer, see OPERATOR-GUIDE §5.1). Logs use the [SandboxRecovery] prefix.

Transient transport drops (retry, don’t reprovision)

A third failure signature is not a dead container: Network connection lost. (and similar socket/WebSocket transport messages). It appears when the DO↔container connection drops while the container is alive — most visibly while the Gateway is applying a config/plugin reload (e.g. adding a model provider in the Control UI), which restarts the Gateway and briefly tears down the container RPC/WS transports.

isContainerTransportError (gateway/sandbox-recovery.ts) classifies these, and isStaleContainerError explicitly excludes them (even when the SDK wraps the message in its sandbox.<op> error prefix) so a transient drop never destroys a live instance. ensureGateway instead retries with a short backoff: the port probe (isGatewayPortOpenResilient, up to TRANSPORT_RETRY_ATTEMPTS) and the startProcess spawn (once). Before this, a reload that dropped the transport mid probe fell straight through to start-openclaw.sh → Failed to start process → a 502 to the user, even though the Gateway came back seconds later. The retry keeps the request alive across the reload window.

The manual lever is POST /debug/reprovision (DEBUG_ROUTES, ROOT): it destroys the instance, clears the startup/persistence caches, and resets the orphan streak without ever contacting the gateway — so it works when the container is entirely gone. Compare POST /debug/restart-openclaw (gateway-aware, checkpointed) and POST /debug/destroy-container (teardown only, used by E2E).

Activity-Driven Wake (`ActivityScheduler`)

Automations (“jobs”) run inside the Gateway, so the Gateway must be running for a schedule to fire. Historically a Worker cron trigger woke every tenant on every tick (expensive and since removed). src/activity/ replaces it with a per-tenant activity clock:

ActivityScheduler Durable Object (SQLite-backed, binding ACTIVITY) — stores a mirror of the tenant’s automation schedule and owns a single alarm. Deliberately a separate DO from the SDK Sandbox DO, which already uses its own alarm for sleepAfter.
Mirror — warm paths (GET /api/health, only when readiness.ok) read openclaw automations list --json and push the earliest nextRunAtMs into the DO. The DO throttles this to once per 5 min, and it never wakes a sleeping container.
Alarm (activity/alarm.ts) — fires CRON_WAKE_AHEAD_MINUTES (default 10) before the next job, wakes + heals via ensureGateway (which recovers an orphaned instance — see above), then re-reads the schedule and re-arms. No enabled jobs → alarm cleared → the container sleeps. A due-but-pending job is woken once per occurrence and re-checked on a bounded retry, so it can never spin.

Net: healing/wakeup is independent of user traffic, gated on real job activity, and free when idle. Traffic may cold-start the container; jobs get a pre-warmed, healed one. Requires the ACTIVITY binding in wrangler.jsonc and in the lerma provisioner (tenant_provisioner.ex resource_bindings) + the new_sqlite_classes migration (dispatch_namespace.ex @do_sqlite_classes).

Health & Status Endpoints

GET /api/health is the unified health endpoint: it returns the startup/readiness/liveness probes plus the canonical status (running|starting|stopped), processId and processType. It requires no authentication inside the Worker, but at the Cloudflare edge it is Access-protected by path-scoped applications (*.fouria.io/api/health) that admit Paso4 Root Users (browser) and the lerma health-monitor service token (headless). GET /api/status, GET /api/admin/gateway/status/polling and GET /api/admin/gateway/status/ws are thin aliases over the same snapshot. GET /api/health/ws streams the snapshot every ~2s over WebSocket.

By default /api/health warms the container (lerma keep-alive). ?passive=1 probes without warming — the ROOT /_admin/diagnostics surface uses it so an operator can debug an instance whose container is down. The admin dashboard subscribes to the health WebSocket (with an HTTP polling fallback) and never blocks the UI on gateway status.

Detached gateway detection

The gateway runs as a detached background child of start-openclaw.sh (openclaw gateway run ... &), so after the script exits (exit 0 once the gateway is reachable) the gateway binary is invisible to listProcesses(). To avoid false no_process reports on log/status surfaces, those endpoints (GET /api/admin/logs, GET /debug/logs, GET /debug/stop-gateway, POST /api/admin/gateway/restart, GET /api/admin/audit-logs) call findExistingGatewayProcess(sandbox, { probeLive: true }), which falls back to a bounded /health probe and — when the gateway responds — returns a synthetic gateway-live process whose logs are read from /tmp/gateway-stdout.log. The passive health snapshot path deliberately does not probe so a down container fails fast.

Log Levels

Level	When to use
`debug`	Local development
`info`	Production default
`warn`	High-traffic production
`error`	Security-sensitive production

TypeScript & Tooling

Type Generation

Worker types are auto-generated:

bun run types   # wrangler types → worker-configuration.d.ts

Never manually edit worker-configuration.d.ts — it’s regenerated from wrangler.jsonc.

Testing

bun run test             # Unit tests (Vitest)
bun run test:coverage    # With coverage report
bun run test:e2e         # E2E tests (cctr + Playwright)
bun run test:e2e:cli     # CLI-only E2E (cctr)

Formatting & Linting

bun run format          # oxfmt (auto-format)
bun run format:check    # Check formatting
bun run lint            # oxlint
bun run typecheck       # tsc --noEmit

Common Tasks

Adding a New API Endpoint

Create route handler in src/routes/api/<domain>.ts
Register in src/routes/api.ts
Add client method in src/client/api.ts
Write tests (unit + integration)
If new dependencies: update package.json

Adding a New Environment Variable

Add to OpenClawEnv interface in src/types.ts
If passed to container: add to buildEnvVars() in src/gateway/env.ts
Update .dev.vars.example
Document in ../../docs/ENV-VARIABLES.md