# FourIA — Worker Dashboard Architecture Deep-dive into the `apps/fouria` component, the core Tenant Worker of FourIA. > See [../docs/DEVELOPMENT.md](../../docs/DEVELOPMENT.md) for project-level overview and [../docs/ENV-VARIABLES.md](../../docs/ENV-VARIABLES.md) for environment variable reference. --- ## Architecture Overview ``` Browser │ ▼ ┌─────────────────────────────────────────┐ │ Cloudflare Worker (index.ts) │ │ │ │ Middleware: Logging → Auth → RBAC │ │ │ │ Routes: │ │ /api/* → Hono API routes │ │ /_admin/* → React SPA (Vite build) │ │ /debug/* → Debug endpoints (ROOT) │ │ /* → Proxy to OpenClaw GW │ │ │ │ Activation: Worker restore → onboard │ │ → patch → launch gateway │ └──────────────┬──────────────────────────┘ │ ▼ ┌─────────────────────────────────────────┐ │ Cloudflare Sandbox Container │ │ │ │ start-openclaw.sh: │ │ 1. openclaw onboard (auth setup) │ │ 2. Enable plugins (unified billing) │ │ 3. moltlazy patch (config injection) │ │ 4. Start Cloudflare Tunnel (optional) │ │ 5. background gateway + health poll │ │ │ │ ┌────────────────────────────────────┐ │ │ │ OpenClaw Gateway │ │ │ │ - Control UI on port 18789 │ │ │ │ - WebSocket RPC protocol │ │ │ │ - Agent runtime │ │ │ └────────────────────────────────────┘ │ └─────────────────────────────────────────┘ ``` --- ## Frontend Architecture (Vite + React SPA) The dashboard is a React 19 SPA built with Vite 6 and served from `/_admin/`. ### Build & Dev Flow ``` bun run dev → Vite dev server on :8787 │ Hot-reload for frontend changes │ Proxies /api/*, /debug/* to :8788 (worker) │ bun run dev:worker → wrangler dev on :8788 │ Runs actual Worker code │ Backend changes require restart │ bun start → bun run build:docker && wrangler dev Full local stack (no hot-reload) ``` > **Key insight:** `bun run dev` only handles the Vite dev server for the frontend. Backend changes in `src/routes/`, `src/auth/`, `src/gateway/`, etc., require re-running `bun run dev:worker`. ### Vite Configuration ```typescript // vite.config.ts — key settings base: '/_admin/' // All assets served under /_admin/ server: { port: 8787 } // Dev server port proxy: { '/api': 'http://localhost:8788', '/debug': 'http://localhost:8788', '/cdp': 'http://localhost:8788', } ``` ### Client Module Structure ``` src/client/ ├── App.tsx # Root component with React Router ├── main.tsx # Entry point — mounts React app ├── api.ts # Typed API client → /api/admin/* ├── pages/ # Route-level pages │ ├── HomePage.tsx # Dashboard overview │ ├── AgentsPage.tsx # Agent management │ ├── ChannelsPage.tsx # Connections/integrations │ ├── KnowledgePage.tsx # Knowledge base/wiki │ ├── AccountPage.tsx # User/team management │ ├── IntegrationsPage.tsx # Integration config │ ├── SecretsPage.tsx # Vault management │ ├── BackupsPage.tsx # Backup management │ ├── DiagnosticDashboard.tsx # Costs + diagnostics │ ├── AuditLogsPage.tsx # Audit log viewer │ └── AutomationEditorPage.tsx # Automation workflows ├── components/ # Shared UI components └── hooks/ # Custom React hooks ``` ### Frontend ↔ Backend Communication The frontend (`src/client/api.ts`) calls `/api/admin/*` endpoints. These are served by the Worker's Hono routes (`src/routes/api.ts`), which: 1. Verify RBAC (`requireWriteRole()` for mutations) 2. Execute OpenClaw CLI commands inside the Sandbox container 3. Parse CLI output and return JSON --- ## Backend Architecture (Hono Worker) ### Request Pipeline ``` Incoming Request │ ├─► Logging middleware (all routes) ├─► Sandbox init (getSandbox() into context) │ ├─► Public routes (no auth): │ /sandbox-health, /api/otel/* │ ├─► CF Access Auth middleware: │ Extract JWT → validate → set accessUser │ ├─► RBAC resolution: │ resolveCustomerRole → set userRole, customerSlug │ ├─► Route dispatch: │ /api/* → API routes (auth + RBAC) │ /_admin/* → Admin UI (SPA assets) │ /debug/* → Debug routes (ROOT only) │ /* → Catch-all proxy to gateway │ └─► Gateway proxy: HTTP proxy → containerFetch(port 18789) WS proxy → WebSocket relay with token injection ``` ### API Route Structure ```typescript // src/routes/api.ts — route aggregation api.route('/admin', adminApi) ├── /whoami, /version, /logs, /gateway-api, /debug-status, /gateway-token (misc) ├── /devices/* → devicesRoutes ├── /storage/* → storageRoutes ├── /gateway/* → gatewayRoutes ├── /config/* → configRoutes ├── /cron/* → cronRoutes ├── /dreaming/* → dreamingRoutes ├── /users/* → usersRoutes ├── /integrations/* → integrationsRoutes ├── /knowledge/* → knowledgeRoutes ├── /secrets/* → secretsRoutes ├── /agents/* → agentsRoutes ├── /models/* → modelsRoutes ├── /channels/* → channelsRoutes ├── /costs/* → costsRoutes ├── /vault/* → vaultRoutes └── /audit-logs/* → auditLogsRoutes // Write gate: POST/PUT/DELETE/PATCH require ADMIN or ROOT // Read gate: GET/HEAD available to all authenticated roles ``` --- ## Container Startup Flow (`start-openclaw.sh`) The container entrypoint (`start-openclaw.sh`) runs sequentially: ``` 1. Fast-path check: Is gateway already running? → exit 0 2. Worker-side lifecycle coordination - Restore, snapshot, restart, and destroy are coordinated by the Worker. - The script itself has no `flock`; OpenClaw's native lock only protects gateway ownership, not onboarding/config/plugin filesystem mutations. 3. Pre-backup optimization: - Move logs to /tmp (excluded from squashfs snapshots) - Clean stale backup files 4. Onboard (if needed): - Detect Cloudflare AI Gateway auth from env vars - If AI Gateway auth is configured, preinstall the `cloudflare-ai-gateway` provider plugin (pinned, `--accept-capabilities`) **before** onboard — non-interactive onboarding cannot approve plugin capabilities, so without the preinstall its plugin phase fails and burns the 60s onboard cap on every cold start. With the plugin present, the phase is a no-op and onboarding completes cleanly. - Run: openclaw onboard --non-interactive --accept-risk 5. Enable plugins: - cloudflare-unified-billing (always) — entry opts into `hooks.allowConversationAccess=true` so the `agent_end`/`message_received` hooks run (OpenClaw 2026.8.2 blocks non-bundled conversation hooks without it) - cloudflare-ai-gateway (idempotent ensure — skips if already installed) - diagnostics-otel (conditional on OTEL_EXPORTER_OTLP_METRICS_ENDPOINT) 6. Patch config via moltlazy: - Run: node /app/moltlazy/dist/cli.js patch - Generates per-section includes (moltlazy-session/tools/logging.json) injected via per-section $include; gateway/agents written inline (no root $include) - References the gateway token as an env SecretRef (kept in the container environment) - Run: node /app/moltlazy/dist/cli.js validate - **Validation is FATAL**: an invalid config would prevent the gateway from booting, so the script aborts (exit 1) instead of proceeding. 7. Harden file permissions: - chmod 600 moltlazy-*.json, chmod 700 .openclaw 8. Cloudflare Tunnel (optional): - Start cloudflared for obsidian-sync (port 18790) 9. Start gateway: - background `openclaw gateway run --port 18789 --verbose --allow-unconfigured --bind lan` - poll `/health`, then exit while the gateway child remains detached ``` ### Startup Progress (`/api/startup-progress`) The loading page shows a transparent boot stepper + live log while the gateway starts. `start-openclaw.sh` declares its boot phases in a single `STEPS` array (single source of truth) and emits progress to two `/tmp` files on each phase (kept in `/tmp` so they stay out of squashfs backups and R2): | File | Content | | ---------------------------- | ----------------------------------------------------------------------------- | | `/tmp/startup-progress.json` | `{ steps:[{id,label}], currentStep, currentStepId, totalSteps, skipped, ts }` | | `/tmp/startup.log` | Clean, **secret-free**, chronological boot detail (verbosity-gated) | `mark_step [skipped]` resolves the label/index from `STEPS`, writes the progress JSON atomically (`mktemp` + `mv`), and echoes the transition to the container stdout. `log_boot ` writes clean boot detail to `/tmp/startup.log`. Step transitions are intentionally **not** written to the log — the loading page's stepper shows step progress, so the log stays clean (no duplicated step/checked info). Adding a phase = one `STEPS` row + one `mark_step` call — the loading page renders the list dynamically, so no other code changes. `GET /api/startup-progress` (public, no warm-up) reads both files and returns `{ available, steps, currentStep, currentStepId, totalSteps, skipped, ts, log }`. It is bounded by `STARTUP_PROGRESS_TIMEOUT_MS` and never calls `restoreIfNeeded`/`ensureGateway`, so it adds no load to the health path. The loading page consumes this data over the **`GET /api/startup/ws`** WebSocket (not by polling). Each push is a combined `{ ...HealthSnapshot, progress }` payload — the health snapshot (which drives readiness/redirect) plus the boot steps + log. The endpoint is **non-passive**: it warms/starts the gateway on every push, so the loading page itself drives startup over one push channel instead of hammering `/api/health` + `/api/startup-progress` with short-interval HTTP polls. The loading page shows the boot log panel **only while the gateway is being (re-)provisioned** — i.e. while `start-openclaw.sh` is running (or about to run). Once the gateway binary is already running and the page is merely waiting for it to become ready (`processType === 'gateway'` in the `/api/health` snapshot), it hides the boot log and shows the FourIA logo instead. The loading page uses a **single push channel** — `GET /api/startup/ws` — and does not poll while the WebSocket is open. Flow: 1. **Handshake** — the first WS push reports whether the gateway is already running; if so the page redirects immediately, otherwise the stream keeps warming and pushing step updates. 2. **Boot phase** — while boot steps are in progress, the sandbox pushes the health snapshot + `progress` (steps + log) every `STARTUP_WS_POLL_INTERVAL_MS` (~5s). No client polling. 3. **Fallback** — if the WebSocket drops (e.g. local `wrangler dev`, a network blip), the page falls back to slow HTTP polling (`/api/health` every 5s) until the overall wall-clock budget (`MAX_WAIT_MS`, 8 minutes) is exhausted. The page only declares **failure** on a _permanent_ container error (`degraded: true` in the `/api/health` snapshot — deleted image, missing app, where a rescue is required), surfaced immediately via the container error. Every other state — still booting, gateway reachable but warming up, restore in progress — keeps waiting until the wall-clock budget is exhausted, because a cold start legitimately takes 3–5+ minutes (onboard + plugin installs + gateway warmup). Mixed probe states (e.g. `startup.ok` lagging behind `readiness.ok` while a backup restore is pending) are **not** treated as failures; only the budget timeout triggers the doctor diagnostics message. The loading page treats the canonical gateway status (`status === 'running'`, driven by the readiness probe) as "ready" rather than trusting `startup.ok` alone, because `startup.ok` is gated on backup-restore state that can lag a healthy, already-serving gateway. **Security invariant:** the boot log only contains curated `log_boot` lines — the script's raw stdout (which can contain the AI Gateway API key) must never be written to `/tmp/startup.log`. Verbosity is controlled by `STARTUP_LOG_VERBOSITY` (`steps` | `detailed` (default) | `verbose`), passed through from the Worker via `buildEnvVars()`. ### `moltlazy patch` — Config Injection `moltlazy patch` is the **mandatory config system**. It runs every startup and generates: | Config Section | Type | Content | | ------------------ | ------------- | ------------------------------------------------------------------- | | `gateway.*` | Mandatory | Port, mode, bind, auth | | `session.*` | Mandatory | Session scope, DM settings | | `tools.toolSearch` | Mandatory | Tool search configuration | | `logging.*` | Mandatory | Log levels, format, destinations | | `agents.defaults` | Always-on | Default model, system prompt | | `agents.entries` | Always-on | Premade agents keyed by id (dev-only `fouria-builder`, `fouria-qa`) | | `skills.entries` | Always-on | Built-in skills | | `memory.*` | Feature-gated | Memory backends (knowledge graph) | | `channels.*` | Env-gated | Channel tokens (Telegram, Discord, etc.) | | `plugins.*` | Feature-gated | Plugin management config | The moltlazy-owned sections (`session`/`tools`/`logging`) are generated into per-section include files and injected into `openclaw.json` via per-section `$include` directives — **not a root `$include`** (root includes fail closed for Control UI saves / `config.apply`). They are regenerated from scratch every startup — never edit manually. `gateway.*` and `agents.*` are written inline into `openclaw.json`; the gateway token is an **env SecretRef** (resolved from the container environment, which `start-openclaw.sh` keeps set). ### Why not `packages/moltlazy/sdk`? The SDK module (`sdk/`) provides a typed RPC client over the OpenClaw Admin HTTP API. **It is not integrated in the Worker runtime for v1.** The Worker instead uses CLI calls (`sandbox.startProcess('openclaw ... )`) to interact with the gateway. The SDK is planned for future integration when the Worker can directly call the Admin API via `GATEWAY_RPC` DO. --- ## RBAC & Access Control ### Roles | Role | Permissions | Access Scope | | ------------- | --------------------------------------------- | -------------------------------- | | **ROOT** | `admin:platform`, `admin:customer`, `api:all` | All tenants, all debug endpoints | | **ADMIN** | `admin:customer`, `api:all` | Own tenant only | | **BASE_USER** | `api:own` | Own tenant, read-only | ### Dashboard Pages by Role | Page | ROOT | ADMIN | BASE_USER | | ------------------ | ---- | ----- | --------- | | Dashboard Home | Yes | Yes | Yes | | Agents | Yes | Yes | Read-only | | Connections | Yes | Yes | Read-only | | Knowledge | Yes | Yes | Read-only | | Account/Team | Yes | Yes | Read-only | | Integrations | Yes | Yes | No | | Secrets Vault | Yes | Yes | No | | Backups | Yes | Yes | No | | Diagnostics | Yes | No | No | | Debug (`/debug/*`) | Yes | No | No | | Audit Logs | Yes | No | No | > **ROOT** is the maintainers' intended role. It has full access to debugging tools, container inspection, and platform-level diagnostics at `/_admin/diagnostic` and `/debug/*`. ROOT is restricted to diagnostics-only API paths (`DIAGNOSTICS_PATHS` in `src/routes/api.ts`) — management pages (Agents, Connections, Backups, etc.) are ADMIN-gated. As a diagnostic exception, ROOT can still verify backup connectivity (`Test Backup Connectivity` on the Diagnostic Dashboard → `POST /api/admin/storage/test-connectivity`) without the Backups page. > In a **harness/dev mode** (`DEV_MODE` or `E2E_TEST_MODE`), `/debug/*` additionally permits `ADMIN` so the automated E2E suite can drive CLI/operator routes from a bare `workers.dev` worker that has no Cloudflare Access or Dispatch Worker in front of it. Real production (`DEV_MODE=false`, `E2E_TEST_MODE=false`) stays ROOT-only per the table above. ### Auth Flow 1. Cloudflare Access intercepts request → user authenticates (email OTP, Google, GitHub) 2. Access injects `Cf-Access-Jwt-Assertion` header 3. Worker verifies JWT against Cloudflare Access JWKS endpoint 4. `resolveCustomerRole()` resolves the RBAC role — Paso4 ROOT users (platform Access AUD) are always `ROOT`; client users get their lerma-assigned role (`ADMIN`/`BASE_USER`) forwarded by the Dispatch Worker via `X-FourIA-Role` 5. `userRole` and `customerSlug` are set in Hono context 6. Route handlers check role before serving content --- ## Development Tips ### Debugging the Container ```bash # Get container ID from Worker logs docker ps | grep sandbox # Enter container shell docker exec -it bash # Inside container: check openclaw status openclaw status openclaw devices list --json openclaw config get gateway.port # Check logs tail -f /tmp/openclaw-logs/*.log ``` ### Understanding Boot Logs During cold start, these log patterns are normal: ``` [Gateway] listProcesses returned 0 processes [Gateway] Starting new OpenClaw gateway... [ERROR] Uncaught ProcessReadyTimeoutError: Waiting for: port 18789 (TCP) ``` The gateway takes **up to 3 minutes** to become ready. The Worker retries automatically. If errors persist beyond 3 minutes: 1. Check `docker logs ` for OpenClaw errors 2. Verify CF AI Gateway config (`CLOUDFLARE_AI_GATEWAY_API_KEY` + `CLOUDFLARE_API_TOKEN` + `CF_AI_GATEWAY_ACCOUNT_ID` + `CF_AI_GATEWAY_GATEWAY_ID`) 3. Run `bun run dev:worker` to see full Worker output ### Service Integrations (Cloudflare Tunnel) The container can run Cloudflare Tunnels for exposing internal services: ```bash # Obsidian sync on port 18790 # Requires CLOUDFLARE_TUNNEL_TOKEN secret # start-openclaw.sh auto-starts cloudflared if token is set ``` ### CRON and Scheduled Tasks The Worker no longer uses a cron trigger. Containers are kept awake for non-frozen tenants by the lerma `HealthMonitor`, which probes the `GET /api/health` endpoint every few minutes (authenticated at the Cloudflare edge with a service token — see the path-scoped public-health Access applications in `apps/fouria/iac/access.tf`). The endpoint cold-starts the gateway via `ensureGateway` when it is not running, and also schedules interval-gated R2 backups in the background. This replaced the old `*/15 * * * *` wake/cron trigger. The container runs scheduled tasks via OpenClaw's cron system. ### Backup and Hard-Restart Policy Backups are snapshots of `/home/openclaw`, not a live filesystem journal. Agent workspaces are therefore kept under `/home/openclaw/workspace`; files outside that tree are not protected by the Sandbox snapshot. The Sandbox SDK's `createBackup` returns a handle marked with `localBucket: true` when `DEV_MODE` selects the local R2 binding (`persistence.ts`). That flag is persisted in `backup-handle.json` and forwarded to `restoreBackup()`, so a local backup is always restored through the local binding — never through the remote presigned-URL path, which 404s for archives that exist only in the local bucket. Legacy catalog entries without the flag default to the current `DEV_MODE`, and a presigned-URL 404 during restore is treated as a gone backup and pruned from the catalog. Automatic snapshots are interval-gated from non-passive `GET /api/health` calls. The explicit manual checkpoint is `POST /api/admin/storage/sync`. Soft reload (`POST /api/admin/gateway/restart`) does not snapshot. `POST /api/admin/storage/test-connectivity` (ADMIN/ROOT) is a read-only diagnostic that verifies the R2 backup bucket binding is reachable by HEADing the backup catalog handle — a 404 (no handle yet) still counts as connected. It is whitelisted for ROOT through the diagnostics gate (`DIAGNOSTICS_PATHS` in `src/routes/api.ts`) so platform operators can test backup connectivity from the Diagnostic Dashboard without explicit access to the ADMIN-gated Backups page. Hard restart (`POST /debug/restart-openclaw`) checkpoints before killing the gateway. If the checkpoint fails, the gateway is left running. After a successful checkpoint, the next activation restores the committed snapshot. ```mermaid flowchart TD HM["Lerma HealthMonitor
active /api/health"] --> S["scheduleBackupIfNeeded"] MAN["POST /api/admin/storage/sync"] --> C["createSnapshot"] S --> G{"credentials + interval + lock"} G -- no --> N["no-op"] G -- yes --> C C --> A["Sandbox snapshot
/home/openclaw"] A --> E["optional F4E1 encryption"] E --> P["commit backup-handle.json"] HR["POST /debug/restart-openclaw"] --> CP["checkpoint first"] CP --> K["graceful kill + restore marker"] K --> R["restore committed snapshot
on next activation"] SR["POST /api/admin/gateway/restart"] --> SOFT["SIGUSR1 in-process reload
no snapshot"] ``` Do not perform repeated hard restarts without verifying a successful checkpoint. The checkpoint protects only files inside `/home/openclaw` and only after the backup catalog has been committed. ### Sandbox Recovery (stale + orphaned) When the platform terminates a container behind a live Sandbox DO (idle recycle, maintenance, SIGTERM), every RPC into it fails with the SDK's `HTTP error! status: 500` wrapper — and retrying against the same instance fails forever, bricking the tenant. `gateway/sandbox-recovery.ts` classifies these dead-container errors (`isStaleContainerError`) and recovers (`recoverStaleSandbox`): destroy the instance, reset the startup-probe and persistence caches. The next RPC auto-provisions a fresh container and the normal cold-start flow runs (R2 restore → `start-openclaw.sh` → gateway). A container destroyed **out-of-band** (Cloudflare Containers dashboard "Destroy", failed operator force-stop) is a second failure class: the DO is alive but points at a placement that no longer exists, so RPCs fail with the SDK's _retryable_ admission errors (`no container instance available`, `maximum number of running container instances exceeded`) instead of the 500 signature. Those are legitimate during a cold start, so they are not treated as stale on their own. `isContainerUnavailableError` + `trackProvisioningFailure` escalate them to a recovery once the streak reaches `ORPHANED_INSTANCE_MIN_FAILURES` **and** spans `ORPHANED_INSTANCE_GRACE_MS` (120s — longer than the SDK's ≈110s provisioning budget, so a slow-but-healthy cold start is never cancelled mid-provision). The streak is cleared whenever a sandbox RPC succeeds. `maybeRecoverSandbox` unifies both classes for callers. Recovery is wired into three failure points in `gateway/process.ts`: `findExistingGatewayProcess` (listProcesses failures), `checkGatewayHealth` (exec failures), and the `ensureGateway` `startProcess` path (recover + retry once). It is cooldown-guarded (`STALE_RECOVERY_COOLDOWN_MS`, 60s) and never fires for deploy-window `OperationInterruptedError`. Permanent image errors still require the rescue tool (`rescueContainer`, see OPERATOR-GUIDE §5.1). Logs use the `[SandboxRecovery]` prefix. #### Transient transport drops (retry, don't reprovision) A third failure signature is not a dead container: `Network connection lost.` (and similar socket/WebSocket transport messages). It appears when the DO↔container connection drops _while the container is alive_ — most visibly while the Gateway is applying a config/plugin reload (e.g. adding a model provider in the Control UI), which restarts the Gateway and briefly tears down the container RPC/WS transports. `isContainerTransportError` (`gateway/sandbox-recovery.ts`) classifies these, and `isStaleContainerError` explicitly **excludes** them (even when the SDK wraps the message in its `sandbox. error` prefix) so a transient drop never destroys a live instance. `ensureGateway` instead retries with a short backoff: the port probe (`isGatewayPortOpenResilient`, up to `TRANSPORT_RETRY_ATTEMPTS`) and the `startProcess` spawn (once). Before this, a reload that dropped the transport mid probe fell straight through to `start-openclaw.sh` → `Failed to start process` → a 502 to the user, even though the Gateway came back seconds later. The retry keeps the request alive across the reload window. The manual lever is `POST /debug/reprovision` (DEBUG_ROUTES, ROOT): it destroys the instance, clears the startup/persistence caches, and resets the orphan streak without ever contacting the gateway — so it works when the container is entirely gone. Compare `POST /debug/restart-openclaw` (gateway-aware, checkpointed) and `POST /debug/destroy-container` (teardown only, used by E2E). ### Activity-Driven Wake (`ActivityScheduler`) Automations ("jobs") run **inside the Gateway**, so the Gateway must be running for a schedule to fire. Historically a Worker cron trigger woke every tenant on every tick (expensive and since removed). `src/activity/` replaces it with a per-tenant activity clock: - **`ActivityScheduler` Durable Object** (SQLite-backed, binding `ACTIVITY`) — stores a mirror of the tenant's automation schedule and owns a single alarm. Deliberately a **separate** DO from the SDK `Sandbox` DO, which already uses its own alarm for `sleepAfter`. - **Mirror** — warm paths (`GET /api/health`, only when `readiness.ok`) read `openclaw automations list --json` and push the earliest `nextRunAtMs` into the DO. The DO throttles this to once per 5 min, and it never wakes a sleeping container. - **Alarm** (`activity/alarm.ts`) — fires `CRON_WAKE_AHEAD_MINUTES` (default 10) before the next job, wakes + heals via `ensureGateway` (which recovers an orphaned instance — see above), then re-reads the schedule and re-arms. No enabled jobs → alarm cleared → the container sleeps. A due-but-pending job is woken once per occurrence and re-checked on a bounded retry, so it can never spin. Net: healing/wakeup is independent of user traffic, gated on real job activity, and free when idle. Traffic may cold-start the container; jobs get a pre-warmed, healed one. Requires the `ACTIVITY` binding in `wrangler.jsonc` and in the lerma provisioner (`tenant_provisioner.ex` `resource_bindings`) + the `new_sqlite_classes` migration (`dispatch_namespace.ex` `@do_sqlite_classes`). ### Health & Status Endpoints `GET /api/health` is the **unified** health endpoint: it returns the startup/readiness/liveness probes plus the canonical `status` (`running|starting|stopped`), `processId` and `processType`. It requires no authentication inside the Worker, but at the Cloudflare edge it is Access-protected by path-scoped applications (`*.fouria.io/api/health`) that admit Paso4 Root Users (browser) and the lerma health-monitor service token (headless). `GET /api/status`, `GET /api/admin/gateway/status/polling` and `GET /api/admin/gateway/status/ws` are thin aliases over the same snapshot. `GET /api/health/ws` streams the snapshot every ~2s over WebSocket. By default `/api/health` **warms** the container (lerma keep-alive). `?passive=1` probes **without** warming — the ROOT `/_admin/diagnostics` surface uses it so an operator can debug an instance whose container is down. The admin dashboard subscribes to the health WebSocket (with an HTTP polling fallback) and never blocks the UI on gateway status. #### Detached gateway detection The gateway runs as a **detached background child** of `start-openclaw.sh` (`openclaw gateway run ... &`), so after the script exits (exit 0 once the gateway is reachable) the gateway binary is **invisible to `listProcesses()`**. To avoid false `no_process` reports on log/status surfaces, those endpoints (`GET /api/admin/logs`, `GET /debug/logs`, `GET /debug/stop-gateway`, `POST /api/admin/gateway/restart`, `GET /api/admin/audit-logs`) call `findExistingGatewayProcess(sandbox, { probeLive: true })`, which falls back to a bounded `/health` probe and — when the gateway responds — returns a synthetic `gateway-live` process whose logs are read from `/tmp/gateway-stdout.log`. The passive health snapshot path deliberately does **not** probe so a down container fails fast. ### Log Levels | Level | When to use | | ------- | ----------------------------- | | `debug` | Local development | | `info` | Production default | | `warn` | High-traffic production | | `error` | Security-sensitive production | --- ## TypeScript & Tooling ### Type Generation Worker types are auto-generated: ```bash bun run types # wrangler types → worker-configuration.d.ts ``` **Never manually edit `worker-configuration.d.ts`** — it's regenerated from `wrangler.jsonc`. ### Testing ```bash bun run test # Unit tests (Vitest) bun run test:coverage # With coverage report bun run test:e2e # E2E tests (cctr + Playwright) bun run test:e2e:cli # CLI-only E2E (cctr) ``` ### Formatting & Linting ```bash bun run format # oxfmt (auto-format) bun run format:check # Check formatting bun run lint # oxlint bun run typecheck # tsc --noEmit ``` --- ## Common Tasks ### Adding a New API Endpoint 1. Create route handler in `src/routes/api/.ts` 2. Register in `src/routes/api.ts` 3. Add client method in `src/client/api.ts` 4. Write tests (unit + integration) 5. If new dependencies: update `package.json` ### Adding a New Environment Variable 1. Add to `OpenClawEnv` interface in `src/types.ts` 2. If passed to container: add to `buildEnvVars()` in `src/gateway/env.ts` 3. Update `.dev.vars.example` 4. Document in [../../docs/ENV-VARIABLES.md](../../docs/ENV-VARIABLES.md)