# Fouria

Cloudflare Worker that manages OpenClaw instances inside Cloudflare Sandbox containers. This is the **core component** of FourIA.

> **Dev guide:** [docs/WORKER-DEVELOPMENT.md](/llms/platform/apps/fouria/worker-development/index.md) — deep-dive architecture, frontend/backend patterns, RBAC
> **Cloudflare setup & operations:** [docs/OPERATOR-GUIDE.md](/llms/platform/apps/fouria/operator-guide/index.md) — Access, R2, AI Gateway, Tunnel, CDP, backups, rescue/restart, troubleshooting
> **Project docs:** [../../docs/DEVELOPMENT.md](/llms/platform/reference/development/index.md) — full onboarding guide
> **Env vars:** [../../docs/ENV-VARIABLES.md](/llms/platform/reference/env-variables/index.md) — complete reference
> **Operating & recovery:** [docs/OPERATOR-GUIDE.md](/llms/platform/apps/fouria/operator-guide/index.md) — health, debug options, backups, rescue/restart, troubleshooting

---

## Architecture

```
Browser → Worker (Hono) → Sandbox Container (OpenClaw Gateway on :18789)
                              │
                    start-openclaw.sh:
                      1. OpenClaw onboard (auth setup)
                      2. moltlazy patch (config injection)
                      3. OpenClaw gateway run
```

The Worker proxies HTTP/WebSocket to the gateway, provides the admin dashboard at `/_admin/`, and exposes a REST API at `/api/*`.

---

## Development

```bash
# Install dependencies (from repo root)
bun install

# Build shared packages (moltlazy + plugin)
bun run build:all

# Create .dev.vars (from root of this package)
cp .dev.vars.example .dev.vars
# Add at minimum: CLOUDFLARE_AI_GATEWAY_API_KEY (+ CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID), MOLTBOT_GATEWAY_TOKEN
# For local unified-billing REST requests, also set CLOUDFLARE_API_TOKEN to the same AI-scoped credential.
```

### Running Locally

```bash
# Terminal 1: Frontend (Vite hot-reload on :8787)
bun run dev

# Terminal 2: Backend Worker (wrangler dev on :8788)
bun run dev:worker

# Full local stack (build all + wrangler dev)
bun start
```

> Frontend changes hot-reload. Backend changes in `src/` require re-running `bun run dev:worker`.

### Vite Proxy

`bun run dev` (Vite) proxies `/api/*`, `/debug/*`, `/cdp/*`, and static assets to the Worker on port 8788. The Worker itself runs on port 8788 via `bun run dev:worker`.

---

## Key Files

| File                     | Purpose                                                      |
| ------------------------ | ------------------------------------------------------------ |
| `src/index.ts`           | Main Hono app: middleware pipeline, WS proxy, route mounting |
| `src/routes/api.ts`      | API route aggregator (delegates to domain modules)           |
| `src/gateway/process.ts` | Container lifecycle: find, start, restart, destroy           |
| `src/gateway/env.ts`     | Builds env vars passed to container                          |
| `src/auth/`              | CF Access JWT auth + RBAC (ROOT/ADMIN/BASE_USER)             |
| `src/client/`            | React SPA (Vite 6, React 19, react-router-dom 7)             |
| `start-openclaw.sh`      | Container entrypoint: onboard → moltlazy patch → gateway     |
| `Dockerfile`             | Sandbox container image                                      |
| `wrangler.jsonc`         | Cloudflare Worker config (bindings, envs, DO migrations)     |
| `iac/`                   | OpenTofu infrastructure-as-code                              |

---

## Frontend (`src/client/`)

React 19 SPA built with Vite 6, served from `/_admin/`.

### Pages

| Page           | Route                  | Purpose                    |
| -------------- | ---------------------- | -------------------------- |
| Dashboard Home | `/_admin/`             | Overview and status        |
| Agents         | `/_admin/agents`       | Agent CRUD and bindings    |
| Connections    | `/_admin/connections`  | Channel integrations       |
| Knowledge      | `/_admin/knowledge`    | Memory wiki and vault      |
| Integrations   | `/_admin/integrations` | Integration configuration  |
| Secrets        | `/_admin/secrets`      | Vault management           |
| Backups        | `/_admin/backups`      | Backup history and restore |
| Diagnostics    | `/_admin/diagnostic`   | Costs, usage, OTel metrics |
| Audit Logs     | `/_admin/audit-logs`   | Activity log viewer        |
| Automation     | `/_admin/automation`   | Workflow editor            |

### API Client (`src/client/api.ts`)

The frontend API client calls `/api/admin/*` endpoints. These are Hono routes that wrap OpenClaw CLI calls inside the Sandbox container.

---

## Backend (`src/`)

### API Routes

All API routes are at `/api/admin/*`, protected by CF Access auth + RBAC:

| Module         | Endpoints                       |
| -------------- | ------------------------------- |
| `misc`         | `/whoami`, `/version`, `/logs`  |
| `devices`      | Device pairing approval         |
| `gateway`      | Status, restart, lifecycle      |
| `config`       | Get/set/validate/fix config     |
| `agents`       | Agent CRUD, bindings, persona   |
| `models`       | Model listing and selection     |
| `channels`     | Channel status and accounts     |
| `integrations` | Integration config              |
| `knowledge`    | Knowledge tree, wiki management |
| `secrets`      | Vault audit and reload          |
| `dreaming`     | Memory dreaming config          |
| `costs`        | Cost pricing and metrics        |
| `cron`         | Cron job management             |
| `vault`        | Encrypted secrets store         |
| `audit-logs`   | Activity log retrieval          |
| `users`        | Team member management          |

### RBAC Roles

| Role          | Permissions                                   | Access Scope                 |
| ------------- | --------------------------------------------- | ---------------------------- |
| **ROOT**      | `admin:platform`, `admin:customer`, `api:all` | All tenants, debug endpoints |
| **ADMIN**     | `admin:customer`, `api:all`                   | Own tenant only              |
| **BASE_USER** | `api:own`                                     | Own tenant, read-only        |

### Operational Modes

| Variable                | Purpose                                                                                     |
| ----------------------- | ------------------------------------------------------------------------------------------- |
| `DEV_MODE`              | Skips CF Access auth + bypasses device pairing                                              |
| `E2E_TEST_MODE`         | Skips CF Access auth but keeps device pairing                                               |
| `DEBUG_ROUTES`          | Enables `/debug/*` endpoints (ROOT role only)                                               |
| `DEMO_MODE`             | Enables read-only dashboard with sample data                                                |
| `STARTUP_LOG_VERBOSITY` | Clean boot-log verbosity for the loading page: `steps` \| `detailed` (default) \| `verbose` |

---

## Health Checks

Every User Worker exposes a **single, unified health endpoint** that all internal and external consumers share.

### `GET /api/health` — unified snapshot (public, no auth)

This is the **source of truth** for instance health. It runs Kubernetes-style probes against the OpenClaw gateway inside the sandbox container and returns a structured snapshot. `lerma`'s `HealthMonitor`/`HealthPoller` poll it to keep `clients.status` / `instances.status` accurate, the loading page polls it during cold start, and the admin dashboard (`/_admin`) streams it.

```json
{
  "ts": "2026-08-15T12:00:00.000Z",
  "startup": { "ok": true, "status": "success", "latencyMs": 12 },
  "readiness": { "ok": true, "status": "success", "latencyMs": 8 },
  "liveness": { "ok": true, "status": "success", "latencyMs": 5 },
  "restoreStatus": { "hadBackup": true, "restored": true },
  "status": "running",
  "processId": "proc_123",
  "processType": "gateway"
}
```

| Field           | Type                         | Description                                                                 |
| --------------- | ---------------------------- | --------------------------------------------------------------------------- |
| `startup`       | `ProbeResult`                | One-time init: liveness + readiness + backup-restore confirmation           |
| `readiness`     | `ProbeResult`                | Ready to serve traffic: script exited, gateway binary running, `/health` up |
| `liveness`      | `ProbeResult`                | Gateway `/health` responds with `{"ok": true}`                              |
| `restoreStatus` | `{ hadBackup, restored }`    | Whether an R2 backup existed and was restored                               |
| `status`        | `running\|starting\|stopped` | Canonical gateway status derived from the probes                            |
| `processId`     | `string?`                    | Detected gateway/startup-script process id, if any                          |
| `processType`   | `gateway\|script?`           | Whether the detected process is the gateway binary or the startup script    |

`ProbeResult` = `{ ok: boolean, status: "success"|"failure"|"unknown", latencyMs: number, error?: string }`.

**Status derivation:**

| Condition                         | `status`   |
| --------------------------------- | ---------- |
| `readiness.ok`                    | `running`  |
| Readiness fails, a process exists | `starting` |
| Readiness fails, no process       | `stopped`  |

**Activating vs passive.** By default a call to `/api/health` **warms** the container — if the gateway is not running it triggers `ensureGateway` + `restoreIfNeeded` (this is how `lerma` keeps sleepers alive and recovers stopped instances). Pass `?passive=1` to **fail fast instead of warming**; passive probes use a short timeout and never start the container. The ROOT diagnostics dashboard uses passive probes so an operator can debug an instance whose container is down.

```bash
curl -s https://{slug}.fouria.io/api/health           # activating (warm + recover)
curl -s https://{slug}.fouria.io/api/health?passive=1  # passive (fail fast, ROOT debug)
```

### `GET /api/health/ws` — WebSocket stream (public)

Pushes a fresh `/api/health` snapshot every ~5s over a WebSocket, letting the admin dashboard replace HTTP polling. Supports the same `?passive=1` flag.

```
wss://{slug}.fouria.io/api/health/ws?passive=1
```

### `GET /api/status` — legacy alias (public, no auth)

Kept for backward compatibility (E2E scripts, `server/wait-ready`, dispatch e2e). Delegates to the same snapshot and returns the legacy shape `{ ok, status, processId, e2eMode, pairing }`; `pairing` is `{ enabled: true }` when healthy.

### `GET /api/startup-progress` — boot stepper + log (public, no warm-up)

Drives the loading page during cold start. Returns the boot-phase stepper and
curated boot log emitted by `start-openclaw.sh` (see `STEPS`/`mark_step` in the
script and WORKER-DEVELOPMENT.md). Fail-fast and **never warms** the container
(no `ensureGateway`/`restoreIfNeeded`), so it adds no load to the health path.

```json
{
  "available": true,
  "steps": [
    { "id": "init", "label": "Preparing container" },
    { "id": "onboard", "label": "Configuring gateway" }
  ],
  "currentStep": 1,
  "currentStepId": "onboard",
  "totalSteps": 7,
  "skipped": false,
  "ts": 1787052449,
  "log": "[startup] (1/7) Preparing container\n..."
}
```

Returns `{ "available": false }` when the container is cold/off or the image
hasn't emitted progress yet.

### Protected status endpoints (CF Access)

`GET /api/admin/gateway/status/polling` and `GET /api/admin/gateway/status/ws` are thin aliases over the same snapshot for the authenticated admin API and the moltlazy SDK. The `POST /api/admin/gateway/restart` handler performs an in-process **soft** reload via `SIGUSR1`; it does not create a snapshot. Use `POST /api/admin/storage/sync` for an explicit checkpoint.

For a **checkpointed hard restart** (checkpoint → kill → restore marker → reprovision through `start-openclaw.sh`), use `POST /debug/restart-openclaw` (requires `DEBUG_ROUTES=true`; button in the diagnostic dashboard). If the checkpoint fails, the gateway is left running.

> See [`../../docs/WORKER-HEALTH.md`](/llms/platform/reference/worker-health/index.md) for the full probe contract and `lerma` consumer semantics.

---

## Cloudflare API Token Permissions

This project does not use a Cloudflare API token at runtime — per-tenant secrets are provisioned by `lerma`. The following tokens are used during deployment and E2E testing:

| Token                       | Permission              | Purpose                                      |
| --------------------------- | ----------------------- | -------------------------------------------- |
| `DEPLOY_WORKER_API_TOKEN`   | `Workers Scripts:Edit`  | `wrangler deploy` and `wrangler secret bulk` |
| `LERMA_API_TOKEN` (CI only) | Full provisioning scope | E2E test infrastructure (Tofu init)          |

All credentials are documented in [docs/schemas/secrets-manifest.schema.json](/assets/docs/schemas/secrets-manifest.schema.json).

## Testing

### Unit Tests (Vitest)

```bash
bun run test             # Unit tests (Vitest)
bun run test:coverage    # With coverage report
bun run test:watch       # Watch mode
bun run typecheck        # TypeScript check
bun run lint             # oxlint
bun run format:check     # Formatting check
```

### E2E Tests

Two E2E test tiers exist: **cloud-deployed** (cctr, real CF infra) and **local-API** (Vitest, mocked deps).

```bash
bun run test:e2e         # Full E2E: cctr + Playwright (deploys real infra)
bun run test:e2e:cli     # CLI-only E2E: cctr (skips browser tests)
```

#### E2E Test Design Guide

Tests are in `test/e2e/`. Each `.txt` file is a **cctr corpus** — a plain-text test script where each section is a shell command with assertions.

##### cctr File Structure

```
===                           ← section header (test name or empty)
command description           ← %require means: if this fails, skip rest
%require                      ← optional: block subsequent tests on failure
===
shell command(s) here          ← executed with /bin/bash
---
{{ output_name: type }}        ← capture command output as typed variable
---
where                         ← assertion block
* output_name contains "text"  ← substring check
* output_name matches "^\\d+"  ← regex check
* output_name != ""            ← inequality
```

##### Naming Convention

Files run in **alphabetical order**. Prefix determines execution sequence:

| Prefix      | Purpose                           | Example                                 |
| ----------- | --------------------------------- | --------------------------------------- |
| `_setup`    | Deploy infra, start browser, wait | `_setup.txt`                            |
| `_teardown` | Dump logs, stop browser, destroy  | `_teardown.txt`                         |
| `a_cli_*`   | CLI integration tests (core APIs) | `a_cli_misc.txt`, `b_cli_devices.txt`   |
| `f_cli_*`   | CLI integration tests (advanced)  | `f_cli_agents.txt`, `g_cli_secrets.txt` |
| `pairing_*` | Browser-based (plwr) tests        | `pairing_and_conversation.txt`          |
| `z_cli_*`   | Staging/prod-specific             | `z_cli_staging.txt`                     |
| `zzz_*`     | Must run last                     | `zzz_cron_wake.txt`                     |

##### cctr Features

| Feature          | Syntax                                           | Example                                         |
| ---------------- | ------------------------------------------------ | ----------------------------------------------- |
| Section header   | `===` on its own lines, with description between | `===` / `wait for gateway` / `===`              |
| Block on failure | `%require` on line after `===`                   | `%require` → if fails, skip all later tests     |
| Output capture   | `{{ name: type }}`                               | `{{ result: json object }}`                     |
| Assertions       | `where` block after output                       | `* result contains "pending"`                   |
| Regex assertion  | `* name matches "pattern"`                       | `* version matches "^v\\d+"`                    |
| Equality         | `* name != ""`                                   | `* output != ""`                                |
| Complex objects  | Parenthesized boolean expressions                | `* (status.ok == true) or (status.ok == false)` |

##### Variable Types

| Type          | Use case                                     |
| ------------- | -------------------------------------------- |
| `string`      | Text output (version numbers, status codes)  |
| `json object` | JSON `{}` response (devices, config, agents) |
| `json array`  | JSON `[]` response (list endpoints)          |

##### Fixtures

The `fixture/` directory contains helper scripts that get copied to `$CCTR_FIXTURE_DIR` at runtime:

| Script              | Purpose                                                                                               |
| ------------------- | ----------------------------------------------------------------------------------------------------- |
| `debug-cli`         | Executes OpenClaw CLI commands in sandbox via `/debug/cli` endpoint. Extracts JSON from mixed stdout. |
| `curl-auth`         | Authenticated curl against the deployed Worker URL                                                    |
| `server/start`      | Deploys Worker + Terraform infra to Cloudflare                                                        |
| `server/stop`       | Deletes Worker + destroys Terraform resources                                                         |
| `server/deploy`     | Deploys worker code + builds all dependencies                                                         |
| `server/wait-ready` | Polls `/api/health` until the gateway is up                                                           |
| `start-browser`     | Launches Playwright Chromium for plwr tests                                                           |
| `stop-browser`      | Stops Playwright and saves video                                                                      |

##### Example: CLI E2E Test

```bash
# f_cli_agents.txt — Agent CRUD via debug CLI endpoint
===
list agents from openclaw.json
===
WORKER_URL=$(cat "$CCTR_FIXTURE_DIR/worker-url.txt")
./curl-auth -s "$WORKER_URL/debug/container-config" | jq '.config.agents'
---
{{ result: json object }}
---
where
* result contains "list"
* result contains "defaults"

===
create a test agent via CLI
%require
===
WORKER_URL=$(cat "$CCTR_FIXTURE_DIR/worker-url.txt")
AGENT_ID="e2e-test-agent-$(date +%s)"
./debug-cli "openclaw agents add $AGENT_ID --workspace ... --json"
---
{{ result: json object }}
---
where
* result contains "id" or result contains "success"

===
delete test agent
===
./debug-cli "openclaw agents delete $AGENT_ID --force --json"
---
{{ result: json object }}
---
where
* result contains "id" or result contains "success" or (result contains "error" and result contains "not found")
```

##### Example: Browser E2E Test (plwr)

```bash
# pairing_and_conversation.txt — Device approval + chat via browser
===
navigate to admin page
%require
===
if [ "${E2E_SKIP_BROWSER_TESTS:-}" = "1" ]; then
    echo "Skipping browser test"
    exit 0
fi
TOKEN=$(cat "$CCTR_FIXTURE_DIR/gateway-token.txt")
WORKER_URL=$(cat "$CCTR_FIXTURE_DIR/worker-url.txt")
plwr -S moltworker-e2e open "$WORKER_URL/_admin/?token=$TOKEN" -T 120000

===
wait for pending devices and approve
%require
===
plwr -S moltworker-e2e wait 'button:has-text("Approve All")' -T 120000
plwr -S moltworker-e2e click 'button:has-text("Approve All")' -T 10000
plwr -S moltworker-e2e wait-not 'button:has-text("Approv")' -T 30000

===
send math question and verify answer
===
plwr -S moltworker-e2e fill textarea 'What is 847293 + 651824? Reply with just the number.'
plwr -S moltworker-e2e press Enter
plwr -S moltworker-e2e wait 'text=1499117' -T 120000
```

###### Running E2E via act (CI Simulation)

NixOS users can run CI workflows locally with `act`. The E2E job deploys real Cloudflare infrastructure (Worker + Terraform), runs tests, and destroys it.

```bash
# Prerequisites: create a GitHub personal access token (classic) with repo scope
# Store it and Cloudflare credentials in a .secrets file:
cat > .secrets << 'SECRETS'
GITHUB_TOKEN=ghp_your_personal_access_token
LERMA_API_TOKEN=your_lerma_api_token
E2E_CF_ACCOUNT_ID=your_account_id
E2E_WORKERS_SUBDOMAIN=your_subdomain
E2E_CF_ACCESS_TEAM_DOMAIN=yourteam.cloudflareaccess.com
E2E_R2_ACCESS_KEY_ID=your_r2_access_key
E2E_R2_SECRET_ACCESS_KEY=your_r2_secret_key
CLOUDFLARE_AI_GATEWAY_API_KEY=your_ai_gateway_key
CF_AI_GATEWAY_ACCOUNT_ID=your_account_id
CF_AI_GATEWAY_GATEWAY_ID=your_gateway_id
SECRETS

# Run the unit + lint workflow (no Cloudflare credentials needed beyond GITHUB_TOKEN)
act -W .github/workflows/test.yml -j unit

# Run the full E2E job (requires Cloudflare credentials)
act -W .github/workflows/test.yml -j e2e

# Run specific event (e2e only runs on develop branch by default, so simulate push to develop)
act -W .github/workflows/test.yml -j e2e -e <(echo '{"ref":"refs/heads/develop"}')
```

> The `GITHUB_TOKEN` is needed because `act` clones GitHub Actions (actions/checkout, etc.) via HTTPS. Without a valid token, git operations fail with "authentication required." Create a [classic PAT](https://github.com/settings/tokens) with `repo` scope.

##### Running E2E Tests Directly (no act)

If you have Cloudflare credentials in a `.dev.vars` file, run cctr directly:

```bash
# 1. Copy E2E credentials
cp test/e2e/.dev.vars.example test/e2e/.dev.vars
# Edit .dev.vars with your Cloudflare credentials

# 2. Install tools
npx playwright install chromium  # For browser tests
# cctr: cargo install cctr or brew install andreasjansson/tap/cctr
# plwr: see https://github.com/andreasjansson/plwr

# 3. Run tests
cctr test/e2e/                           # All tests
cctr test/e2e/ -p pairing                # Specific test file
cctr test/e2e/ -vv                       # Verbose (stream output)
PLAYWRIGHT_HEADED=1 cctr test/e2e/       # See browser during tests
E2E_SKIP_BROWSER_TESTS=1 cctr test/e2e/  # CLI tests only
```

#### Vitest E2E Pattern (for API routes)

For tests that don't need real Cloudflare infra, use Vitest with a real Hono app instance:

```typescript
import { Hono } from 'hono';
import { createMockEnv, suppressConsole } from '../src/test-utils';

function createTestApp() {
  const app = new Hono<AppEnv>();
  app.use('*', async (c, next) => {
    c.set('userRole', 'ADMIN' as any);
    c.set('customerSlug', 'test-customer');
    c.env = { FOURIA_API_URL: 'https://api.example.com', ...c.env } as any;
    await next();
  });
  app.route('/', myRoutes);
  return app;
}

function makeRequest(app: Hono<AppEnv>, path: string, opts = {}) {
  const env = createMockEnv(opts.env);
  return app.request(path, { method: opts.method || 'GET' }, env);
}

describe('my endpoint', () => {
  beforeEach(() => suppressConsole());

  it('returns expected structure', async () => {
    const res = await makeRequest(createTestApp(), '/');
    expect(res.status).toBe(200);
    const body = await res.json();
    expect(body).toHaveProperty('expectedKey');
  });

  it('forwards to FourIA API with correct headers', async () => {
    vi.stubGlobal(
      'fetch',
      vi.fn().mockResolvedValue(new Response(JSON.stringify({ result: 'ok' }), { status: 200 })),
    );
    const res = await makeRequest(createTestApp(), '/endpoint?param=value');
    expect(res.status).toBe(200);
    vi.unstubAllGlobals();
  });
});
```

##### Design Principles

1. **Tests deploy real infra, run against it, destroy it.** Each cctr suite provisions a live Worker and container. No mocks — tests validate against the actual Cloudflare environment (R2 mounts, cold starts, Access auth, network latency).

2. **Sequential naming enforces order.** Files run alphabetically. Use numeric prefixes (`a_`, `b_`, `c_`) to build dependencies (gateway must be up before agent tests run).

3. **`%require` for hard gates.** If the gateway isn't reachable, skip all downstream tests. Use sparingly — only for prerequisites that invalidate all later tests.

4. **fixture/ scripts provide a stable CLI surface.** The `debug-cli` and `curl-auth` scripts abstract away URL/token management so test files focus on the scenario.

5. **Browser tests are skippable.** Set `E2E_SKIP_BROWSER_TESTS=1` to run only CLI tests. Every browser section checks this flag and exits 0 if set.

6. **Setup/teardown are tests too.** `_setup.txt` and `_teardown.txt` are full cctr files with assertions. If setup fails, the suite doesn't run. Teardown always runs.

7. **Vitest E2E supplements cctr.** For API-level integration tests that don't need a live container (cost pipelines, OTel ingestion, RBAC enforcement), use Vitest with `createTestApp()` and mocked `fetch`. These run fast, don't require credentials, and validate the full middleware pipeline.

---

## Deployment

```bash
bun run deploy:staging   # Deploy to staging
bun run deploy           # Deploy to production
```

---

## Quick Debug Reference

```bash
# View live logs
npx wrangler tail

# Enter running container
docker exec -it <container_id> bash

# Inside container: check gateway
openclaw status
openclaw devices list --json
tail -f /tmp/openclaw-logs/*.log

# Check secrets
npx wrangler secret list
```

### Normal Boot Warnings

During cold start (first request), these errors are **normal**:

```
[ERROR] [api/status] Port wait failed: Process did not become ready within 5000ms
[ERROR] Uncaught ProcessReadyTimeoutError: Waiting for: port 18789 (TCP)
```

The gateway takes up to 3 minutes to become ready. Retries are automatic. If errors persist beyond 3 minutes, check container logs.