Fouria
The multi-tenant Cloudflare Worker that runs OpenClaw inside a Sandbox container.
Cloudflare Worker that manages OpenClaw instances inside Cloudflare Sandbox containers. This is the core component of FourIA.
Dev guide: docs/WORKER-DEVELOPMENT.md — deep-dive architecture, frontend/backend patterns, RBAC Cloudflare setup & operations: docs/OPERATOR-GUIDE.md — Access, R2, AI Gateway, Tunnel, CDP, backups, rescue/restart, troubleshooting Project docs: ../../docs/DEVELOPMENT.md — full onboarding guide Env vars: ../../docs/ENV-VARIABLES.md — complete reference Operating & recovery: docs/OPERATOR-GUIDE.md — health, debug options, backups, rescue/restart, troubleshooting
Architecture
Browser → Worker (Hono) → Sandbox Container (OpenClaw Gateway on :18789)
│
start-openclaw.sh:
1. OpenClaw onboard (auth setup)
2. moltlazy patch (config injection)
3. OpenClaw gateway run
The Worker proxies HTTP/WebSocket to the gateway, provides the admin dashboard at /_admin/, and exposes a REST API at /api/*.
Development
# Install dependencies (from repo root)
bun install
# Build shared packages (moltlazy + plugin)
bun run build:all
# Create .dev.vars (from root of this package)
cp .dev.vars.example .dev.vars
# Add at minimum: CLOUDFLARE_AI_GATEWAY_API_KEY (+ CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID), MOLTBOT_GATEWAY_TOKEN
# For local unified-billing REST requests, also set CLOUDFLARE_API_TOKEN to the same AI-scoped credential.Running Locally
# Terminal 1: Frontend (Vite hot-reload on :8787)
bun run dev
# Terminal 2: Backend Worker (wrangler dev on :8788)
bun run dev:worker
# Full local stack (build all + wrangler dev)
bun start
Frontend changes hot-reload. Backend changes in
src/require re-runningbun run dev:worker.
Vite Proxy
bun run dev (Vite) proxies /api/*, /debug/*, /cdp/*, and static assets to the Worker on port 8788. The Worker itself runs on port 8788 via bun run dev:worker.
Key Files
| File | Purpose |
|---|---|
src/index.ts | Main Hono app: middleware pipeline, WS proxy, route mounting |
src/routes/api.ts | API route aggregator (delegates to domain modules) |
src/gateway/process.ts | Container lifecycle: find, start, restart, destroy |
src/gateway/env.ts | Builds env vars passed to container |
src/auth/ | CF Access JWT auth + RBAC (ROOT/ADMIN/BASE_USER) |
src/client/ | React SPA (Vite 6, React 19, react-router-dom 7) |
start-openclaw.sh | Container entrypoint: onboard → moltlazy patch → gateway |
Dockerfile | Sandbox container image |
wrangler.jsonc | Cloudflare Worker config (bindings, envs, DO migrations) |
iac/ | OpenTofu infrastructure-as-code |
Frontend (src/client/)
React 19 SPA built with Vite 6, served from /_admin/.
Pages
| Page | Route | Purpose |
|---|---|---|
| Dashboard Home | /_admin/ | Overview and status |
| Agents | /_admin/agents | Agent CRUD and bindings |
| Connections | /_admin/connections | Channel integrations |
| Knowledge | /_admin/knowledge | Memory wiki and vault |
| Integrations | /_admin/integrations | Integration configuration |
| Secrets | /_admin/secrets | Vault management |
| Backups | /_admin/backups | Backup history and restore |
| Diagnostics | /_admin/diagnostic | Costs, usage, OTel metrics |
| Audit Logs | /_admin/audit-logs | Activity log viewer |
| Automation | /_admin/automation | Workflow editor |
API Client (src/client/api.ts)
The frontend API client calls /api/admin/* endpoints. These are Hono routes that wrap OpenClaw CLI calls inside the Sandbox container.
Backend (src/)
API Routes
All API routes are at /api/admin/*, protected by CF Access auth + RBAC:
| Module | Endpoints |
|---|---|
misc | /whoami, /version, /logs |
devices | Device pairing approval |
gateway | Status, restart, lifecycle |
config | Get/set/validate/fix config |
agents | Agent CRUD, bindings, persona |
models | Model listing and selection |
channels | Channel status and accounts |
integrations | Integration config |
knowledge | Knowledge tree, wiki management |
secrets | Vault audit and reload |
dreaming | Memory dreaming config |
costs | Cost pricing and metrics |
cron | Cron job management |
vault | Encrypted secrets store |
audit-logs | Activity log retrieval |
users | Team member management |
RBAC Roles
| Role | Permissions | Access Scope |
|---|---|---|
| ROOT | admin:platform, admin:customer, api:all | All tenants, debug endpoints |
| ADMIN | admin:customer, api:all | Own tenant only |
| BASE_USER | api:own | Own tenant, read-only |
Operational Modes
| Variable | Purpose |
|---|---|
DEV_MODE | Skips CF Access auth + bypasses device pairing |
E2E_TEST_MODE | Skips CF Access auth but keeps device pairing |
DEBUG_ROUTES | Enables /debug/* endpoints (ROOT role only) |
DEMO_MODE | Enables read-only dashboard with sample data |
STARTUP_LOG_VERBOSITY | Clean boot-log verbosity for the loading page: steps | detailed (default) | verbose |
Health Checks
Every User Worker exposes a single, unified health endpoint that all internal and external consumers share.
GET /api/health — unified snapshot (public, no auth)
This is the source of truth for instance health. It runs Kubernetes-style probes against the OpenClaw gateway inside the sandbox container and returns a structured snapshot. lerma’s HealthMonitor/HealthPoller poll it to keep clients.status / instances.status accurate, the loading page polls it during cold start, and the admin dashboard (/_admin) streams it.
{
"ts": "2026-08-15T12:00:00.000Z",
"startup": { "ok": true, "status": "success", "latencyMs": 12 },
"readiness": { "ok": true, "status": "success", "latencyMs": 8 },
"liveness": { "ok": true, "status": "success", "latencyMs": 5 },
"restoreStatus": { "hadBackup": true, "restored": true },
"status": "running",
"processId": "proc_123",
"processType": "gateway"
}| Field | Type | Description |
|---|---|---|
startup | ProbeResult | One-time init: liveness + readiness + backup-restore confirmation |
readiness | ProbeResult | Ready to serve traffic: script exited, gateway binary running, /health up |
liveness | ProbeResult | Gateway /health responds with {"ok": true} |
restoreStatus | { hadBackup, restored } | Whether an R2 backup existed and was restored |
status | running|starting|stopped | Canonical gateway status derived from the probes |
processId | string? | Detected gateway/startup-script process id, if any |
processType | gateway|script? | Whether the detected process is the gateway binary or the startup script |
ProbeResult = { ok: boolean, status: "success"|"failure"|"unknown", latencyMs: number, error?: string }.
Status derivation:
| Condition | status |
|---|---|
readiness.ok | running |
| Readiness fails, a process exists | starting |
| Readiness fails, no process | stopped |
Activating vs passive. By default a call to /api/health warms the container — if the gateway is not running it triggers ensureGateway + restoreIfNeeded (this is how lerma keeps sleepers alive and recovers stopped instances). Pass ?passive=1 to fail fast instead of warming; passive probes use a short timeout and never start the container. The ROOT diagnostics dashboard uses passive probes so an operator can debug an instance whose container is down.
curl -s https://{slug}.fouria.io/api/health # activating (warm + recover)
curl -s https://{slug}.fouria.io/api/health?passive=1 # passive (fail fast, ROOT debug)GET /api/health/ws — WebSocket stream (public)
Pushes a fresh /api/health snapshot every ~5s over a WebSocket, letting the admin dashboard replace HTTP polling. Supports the same ?passive=1 flag.
wss://{slug}.fouria.io/api/health/ws?passive=1GET /api/status — legacy alias (public, no auth)
Kept for backward compatibility (E2E scripts, server/wait-ready, dispatch e2e). Delegates to the same snapshot and returns the legacy shape { ok, status, processId, e2eMode, pairing }; pairing is { enabled: true } when healthy.
GET /api/startup-progress — boot stepper + log (public, no warm-up)
Drives the loading page during cold start. Returns the boot-phase stepper and
curated boot log emitted by start-openclaw.sh (see STEPS/mark_step in the
script and WORKER-DEVELOPMENT.md). Fail-fast and never warms the container
(no ensureGateway/restoreIfNeeded), so it adds no load to the health path.
{
"available": true,
"steps": [
{ "id": "init", "label": "Preparing container" },
{ "id": "onboard", "label": "Configuring gateway" }
],
"currentStep": 1,
"currentStepId": "onboard",
"totalSteps": 7,
"skipped": false,
"ts": 1787052449,
"log": "[startup] (1/7) Preparing container\n..."
}
Returns { "available": false } when the container is cold/off or the image
hasn’t emitted progress yet.
Protected status endpoints (CF Access)
GET /api/admin/gateway/status/polling and GET /api/admin/gateway/status/ws are thin aliases over the same snapshot for the authenticated admin API and the moltlazy SDK. The POST /api/admin/gateway/restart handler performs an in-process soft reload via SIGUSR1; it does not create a snapshot. Use POST /api/admin/storage/sync for an explicit checkpoint.
For a checkpointed hard restart (checkpoint → kill → restore marker → reprovision through start-openclaw.sh), use POST /debug/restart-openclaw (requires DEBUG_ROUTES=true; button in the diagnostic dashboard). If the checkpoint fails, the gateway is left running.
See
../../docs/WORKER-HEALTH.mdfor the full probe contract andlermaconsumer semantics.
Cloudflare API Token Permissions
This project does not use a Cloudflare API token at runtime — per-tenant secrets are provisioned by lerma. The following tokens are used during deployment and E2E testing:
| Token | Permission | Purpose |
|---|---|---|
DEPLOY_WORKER_API_TOKEN | Workers Scripts:Edit | wrangler deploy and wrangler secret bulk |
LERMA_API_TOKEN (CI only) | Full provisioning scope | E2E test infrastructure (Tofu init) |
All credentials are documented in docs/schemas/secrets-manifest.schema.json.
Testing
Unit Tests (Vitest)
bun run test # Unit tests (Vitest)
bun run test:coverage # With coverage report
bun run test:watch # Watch mode
bun run typecheck # TypeScript check
bun run lint # oxlint
bun run format:check # Formatting checkE2E Tests
Two E2E test tiers exist: cloud-deployed (cctr, real CF infra) and local-API (Vitest, mocked deps).
bun run test:e2e # Full E2E: cctr + Playwright (deploys real infra)
bun run test:e2e:cli # CLI-only E2E: cctr (skips browser tests)E2E Test Design Guide
Tests are in test/e2e/. Each .txt file is a cctr corpus — a plain-text test script where each section is a shell command with assertions.
cctr File Structure
=== ← section header (test name or empty)
command description ← %require means: if this fails, skip rest
%require ← optional: block subsequent tests on failure
===
shell command(s) here ← executed with /bin/bash
---
{{ output_name: type }} ← capture command output as typed variable
---
where ← assertion block
* output_name contains "text" ← substring check
* output_name matches "^\\d+" ← regex check
* output_name != "" ← inequalityNaming Convention
Files run in alphabetical order. Prefix determines execution sequence:
| Prefix | Purpose | Example |
|---|---|---|
_setup | Deploy infra, start browser, wait | _setup.txt |
_teardown | Dump logs, stop browser, destroy | _teardown.txt |
a_cli_* | CLI integration tests (core APIs) | a_cli_misc.txt, b_cli_devices.txt |
f_cli_* | CLI integration tests (advanced) | f_cli_agents.txt, g_cli_secrets.txt |
pairing_* | Browser-based (plwr) tests | pairing_and_conversation.txt |
z_cli_* | Staging/prod-specific | z_cli_staging.txt |
zzz_* | Must run last | zzz_cron_wake.txt |
cctr Features
| Feature | Syntax | Example |
|---|---|---|
| Section header | === on its own lines, with description between | === / wait for gateway / === |
| Block on failure | %require on line after === | %require → if fails, skip all later tests |
| Output capture | {{ name: type }} | {{ result: json object }} |
| Assertions | where block after output | * result contains "pending" |
| Regex assertion | * name matches "pattern" | * version matches "^v\\d+" |
| Equality | * name != "" | * output != "" |
| Complex objects | Parenthesized boolean expressions | * (status.ok == true) or (status.ok == false) |
Variable Types
| Type | Use case |
|---|---|
string | Text output (version numbers, status codes) |
json object | JSON {} response (devices, config, agents) |
json array | JSON [] response (list endpoints) |
Fixtures
The fixture/ directory contains helper scripts that get copied to $CCTR_FIXTURE_DIR at runtime:
| Script | Purpose |
|---|---|
debug-cli | Executes OpenClaw CLI commands in sandbox via /debug/cli endpoint. Extracts JSON from mixed stdout. |
curl-auth | Authenticated curl against the deployed Worker URL |
server/start | Deploys Worker + Terraform infra to Cloudflare |
server/stop | Deletes Worker + destroys Terraform resources |
server/deploy | Deploys worker code + builds all dependencies |
server/wait-ready | Polls /api/health until the gateway is up |
start-browser | Launches Playwright Chromium for plwr tests |
stop-browser | Stops Playwright and saves video |
Example: CLI E2E Test
# f_cli_agents.txt — Agent CRUD via debug CLI endpoint
===
list agents from openclaw.json
===
WORKER_URL=$(cat "$CCTR_FIXTURE_DIR/worker-url.txt")
./curl-auth -s "$WORKER_URL/debug/container-config" | jq '.config.agents'
---
{{ result: json object }}
---
where
* result contains "list"
* result contains "defaults"
===
create a test agent via CLI
%require
===
WORKER_URL=$(cat "$CCTR_FIXTURE_DIR/worker-url.txt")
AGENT_ID="e2e-test-agent-$(date +%s)"
./debug-cli "openclaw agents add $AGENT_ID --workspace ... --json"
---
{{ result: json object }}
---
where
* result contains "id" or result contains "success"
===
delete test agent
===
./debug-cli "openclaw agents delete $AGENT_ID --force --json"
---
{{ result: json object }}
---
where
* result contains "id" or result contains "success" or (result contains "error" and result contains "not found")Example: Browser E2E Test (plwr)
# pairing_and_conversation.txt — Device approval + chat via browser
===
navigate to admin page
%require
===
if [ "${E2E_SKIP_BROWSER_TESTS:-}" = "1" ]; then
echo "Skipping browser test"
exit 0
fi
TOKEN=$(cat "$CCTR_FIXTURE_DIR/gateway-token.txt")
WORKER_URL=$(cat "$CCTR_FIXTURE_DIR/worker-url.txt")
plwr -S moltworker-e2e open "$WORKER_URL/_admin/?token=$TOKEN" -T 120000
===
wait for pending devices and approve
%require
===
plwr -S moltworker-e2e wait 'button:has-text("Approve All")' -T 120000
plwr -S moltworker-e2e click 'button:has-text("Approve All")' -T 10000
plwr -S moltworker-e2e wait-not 'button:has-text("Approv")' -T 30000
===
send math question and verify answer
===
plwr -S moltworker-e2e fill textarea 'What is 847293 + 651824? Reply with just the number.'
plwr -S moltworker-e2e press Enter
plwr -S moltworker-e2e wait 'text=1499117' -T 120000Running E2E via act (CI Simulation)
NixOS users can run CI workflows locally with act. The E2E job deploys real Cloudflare infrastructure (Worker + Terraform), runs tests, and destroys it.
# Prerequisites: create a GitHub personal access token (classic) with repo scope
# Store it and Cloudflare credentials in a .secrets file:
cat > .secrets << 'SECRETS'
GITHUB_TOKEN=ghp_your_personal_access_token
LERMA_API_TOKEN=your_lerma_api_token
E2E_CF_ACCOUNT_ID=your_account_id
E2E_WORKERS_SUBDOMAIN=your_subdomain
E2E_CF_ACCESS_TEAM_DOMAIN=yourteam.cloudflareaccess.com
E2E_R2_ACCESS_KEY_ID=your_r2_access_key
E2E_R2_SECRET_ACCESS_KEY=your_r2_secret_key
CLOUDFLARE_AI_GATEWAY_API_KEY=your_ai_gateway_key
CF_AI_GATEWAY_ACCOUNT_ID=your_account_id
CF_AI_GATEWAY_GATEWAY_ID=your_gateway_id
SECRETS
# Run the unit + lint workflow (no Cloudflare credentials needed beyond GITHUB_TOKEN)
act -W .github/workflows/test.yml -j unit
# Run the full E2E job (requires Cloudflare credentials)
act -W .github/workflows/test.yml -j e2e
# Run specific event (e2e only runs on develop branch by default, so simulate push to develop)
act -W .github/workflows/test.yml -j e2e -e <(echo '{"ref":"refs/heads/develop"}')
The
GITHUB_TOKENis needed becauseactclones GitHub Actions (actions/checkout, etc.) via HTTPS. Without a valid token, git operations fail with “authentication required.” Create a classic PAT withreposcope.
Running E2E Tests Directly (no act)
If you have Cloudflare credentials in a .dev.vars file, run cctr directly:
# 1. Copy E2E credentials
cp test/e2e/.dev.vars.example test/e2e/.dev.vars
# Edit .dev.vars with your Cloudflare credentials
# 2. Install tools
npx playwright install chromium # For browser tests
# cctr: cargo install cctr or brew install andreasjansson/tap/cctr
# plwr: see https://github.com/andreasjansson/plwr
# 3. Run tests
cctr test/e2e/ # All tests
cctr test/e2e/ -p pairing # Specific test file
cctr test/e2e/ -vv # Verbose (stream output)
PLAYWRIGHT_HEADED=1 cctr test/e2e/ # See browser during tests
E2E_SKIP_BROWSER_TESTS=1 cctr test/e2e/ # CLI tests onlyVitest E2E Pattern (for API routes)
For tests that don’t need real Cloudflare infra, use Vitest with a real Hono app instance:
import { Hono } from 'hono';
import { createMockEnv, suppressConsole } from '../src/test-utils';
function createTestApp() {
const app = new Hono<AppEnv>();
app.use('*', async (c, next) => {
c.set('userRole', 'ADMIN' as any);
c.set('customerSlug', 'test-customer');
c.env = { FOURIA_API_URL: 'https://api.example.com', ...c.env } as any;
await next();
});
app.route('/', myRoutes);
return app;
}
function makeRequest(app: Hono<AppEnv>, path: string, opts = {}) {
const env = createMockEnv(opts.env);
return app.request(path, { method: opts.method || 'GET' }, env);
}
describe('my endpoint', () => {
beforeEach(() => suppressConsole());
it('returns expected structure', async () => {
const res = await makeRequest(createTestApp(), '/');
expect(res.status).toBe(200);
const body = await res.json();
expect(body).toHaveProperty('expectedKey');
});
it('forwards to FourIA API with correct headers', async () => {
vi.stubGlobal(
'fetch',
vi.fn().mockResolvedValue(new Response(JSON.stringify({ result: 'ok' }), { status: 200 })),
);
const res = await makeRequest(createTestApp(), '/endpoint?param=value');
expect(res.status).toBe(200);
vi.unstubAllGlobals();
});
});Design Principles
-
Tests deploy real infra, run against it, destroy it. Each cctr suite provisions a live Worker and container. No mocks — tests validate against the actual Cloudflare environment (R2 mounts, cold starts, Access auth, network latency).
-
Sequential naming enforces order. Files run alphabetically. Use numeric prefixes (
a_,b_,c_) to build dependencies (gateway must be up before agent tests run). -
%requirefor hard gates. If the gateway isn’t reachable, skip all downstream tests. Use sparingly — only for prerequisites that invalidate all later tests. -
fixture/ scripts provide a stable CLI surface. The
debug-cliandcurl-authscripts abstract away URL/token management so test files focus on the scenario. -
Browser tests are skippable. Set
E2E_SKIP_BROWSER_TESTS=1to run only CLI tests. Every browser section checks this flag and exits 0 if set. -
Setup/teardown are tests too.
_setup.txtand_teardown.txtare full cctr files with assertions. If setup fails, the suite doesn’t run. Teardown always runs. -
Vitest E2E supplements cctr. For API-level integration tests that don’t need a live container (cost pipelines, OTel ingestion, RBAC enforcement), use Vitest with
createTestApp()and mockedfetch. These run fast, don’t require credentials, and validate the full middleware pipeline.
Deployment
bun run deploy:staging # Deploy to staging
bun run deploy # Deploy to production
Quick Debug Reference
# View live logs
npx wrangler tail
# Enter running container
docker exec -it <container_id> bash
# Inside container: check gateway
openclaw status
openclaw devices list --json
tail -f /tmp/openclaw-logs/*.log
# Check secrets
npx wrangler secret listNormal Boot Warnings
During cold start (first request), these errors are normal:
[ERROR] [api/status] Port wait failed: Process did not become ready within 5000ms
[ERROR] Uncaught ProcessReadyTimeoutError: Waiting for: port 18789 (TCP)
The gateway takes up to 3 minutes to become ready. Retries are automatic. If errors persist beyond 3 minutes, check container logs.