Agent Guidelines

Guidelines for AI agents working on this codebase.

Project Overview

This is a Cloudflare Worker that runs OpenClaw (formerly Moltbot/Clawdbot) in a Cloudflare Sandbox container. It provides:

• Proxying to the OpenClaw gateway (web UI + WebSocket)
• Admin UI at /_admin/ for device management
• API endpoints at /api/* for device pairing
• Debug endpoints at /debug/* for troubleshooting

Note: The CLI tool and npm package are now named openclaw. Config files use .openclaw/openclaw.json. Legacy .clawdbot paths are supported for backward compatibility during transition.

Paso4 Extensions (moltlazy/)

The moltlazy/ module is fully owned by the Paso4 developers and provides two systems:

1. Config Module (config/ + moltlazy/index.ts) — filesystem patcher used by start-openclaw.sh at container startup to write openclaw.json before the gateway launches.
2. Features Module (sdk/) — RPC-based config applied to a running gateway via the Admin HTTP API. Supported feature packs: Knowledge Graph, CF Unified Billing.

Project Structure

src/
├── index.ts          # Main Hono app, route mounting
├── types.ts          # TypeScript type definitions (AUTOGENERATED by `bun run types`, do not apply changes directly here, instead re-run the command after implementing the rest of the features)
├── config.ts         # Constants (ports, timeouts, paths)
├── auth/             # Cloudflare Access authentication
│   ├── jwt.ts        # JWT verification
│   ├── jwks.ts       # JWKS fetching and caching
│   └── middleware.ts # Hono middleware for auth
├── gateway/          # OpenClaw gateway management
│   ├── process.ts    # Process lifecycle (find, start)
│   ├── env.ts        # Environment variable building
│   ├── r2.ts         # R2 bucket mounting
│   ├── sync.ts       # R2 backup sync logic
│   └── utils.ts      # Shared utilities (waitForProcess)
├── routes/           # API route handlers
│   ├── api.ts        # /api/* endpoints (devices, gateway)
│   ├── admin.ts      # /_admin/* static file serving
│   └── debug.ts      # /debug/* endpoints
└── client/           # React admin UI (Vite)
    ├── App.tsx
    ├── api.ts        # API client
    └── pages/
moltlazy/             # Paso4 proprietary configuration utilities
├── cli.ts            # CLI entry point for patching config
├── index.ts          # Main logic for config patching
├── types.ts          # Type definitions for OpenClaw config
├── config/
│   └── include.ts     # per-section $include injection (session/tools/logging)
├── config/
├── config/           # Configuration templates
├── agents/           # Agent configuration templates
│   ├── defaults.ts   # Default agent settings
│   └── premade/      # Pre-configured agent definitions
├── models/           # Provider-specific model configurations
├── sdk/              # Features Module — RPC-based config for running gateway
│   ├── client.ts     # Typed RPC client over Admin HTTP API
│   ├── orchestrator.ts # Applies supported feature packs sequentially
│   └── feature-packs/
│       ├── knowledge-graph.ts         # Obsidian memory backend, wiki, QMD
│       └── plugin-cf-unified-billing.ts # CF AI Gateway unified billing
└── tests/            # Tests for configuration utilities

Key Patterns

Environment Variables

• DEV_MODE - Skips CF Access auth AND bypasses device pairing (maps to OPENCLAW_DEV_MODE for container)
• DEBUG_ROUTES - Enables /debug/* routes (disabled by default)
• See src/types.ts for full MoltbotEnv interface
• See docs/schemas/secrets-manifest.schema.json for the secrets manifest

CLI Commands

On OpenClaw 2026.8.2 the CLI no longer accepts --url ws://localhost:18789 (and most local commands reject --token too). Commands run inside the container (the gateway host) either operate on local state or auto-connect to the local Gateway — auth resolves from the OPENCLAW_GATEWAY_TOKEN env SecretRef, so no connection flags are passed. Commands that require an owner on a multi-agent roster take --agent <id>. Never hardcode the id: resolve it at runtime via openclaw agents list --json (prefer main when present, else the first configured agent — e.g. the dev-only fouria-builder/fouria-qa premades when running with DEV_MODE=true; see resolveAgentFlag() in src/routes/api/shared.ts and resolveCliAgentId() in moltlazy/sdk/cli.ts):

sandbox.startProcess('openclaw devices list --json');
// multi-agent commands need an explicit, dynamically-resolved owner:
sandbox.startProcess(`openclaw models list --json${await resolveAgentFlag(sandbox)}`);

Gateway-connecting commands (devices, cron, secrets reload) may still accept --url/--token when given together, but the 2026.8.2 guidance is to drop --url and let the CLI use the configured local target.

CLI commands take 10-15 seconds due to WebSocket connection overhead. Use waitForProcess() helper in src/routes/api.ts.

Success Detection

The CLI outputs “Approved” (capital A). Use case-insensitive checks:

stdout.toLowerCase().includes('approved');

Commands

bun run lint          # Run linting (oxlint)
bun run test          # Run tests (vitest)
bun run test:watch    # Run tests in watch mode
bun run test:gateway:boot     # Local gateway boot smoke test (Docker)
bun run test:gateway:boot:ai  # Same, with real AI Gateway onboard path (.dev.vars.e2e)
bun run build         # Build client (vite)
bun run build:all     # Build client + moltlazy + plugin
bun run build:docker  # Build all + install deps (pre-Docker build)
bun run deploy        # Build all and deploy to production
bun run deploy:dev    # Build all and deploy to dev
bun run dev           # Vite dev server
bun run start         # wrangler dev (local worker)
bun run typecheck     # TypeScript check

Contribution Guidelines

Validation Requirements

Before submitting any contribution:

1. Run lint and tests - All builds must pass:

   bun run lint
   bun run test

2. Environment variable changes - If your contribution adds, updates, or deletes an environment variable, you MUST update it in all three locations:

   • README.md - Add to the secrets reference table
   • src/gateway/env.ts - Add to buildEnvVars() function if passed to container
   • .dev.vars.example - Add to the mapping comments and example

Testing

Tests use Vitest. Test files are colocated with source files (*.test.ts).

Current test coverage:

• auth/jwt.test.ts - JWT decoding and validation
• auth/jwks.test.ts - JWKS fetching and caching
• auth/middleware.test.ts - Auth middleware behavior
• gateway/env.test.ts - Environment variable building
• gateway/process.test.ts - Process finding logic
• gateway/r2.test.ts - R2 mounting logic
• gateway/sync.test.ts - R2 backup sync logic
• moltlazy/tests/index.test.ts - per-section $include injection and config generation

When adding new functionality, add corresponding tests.

Code Style

• Use TypeScript strict mode
• Prefer explicit types over inference for function signatures
• Keep route handlers thin - extract logic to separate modules
• Use Hono’s context methods (c.json(), c.html()) for responses

Documentation

• README.md - User-facing documentation (setup, configuration, usage)
• AGENTS.md - This file, for AI agents

Development documentation goes in AGENTS.md, not README.md.

---

Architecture

Browser
   │
   ▼
┌─────────────────────────────────────┐
│     Cloudflare Worker (index.ts)    │
│  - Starts OpenClaw in sandbox       │
│  - Proxies HTTP/WebSocket requests  │
│  - Passes secrets as env vars       │
└──────────────┬──────────────────────┘
               │
               ▼
┌─────────────────────────────────────┐
│     Cloudflare Sandbox Container    │
│  ┌───────────────────────────────┐  │
│  │     OpenClaw Gateway          │  │
│  │  - Control UI on port 18789   │  │
│  │  - WebSocket RPC protocol     │  │
│  │  - Agent runtime              │  │
│  └───────────────────────────────┘  │
└─────────────────────────────────────┘

Key Files

File	Purpose
src/index.ts	Worker that manages sandbox lifecycle and proxies requests
src/metrics.ts	Cost and usage metrics instrumentation for Analytics Engine
Dockerfile	Container image based on cloudflare/sandbox with Node 22 + OpenClaw
start-openclaw.sh	Startup script: R2 restore → onboard → config patch → launch gateway
wrangler.jsonc	Cloudflare Worker + Container configuration

Cost Observability

The Worker instruments cost metrics using Cloudflare Analytics Engine for per-client cost tracking:

Event Type	Dimensions	Metrics
ai-query	model, provider, customer, actionType	tokensIn, tokensOut, costUsd
container-active	customer, instanceType, action	heartbeat count
r2-operation	customer, opType, opClass	bytesTransferred, durationMs
cron-trigger	customer, triggerReason	wokeContainer flag
ws-session	customer, sessionId	durationSeconds, messagesCount

Analytics Engine Configuration

The ANALYTICS binding writes to dataset fourai_cost_metrics. Query via GraphQL:

query CostMetrics($accountTag: String!) {
  viewer {
    accounts(filter: { accountTag: $accountTag }) {
      analyticsEngineDatasets(filter: { name: "fourai_cost_metrics" }) {
        data(
          query: "SELECT blob1 as event_type, blob2 as customer, blob3 as model, SUM(double1) as tokens_in, SUM(double2) as tokens_out FROM fourai_cost_metrics WHERE timestamp > NOW() - INTERVAL 1 DAY GROUP BY blob1, blob2, blob3"
        ) {
          result
        }
      }
    }
  }
}

Cost API Endpoint

GET /api/admin/costs returns pricing constants and query examples for cost dashboards.

Local Development

npm install
cp .dev.vars.example .dev.vars
# Edit .dev.vars with your CLOUDFLARE_AI_GATEWAY_API_KEY + CLOUDFLARE_API_TOKEN + CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID
npm run start

Environment Variables

For local development, create .dev.vars:

CLOUDFLARE_AI_GATEWAY_API_KEY=your-provider-api-key
CLOUDFLARE_API_TOKEN=your-ai-scoped-api-token  # REST transport; use the same AI-scoped credential locally
CF_AI_GATEWAY_ACCOUNT_ID=your-account-id
CF_AI_GATEWAY_GATEWAY_ID=your-gateway-id
DEV_MODE=true           # Skips CF Access auth + device pairing
DEBUG_ROUTES=true       # Enables /debug/* routes

WebSocket Limitations

Local development with wrangler dev has issues proxying WebSocket connections through the sandbox. HTTP requests work but WebSocket connections may fail. Deploy to Cloudflare for full functionality.

Docker Image Caching

The Dockerfile includes a cache bust comment. When changing start-openclaw.sh, bump the version:

# Build cache bust: 2026-02-06-v28-openclaw-upgrade

Gateway Configuration

OpenClaw configuration is built at container startup:

1. R2 backup is restored if available (with migration from legacy .clawdbot paths)
2. If no config exists, openclaw onboard --non-interactive creates one based on env vars
3. Feature flags come from the per-customer MOLTLAZY_FEATURE_FLAGS binding (managed via the FourIA API)
4. start-openclaw.sh runs moltlazy patch which generates per-section include files (moltlazy-session/tools/logging.json) and injects them as per-section $include directives (no root $include); gateway.*/agents.* are written inline and the gateway token is an env SecretRef
5. Gateway starts with openclaw gateway --allow-unconfigured --bind lan

Feature Flag Configuration

Feature flags are managed via the FourIA API and applied as a per-customer binding (MOLTLAZY_FEATURE_FLAGS, a JSON string). The Worker reads the binding and passes it to the container via buildEnvVars(). The legacy CONFIG_BUCKET / config-cache.ts (R2 + ETag) mechanism was removed.

Feature Flags Example:

{
  "knowledgeGraph": true,
  "unifiedBilling": false
}

The container consumes MOLTLAZY_FEATURE_FLAGS at startup (moltlazy patch generates the per-section include files and injects them as per-section $include into openclaw.json). There is no worker-side cache — the binding is read directly.

AI Provider

Cloudflare AI Gateway is the only provisioned AI provider path. The startup script configures auth when all of these env vars are set:

1. Cloudflare AI Gateway (native): CLOUDFLARE_AI_GATEWAY_API_KEY + CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID

Container Environment Variables

These are the env vars passed TO the container (internal names):

Variable	Config Path	Notes
CLOUDFLARE_AI_GATEWAY_API_KEY	(env var)	Native AI Gateway key
CLOUDFLARE_API_TOKEN	(env var)	AI Gateway REST API Bearer token; tenant binding is derived from the AI key
CF_AI_GATEWAY_ACCOUNT_ID	(env var)	Account ID for AI Gateway
CF_AI_GATEWAY_GATEWAY_ID	(env var)	Gateway ID for AI Gateway
OPENCLAW_GATEWAY_TOKEN	--token flag	Mapped from MOLTBOT_GATEWAY_TOKEN
OPENCLAW_DEV_MODE	controlUi.allowInsecureAuth	Mapped from DEV_MODE
TELEGRAM_BOT_TOKEN	channels.telegram.botToken	@deprecated — E2E only. Use integrations dashboard in production.
DISCORD_BOT_TOKEN	channels.discord.token	@deprecated — E2E only. Use integrations dashboard in production.
SLACK_BOT_TOKEN	channels.slack.botToken	@deprecated — E2E only. Use integrations dashboard in production.
SLACK_APP_TOKEN	channels.slack.appToken	@deprecated — E2E only. Use integrations dashboard in production.

OpenClaw Config Schema

OpenClaw has strict config validation. Common gotchas:

• agents.defaults.model must be { "primary": "model/name" } not a string
• gateway.mode must be "local" for headless operation
• No webchat channel - the Control UI is served automatically
• gateway.bind is not a config option - use --bind CLI flag

See OpenClaw docs for full schema.

Common Tasks

Adding a New API Endpoint

1. Add route handler in src/routes/api.ts
2. Add types if needed in src/types.ts
3. Update client API in src/client/api.ts if frontend needs it
4. Add tests

Adding a New Environment Variable

1. Add to MoltbotEnv interface in src/types.ts
2. If passed to container, add to buildEnvVars() in src/gateway/env.ts
3. Update .dev.vars.example
4. Document in README.md secrets table

Debugging

# View live logs
npx wrangler tail

# Check secrets
npx wrangler secret list

Enable debug routes with DEBUG_ROUTES=true and check /debug/processes.

R2 Backup Encryption (F4E1)

When BACKUP_ENCRYPTION_KEY (base64 32-byte secret) is set, the Worker post-encrypts every snapshot after sandbox.createBackup(): backups/<id>/data.sqsh is replaced by data.sqsh.enc (chunked AES-256-GCM, per-object HKDF key) and meta.json is flagged encrypted: true (kept readable — the SDK needs it). Restores decrypt transiently and delete the plaintext in a finally block. Config versions under configs/ are encrypted with customMetadata.encrypted='true'. Legacy plaintext backups remain restorable. Set BACKUP_ENCRYPTION_REQUIRED=true to fail closed. Crypto lives in src/crypto/. See docs/plans/r2-encryption-at-rest.md.

R2 Storage Notes

Persistence uses the Sandbox SDK createBackup() / restoreBackup() API with the BACKUP_BUCKET binding. The snapshot root is /home/openclaw; it is not an s3fs mount and there is no /data/moltbot runtime path.

Never delete tenant backup objects outside the retention or decommission procedure. Hard restarts must checkpoint before killing the gateway.

• Process status: The sandbox API’s proc.status may not update immediately after a process completes. Instead of checking proc.status === 'completed', verify success by checking for expected output (e.g., timestamp file exists after sync).

• R2 prefix migration: Backups are now stored under openclaw/ prefix in R2 (was clawdbot/). The startup script handles restoring from both old and new prefixes with automatic migration.

Cloudflare Tunnel Setup (Manual Flow)

This section documents how to expose an internal service (e.g. the Obsidian sync port 18790) via a Cloudflare Tunnel with a public hostname. Follow this end-to-end when setting up a new tunnel route.

1. Register and Add a Domain

You need a domain zone in your Cloudflare account before any DNS record can be created.

1. Register a domain (e.g. via Cloudflare Registrar or a third-party registrar like Spaceship/Namecheap).
2. In the Cloudflare Dashboard, select the Paso4 account → Websites → Add a site.
3. Enter the domain name and choose the Free plan.
4. Cloudflare will assign two nameservers, e.g.:
   • albert.ns.cloudflare.com
   • priscilla.ns.cloudflare.com
5. At your registrar, replace the existing nameservers with the Cloudflare ones.
6. Wait for DNS propagation (typically 5–30 minutes). The zone status will change from pending to active.

Current domain: paso4.io (zone ID: 6ce23f544f57a1909578032abcda18dd, account: Paso4)

2. Create the Cloudflare Tunnel

If the tunnel does not yet exist, create it with cloudflared:

cloudflared tunnel create obsidian-sync

This prints a tunnel UUID and writes credentials to ~/.cloudflared/<UUID>.json.

Current tunnel: obsidian-sync — UUID 90792969-7077-40a8-ae10-28e716290d9d

To get the tunnel token (needed for CLOUDFLARE_TUNNEL_TOKEN):

cloudflared tunnel token 90792969-7077-40a8-ae10-28e716290d9d

Set it as a Worker secret:

npx wrangler secret put CLOUDFLARE_TUNNEL_TOKEN

3. Create the DNS CNAME Record

A CNAME record must point from your public subdomain to <UUID>.cfargotunnel.com.

Via Cloudflare Dashboard:

1. Go to paso4.io → DNS → Records → Add record.

2. Fill in:

   Field	Value
   Type	CNAME
   Name	obsidian-sync
   Target	90792969-7077-40a8-ae10-28e716290d9d.cfargotunnel.com
   Proxy status	Proxied (orange cloud — required)
   TTL	Auto

3. Save. Public URL becomes https://obsidian-sync.paso4.io.

Via MCP API (requires DNS:Edit on the API token):

cloudflare.request({
  method: 'POST',
  path: `/zones/6ce23f544f57a1909578032abcda18dd/dns_records`,
  body: {
    type: 'CNAME',
    name: 'obsidian-sync',
    content: '90792969-7077-40a8-ae10-28e716290d9d.cfargotunnel.com',
    proxied: true,
    comment: 'Cloudflare Tunnel route for obsidian-sync',
  },
});

To grant DNS:Edit to the MCP token: https://dash.cloudflare.com/profile/api-tokens → edit the token → add Zone → DNS → Edit.

4. Configure the Tunnel Ingress

Create or update ~/.cloudflared/config.yml on the host running cloudflared:

tunnel: 90792969-7077-40a8-ae10-28e716290d9d
credentials-file: /root/.cloudflared/90792969-7077-40a8-ae10-28e716290d9d.json

ingress:
  - hostname: obsidian-sync.paso4.io
    service: http://localhost:18790
  - service: http_status:404

Then run:

cloudflared tunnel run obsidian-sync

5. Verify

# Check tunnel is connected
cloudflared tunnel info obsidian-sync

# Test the public endpoint
curl -I https://obsidian-sync.paso4.io

Gotchas

• The cfargotunnel.com CNAME only routes traffic for DNS records in the same Cloudflare account as the tunnel. Cross-account routing is blocked.
• The zone must be active (nameservers delegated) for public DNS to resolve. A pending zone means nameserver propagation is not complete yet.
• If the tunnel stops, the DNS record is not deleted — visitors will see a Cloudflare 1016 error.
• The CLOUDFLARE_TUNNEL_TOKEN secret in the Worker is used by start-openclaw.sh to run cloudflared inside the container, exposing port 18790 to the tunnel.

---

FourIA offers Private Cloud only (Business On Premise is not offered at this time). Pricing is credit-based: Credits are the exclusive billable unit, consumed by completed messages and automations. Credits do NOT include AI costs — AI expenses are managed separately via BYOL or AI Gateway billing. Reader users are unlimited and free. Builder seats (users who can create and edit agents) are plan-limited. The v1.0-beta is delivered through the Design Partner Program with a single engaged partner. Partners pay a fixed credit fee negotiated with sales. See docs/VISION.md for the full pricing definition.

Active Technologies

• OpenTofu 1.6+ (HCL) + cloudflare provider (Terraform/OpenTofu) (001-opentofu-deploy)
• Cloudflare R2 (for FourIA data), OpenTofu Local/Remote State (for infrastructure tracking) (001-opentofu-deploy)

Recent Changes

• v0.3.x $include Configuration: immutable per-section $include files (session/tools/logging); gateway/agents inline; gateway token via env SecretRef
• Gateway restart split: POST /api/admin/gateway/restart is the soft reload — sends SIGUSR1 to the live openclaw gateway run process (in-process, no connection drop, no port rebind) via softReloadGateway and falls back to signalRestoreNeeded only when no live gateway is found; POST /debug/restart-openclaw (DEBUG_ROUTES=true, ROOT) is the hard restart — killGateway + resetStartupCache + signalRestoreNeeded + start-openclaw.sh reprovision (in-place, no startGatewayDirect) so existing client tokens keep working
• OPENCLAW_NO_RESPAWN=1 is set in the Dockerfile so the gateway performs in-process restarts (no orphan/respawn loops); softReloadGateway only signals live processes and skips zombies
• Stale sandbox recovery (src/gateway/sandbox-recovery.ts): when the platform terminates the container behind a live Sandbox DO, every RPC fails with sandbox.exec error … HTTP error! status: 500 — isStaleContainerError classifies these and recoverStaleSandbox destroys the dead instance + resets startup/persistence caches so the next RPC auto-provisions a fresh container; wired into findExistingGatewayProcess, checkGatewayHealth and the ensureGateway startProcess retry; cooldown-guarded (60s), never fires for deploy OperationInterruptedError or transient provisioning errors
• Dev-only Fouria agents + client docs: moltlazy ships no production premade agents (vanilla researcher/coder removed) and injects Fouria Builder (fouria-builder) + Fouria Q&A agent (fouria-qa) only when OPENCLAW_DEV_MODE=true; per-tenant client documentation lives in R2 under customers/<slug>/docs/ and is materialized by the Worker into /home/openclaw/clawd/client-docs/ before gateway start (src/services/fouria-docs/, materializeClientDocsIfDev), with a dev-only admin API at /api/admin/fouria-docs (404 outside dev); the Dockerfile installs the cctr binary, platform docs are fetched from the docs site (https://fouria.io, fallback https://fouria-docs.pages.dev, starting at /llms.txt), and start-openclaw.sh keeps security.installPolicy allow in dev so the Builder can self-install plugins
• Orphaned-instance recovery (src/gateway/sandbox-recovery.ts): a container destroyed out-of-band (Cloudflare Containers dashboard “Destroy”, failed force-stop) leaves the DO alive pointing at a dead placement and surfaces the SDK’s retryable admission errors (no container instance available, max container instances exceeded) instead of the 500 signature. isContainerUnavailableError + trackProvisioningFailure escalate to a recovery only once the streak reaches ORPHANED_INSTANCE_MIN_FAILURES and spans ORPHANED_INSTANCE_GRACE_MS (120s, > the SDK’s ≈110s provisioning budget, so slow-but-healthy cold starts are never cancelled); maybeRecoverSandbox unifies both failure classes and clears the streak whenever a sandbox RPC succeeds. Operator lever: POST /debug/reprovision (DEBUG_ROUTES, ROOT) destroys the instance + clears caches + resets the streak without touching the gateway, so it works when the container is entirely gone
• Activity-driven wake (src/activity/): automations run inside the Gateway, so it must be running for schedules to fire. A per-tenant ActivityScheduler Durable Object (binding ACTIVITY, SQLite-backed, separate from the SDK Sandbox DO whose alarm drives sleepAfter) mirrors the tenant’s schedule (openclaw automations list --json, refreshed on warm /api/health when readiness.ok, throttled to 5 min) and arms one alarm CRON_WAKE_AHEAD_MINUTES (default 10) before the next enabled job. On fire it wakes + heals via ensureGateway (recovering an orphaned instance) then re-arms; no enabled jobs → alarm cleared → container sleeps. Healing/wakeup is independent of user traffic, gated on job activity, and free when idle (src/activity/alarm.ts state machine is unit-tested; the DO is a thin adapter). Requires the ACTIVITY binding in wrangler.jsonc + lerma (tenant_provisioner.ex resource_bindings, dispatch_namespace.ex @do_sqlite_classes)