# Agent Guidelines

Guidelines for AI agents working on this codebase.

## Project Overview

This is a Cloudflare Worker that runs [OpenClaw](https://github.com/openclaw/openclaw) (formerly Moltbot/Clawdbot) in a Cloudflare Sandbox container. It provides:

- Proxying to the OpenClaw gateway (web UI + WebSocket)
- Admin UI at `/_admin/` for device management
- API endpoints at `/api/*` for device pairing
- Debug endpoints at `/debug/*` for troubleshooting

**Note:** The CLI tool and npm package are now named `openclaw`. Config files use `.openclaw/openclaw.json`. Legacy `.clawdbot` paths are supported for backward compatibility during transition.

### Paso4 Extensions (`moltlazy/`)

The `moltlazy/` module is fully owned by the **Paso4 developers** and provides two systems:

1. **Config Module** (`config/` + `moltlazy/index.ts`) — filesystem patcher used by `start-openclaw.sh` at container startup to write `openclaw.json` before the gateway launches.
2. **Features Module** (`sdk/`) — RPC-based config applied to a **running** gateway via the Admin HTTP API. Supported feature packs: Knowledge Graph, CF Unified Billing.

## Project Structure

```
src/
├── index.ts          # Main Hono app, route mounting
├── types.ts          # TypeScript type definitions (AUTOGENERATED by `bun run types`, do not apply changes directly here, instead re-run the command after implementing the rest of the features)
├── config.ts         # Constants (ports, timeouts, paths)
├── auth/             # Cloudflare Access authentication
│   ├── jwt.ts        # JWT verification
│   ├── jwks.ts       # JWKS fetching and caching
│   └── middleware.ts # Hono middleware for auth
├── gateway/          # OpenClaw gateway management
│   ├── process.ts    # Process lifecycle (find, start)
│   ├── env.ts        # Environment variable building
│   ├── r2.ts         # R2 bucket mounting
│   ├── sync.ts       # R2 backup sync logic
│   └── utils.ts      # Shared utilities (waitForProcess)
├── routes/           # API route handlers
│   ├── api.ts        # /api/* endpoints (devices, gateway)
│   ├── admin.ts      # /_admin/* static file serving
│   └── debug.ts      # /debug/* endpoints
└── client/           # React admin UI (Vite)
    ├── App.tsx
    ├── api.ts        # API client
    └── pages/
moltlazy/             # Paso4 proprietary configuration utilities
├── cli.ts            # CLI entry point for patching config
├── index.ts          # Main logic for config patching
├── types.ts          # Type definitions for OpenClaw config
├── config/
│   └── include.ts     # per-section $include injection (session/tools/logging)
├── config/
├── config/           # Configuration templates
├── agents/           # Agent configuration templates
│   ├── defaults.ts   # Default agent settings
│   └── premade/      # Pre-configured agent definitions
├── models/           # Provider-specific model configurations
├── sdk/              # SDK Module — typed RPC client over the Admin HTTP API
│   ├── client.ts     # Typed RPC client
│   └── (agents, channels, config, cron, devices, models, gateway, update, …)
└── tests/            # Tests for configuration utilities
```

## Key Patterns

### Environment Variables

- `DEV_MODE` - Local-dev switch: skips CF Access auth AND bypasses device pairing
- `OPENCLAW_DEV_MODE` - Explicit override for the dev-agents flag. Normally derived by the Worker from the instance's own `-dev` hostname suffix (see `src/dev-mode.ts`), which enables the Fouria Builder/Q&A agents, client-doc materialization, and the opencode-credits automation **without** bypassing auth. Forwarded to the container as `OPENCLAW_DEV_MODE`; falls back to `DEV_MODE` for local development
- `DEBUG_ROUTES` - Enables `/debug/*` routes (disabled by default)
- See `src/types.ts` for full `MoltbotEnv` interface
- See [`docs/schemas/secrets-manifest.schema.json`](./docs/schemas/secrets-manifest.schema.json) for the secrets manifest

### CLI Commands

On OpenClaw 2026.8.2 the CLI **no longer accepts** `--url ws://localhost:18789`
(and most local commands reject `--token` too). Commands run inside the container
(the gateway host) either operate on local state or auto-connect to the local
Gateway — auth resolves from the `OPENCLAW_GATEWAY_TOKEN` env SecretRef, so no
connection flags are passed. Commands that require an owner on a multi-agent
roster take `--agent <id>`. Never hardcode the id: resolve it at runtime via
`openclaw agents list --json` (prefer `main` when present, else the first
configured agent — e.g. the dev-only `fouria-builder`/`fouria-qa` premades when
running with `DEV_MODE=true`; see
`resolveAgentFlag()` in `src/routes/api/shared.ts` and `resolveCliAgentId()`
in `moltlazy/sdk/cli.ts`):

```typescript
// Sandbox SDK 1.0: exec takes argv and resolves on launch, not completion.
startShell(sandbox, 'openclaw devices list --json');
// multi-agent commands need an explicit, dynamically-resolved owner:
startShell(sandbox, `openclaw models list --json${await resolveAgentFlag(sandbox)}`);
```

Gateway-connecting commands (`devices`, `cron`, `secrets reload`) may still accept
`--url`/`--token` when given together, but the 2026.8.2 guidance is to drop `--url`
and let the CLI use the configured local target.

CLI commands take 10-15 seconds due to WebSocket connection overhead. Use `waitForProcess()` helper in `src/routes/api.ts`.

### Success Detection

The CLI outputs "Approved" (capital A). Use case-insensitive checks:

```typescript
stdout.toLowerCase().includes('approved');
```

## Commands

```bash
bun run lint          # Run linting (oxlint)
bun run test          # Run tests (vitest)
bun run test:watch    # Run tests in watch mode
bun run test:client   # Run SPA component/page tests (jsdom, vitest.client.config.ts)
bun run test:ui       # Run the Playwright UI suite (Vite dev server + mocked API)
bun run test:ui:headed # Same, with a visible browser
bun run test:ui:render:local # Local-only plwr admin-UI component render timings
bun run test:fuzz:sandbox-v1  # Seeded F4E1 codec fuzz harness ([iterations] [seed])
bun run test:gateway:boot     # Local gateway boot smoke test (Docker)
bun run test:gateway:boot:ai  # Same, with real AI Gateway onboard path (.dev.vars.e2e)
bun run test:software:smoke   # Declared-software manifest → skills → restore smoke (Docker)
bun run build         # Build client (vite)
bun run build:all     # Build client + moltlazy
bun run build:docker  # Build all + install deps (pre-Docker build)
bun run deploy        # Build all and deploy to production
bun run deploy:dev    # Build all and deploy to dev
bun run dev           # Admin SPA Vite dev server (port 8787)
bun run dev:worker    # Worker dev server (Cloudflare Vite plugin, port 8788)
bun run start         # Build all + Worker dev server
bun run typecheck     # TypeScript check
```

## Contribution Guidelines

### Validation Requirements

Before submitting any contribution:

1. **Run lint and tests** - All builds must pass:

   ```bash
   bun run lint
   bun run test
   ```

2. **Environment variable changes** - If your contribution adds, updates, or deletes an environment variable, you MUST update it in all three locations:
   - `README.md` - Add to the secrets reference table
   - `src/gateway/env.ts` - Add to `buildEnvVars()` function if passed to container
   - `.dev.vars.example` - Add to the mapping comments and example

### Testing

Tests use Vitest. Test files are colocated with source files (`*.test.ts`).

Current test coverage:

- `auth/jwt.test.ts` - JWT decoding and validation
- `auth/jwks.test.ts` - JWKS fetching and caching
- `auth/middleware.test.ts` - Auth middleware behavior
- `gateway/env.test.ts` - Environment variable building
- `gateway/process.test.ts` - Process finding logic
- `gateway/r2.test.ts` - R2 mounting logic
- `gateway/sync.test.ts` - R2 backup sync logic
- `moltlazy/tests/index.test.ts` - per-section $include injection and config generation

When adding new functionality, add corresponding tests.

The admin SPA also has a Playwright UI suite (`test/ui/`, config
`playwright.config.ts`) run with `bun run test:ui`. It serves the real dashboard
via the Vite dev server (`bun run dev`) and mocks every `/api/*` call per test,
so it needs no Worker, container, or Cloudflare credentials. It covers the app
shell, role-based navigation/redirects, every admin page's render, and the
notification bell. On NixOS the browsers come from `PLAYWRIGHT_BROWSERS_PATH`
(see `flake.nix`); `playwright.config.ts` selects the newest `chromium-*` there
so the Nix bundle can lag the npm `playwright` version without breaking the run.

A separate **local-only** cctr+`plwr` suite (`test/e2e/ui_components_render.txt`,
`bun run test:ui:render:local`) validates the admin (root) UI components against
the local worker and records their render times (`openMs`/`componentMs`/`waitMs`
plus Navigation Timing/FCP) via `fixture/ui-render-measure`; it self-skips
outside `E2E_LOCAL=true`. See [docs/TESTING.md](/staging/llms/platform/reference/testing/index.md).

## Code Style

- Use TypeScript strict mode
- Prefer explicit types over inference for function signatures
- Keep route handlers thin - extract logic to separate modules
- Use Hono's context methods (`c.json()`, `c.html()`) for responses

## Documentation

- `README.md` - User-facing documentation (setup, configuration, usage)
- `AGENTS.md` - This file, for AI agents

Development documentation goes in AGENTS.md, not README.md.

---

## Architecture

```
Browser
   │
   ▼
┌─────────────────────────────────────┐
│     Cloudflare Worker (index.ts)    │
│  - Starts OpenClaw in sandbox       │
│  - Proxies HTTP/WebSocket requests  │
│  - Passes secrets as env vars       │
└──────────────┬──────────────────────┘
               │
               ▼
┌─────────────────────────────────────┐
│     Cloudflare Sandbox Container    │
│  ┌───────────────────────────────┐  │
│  │     OpenClaw Gateway          │  │
│  │  - Control UI on port 18789   │  │
│  │  - WebSocket RPC protocol     │  │
│  │  - Agent runtime              │  │
│  └───────────────────────────────┘  │
└─────────────────────────────────────┘
```

### Key Files

| File                | Purpose                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `src/index.ts`      | Worker that manages sandbox lifecycle and proxies requests                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `src/metrics.ts`    | Cost and usage metrics instrumentation for Analytics Engine                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `Dockerfile`        | Container image based on `cloudflare/sandbox` with Node 22 + OpenClaw                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `start-openclaw.sh` | Startup entry point: `start` is the production cold-start (acquire the container-wide startup lock **before** setup, hold fd 9 through the Gateway `exec` — a single owner covers setup **and** launch), `post-start` (dev automation + boot-log reset), `setup`/`launch` (standalone smoke/manual modes only), or `full` (legacy: setup + background gateway launch). The Worker drives production cold-start as one `start-openclaw.sh start` call then a **detached** `post-start` (never awaited, so it cannot block readiness); it no longer invokes `setup`/`launch` separately. `setup`/`post-start` must stay child-free (no `tail -f`) — the Worker `waitForExit`s on them. `start`/`setup`/`full` serialize the slow setup body on the exclusive `/tmp/openclaw-setup.lock`, and `start` holds it through the Gateway `exec` (no release before launch); the standalone `launch` takes the dedicated `/tmp/openclaw-gateway-launch.lock`, held for the gateway's whole lifetime (fd inherited by the `exec`'d gateway). Either way exactly one `openclaw gateway run` owner can exist — losers poll `/health` and exit 0 without launching a duplicate; the serial `test/e2e/ac_gateway_startup_concurrency.txt` asserts one owner and the absence of the migration-input-change and SQLite offline-maintenance errors. When the Worker passes `OPENCLAW_RESTORED=true` (a backup was restored), the setup body is skipped entirely and the restored state is launched directly. See [docs/WORKER-DEVELOPMENT.md](/staging/llms/platform/apps/fouria/worker-development/index.md) |
| `wrangler.jsonc`    | Cloudflare Worker + Container configuration                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |

## Cost Observability

The Worker instruments cost metrics using Cloudflare Analytics Engine for per-client cost tracking:

| Event Type         | Dimensions                            | Metrics                        |
| ------------------ | ------------------------------------- | ------------------------------ |
| `ai-query`         | model, provider, customer, actionType | tokensIn, tokensOut, costUsd   |
| `container-active` | customer, instanceType, action        | heartbeat count                |
| `r2-operation`     | customer, opType, opClass             | bytesTransferred, durationMs   |
| `cron-trigger`     | customer, triggerReason               | wokeContainer flag             |
| `ws-session`       | customer, sessionId                   | durationSeconds, messagesCount |

### Analytics Engine Configuration

The `ANALYTICS` binding writes to dataset `fourai_cost_metrics`. Query via GraphQL:

```graphql
query CostMetrics($accountTag: String!) {
  viewer {
    accounts(filter: { accountTag: $accountTag }) {
      analyticsEngineDatasets(filter: { name: "fourai_cost_metrics" }) {
        data(
          query: "SELECT blob1 as event_type, blob2 as customer, blob3 as model, SUM(double1) as tokens_in, SUM(double2) as tokens_out FROM fourai_cost_metrics WHERE timestamp > NOW() - INTERVAL 1 DAY GROUP BY blob1, blob2, blob3"
        ) {
          result
        }
      }
    }
  }
}
```

### Cost API Endpoint

GET `/api/admin/costs` returns pricing constants and query examples for cost dashboards.

## Local Development

```bash
npm install
cp .dev.vars.example .dev.vars
# Edit .dev.vars with your CLOUDFLARE_AI_GATEWAY_API_KEY + CLOUDFLARE_API_TOKEN + CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID
npm run start
```

### Environment Variables

For local development, create `.dev.vars`:

```bash
CLOUDFLARE_AI_GATEWAY_API_KEY=your-provider-api-key
CLOUDFLARE_API_TOKEN=your-ai-scoped-api-token  # REST transport; use the same AI-scoped credential locally
CF_AI_GATEWAY_ACCOUNT_ID=your-account-id
CF_AI_GATEWAY_GATEWAY_ID=your-gateway-id
DEV_MODE=true           # Skips CF Access auth + device pairing
DEBUG_ROUTES=true       # Enables /debug/* routes
```

### WebSocket Limitations

Local development with the Vite dev server has issues proxying WebSocket connections through the sandbox. HTTP requests work but WebSocket connections may fail. Deploy to Cloudflare for full functionality.

## Docker Image Caching

The Dockerfile includes a cache bust comment. When changing `start-openclaw.sh`, bump the version:

```dockerfile
# Build cache bust: 2026-02-06-v28-openclaw-upgrade
```

## Gateway Configuration

OpenClaw configuration is built at container startup:

1. R2 backup is restored if available (with migration from legacy `.clawdbot` paths)
2. If no config exists, `openclaw onboard --non-interactive` creates one based on env vars
3. Feature flags come from the per-customer `MOLTLAZY_FEATURE_FLAGS` binding (managed via the FourIA API)
4. `start-openclaw.sh start` runs `moltlazy patch` (after the restore/onboard above) which generates per-section include files (`moltlazy-session/tools/logging.json`) and injects them as per-section `$include` directives (no root `$include`); `gateway.*`/`agents.*` are written inline and the gateway token is an env SecretRef
5. The Worker calls `/usr/local/bin/start-openclaw.sh start` **once**: that mode acquires the container-wide startup lock **before** running the setup body, runs the patch while holding it, then `exec`s the supervised Gateway under the same lock (the `exec` replaces the wrapper, so the tracked process _is_ the gateway) and runs `start-openclaw.sh post-start` afterward once the port is ready. The Worker no longer invokes `setup` then `launch` separately; those remain standalone smoke/manual modes only.

### Feature Flag Configuration

Feature flags are managed via the **FourIA API** and applied as a **per-customer binding** (`MOLTLAZY_FEATURE_FLAGS`, a JSON string). The Worker reads the binding and passes it to the container via `buildEnvVars()`. The legacy `CONFIG_BUCKET` / `config-cache.ts` (R2 + ETag) mechanism was removed.

**Feature Flags Example:**

```json
{
  "knowledgeGraph": true,
  "unifiedBilling": false,
  "softwareProvisioning": false
}
```

The container consumes `MOLTLAZY_FEATURE_FLAGS` at startup (`moltlazy patch` generates the per-section include files and injects them as per-section `$include` into `openclaw.json`). There is no worker-side cache — the binding is read directly.

### Software provisioning (`softwareProvisioning`)

Software that agent skills install at runtime (plugins, `apt`/`npm` packages,
downloaded binaries) lives in the container filesystem only, so it is lost on a
container-version upgrade (fresh image) or idle-sleep (container destroyed).

The per-customer `softwareProvisioning` flag fixes this:

1. The tenant's **manifest** (`customers/<slug>/software.json` in the
   `BACKUP_BUCKET`) declares each
   package with its actual install command: `{ id, check, install, verify?, timeoutSec? }`.
   Operators write it via `PUT /api/admin/software/manifest` (Access-protected);
   the dev-only Fouria Builder writes it through the sandbox
   `software.r2.internal` bridge (`src/sandbox-software-bridge.ts`) instead,
   because `/api/admin/*` answers `302` to the Access login when called from the
   container. Bridge writes are dev-only; reads work everywhere.
2. On cold start, `ensureGatewayImpl` materializes the manifest into
   `/home/openclaw/.openclaw/software.json` and — only when the flag is on —
   spawns **`moltlazy software provision`** as an **independent supervised process
   through the Sandbox SDK** (`void spawnSoftwareProvision(...)`), not a step in
   `start-openclaw.sh`. It runs in parallel with the gateway boot, never blocks
   readiness, and is non-fatal (per-package timeouts; failures recorded, not raised).
3. It writes live ticks to `/tmp/software-progress.log` (read by
   `GET /api/startup-progress` / `/api/startup/ws` as `softwareLog`) and a
   structured report to `/tmp/software-report.json` (surfaced by
   `GET /api/admin/software/report` as the homepage **Software** widget). Each
   run **truncates** the progress log first (`resetProgressLog`), because `/tmp`
   survives an in-place restart (`/debug/restart-openclaw`), which would
   otherwise interleave ticks from earlier boots.
4. `POST /api/admin/software/provision` (ADMIN/ROOT) reconciles the manifest
   **on demand**, so a manifest uploaded or corrected mid-life does not need a
   restart. It re-materializes the current R2 manifest, runs the same CLI to
   completion bounded by `SOFTWARE_PROVISION_RUN_TIMEOUT_MS` (5 min), and returns
   the fresh report; `409` when the flag is off (`disabled`) or nothing is
   declared (`no_manifest`), `502` when the run fails.
5. Both `GET` endpoints carry `enabled` (the flag), so the dashboard can tell
   "switched off", "asleep", "not run yet" and "nothing declared" apart instead
   of showing one empty state. The widget exposes a **Run provisioning** button
   to ADMIN/ROOT when the flag is on.

The manifest is arbitrary shell executed inside the tenant container, so the
upload endpoint is ADMIN/ROOT-only (never exposed to readers), `timeoutSec` is
per package, and the on-demand run is bounded by the route.

A runnable example ships with the repo:
`test/e2e/fixture/software/gog-goplaces.json` declares the `gogcli` (`gog`) and
`goplaces` binaries that the `openclaw/openclaw/gog` and `.../goplaces` skills
(listed on skills.sh, published on ClawHub as `@steipete/gog` /
`@steipete/goplaces`) need, and `test/e2e-backup/zzz_software_binaries_restore.txt`
(local-only) proves the whole loop: upload the manifest → `openclaw skills
install @steipete/{gog,goplaces} --global` → backup → destroy the container →
restore, after which the binaries are back on `PATH` because the next
provisioning cycle re-installed them (the report `startedAt` advances) while the
skills come back from the snapshot. `scripts/software-restore-smoke.sh`
(`bun run test:software:smoke`) proves the same contract directly against the
sandbox image, without workerd/Miniflare, and is the fallback when the local
container runtime is unavailable.

Key modules: `src/gateway/software-provisioning.ts` (flag + spawn + on-demand
run + report parsing), `src/services/software-manifest/` (R2 store +
materialization), `src/routes/api/software.ts` (manifest read/upload, report,
on-demand provision), `packages/moltlazy/src/software/` (the CLI).

### AI Provider

Cloudflare AI Gateway is the only provisioned AI provider path. The startup script configures
auth when all of these env vars are set:

1. **Cloudflare AI Gateway** (native): `CLOUDFLARE_AI_GATEWAY_API_KEY` + `CF_AI_GATEWAY_ACCOUNT_ID` + `CF_AI_GATEWAY_GATEWAY_ID`

### Container Environment Variables

These are the env vars passed TO the container (internal names):

| Variable                        | Config Path                  | Notes                                                                                                                                                |
| ------------------------------- | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `CLOUDFLARE_AI_GATEWAY_API_KEY` | (env var)                    | Native AI Gateway key                                                                                                                                |
| `CLOUDFLARE_API_TOKEN`          | (env var)                    | AI Gateway REST API Bearer token; tenant binding is derived from the AI key                                                                          |
| `CF_AI_GATEWAY_ACCOUNT_ID`      | (env var)                    | Account ID for AI Gateway                                                                                                                            |
| `CF_AI_GATEWAY_GATEWAY_ID`      | (env var)                    | Gateway ID for AI Gateway                                                                                                                            |
| `OPENCLAW_GATEWAY_TOKEN`        | `--token` flag               | Mapped from `MOLTBOT_GATEWAY_TOKEN`                                                                                                                  |
| `OPENCLAW_DEV_MODE`             | (dev-agents flag)            | Derived from the `-dev` hostname or an `OPENCLAW_DEV_MODE` override; `DEV_MODE` local                                                                |
| `FOURIA_DOCS_CHANNEL`           | (docs channel)               | `latest` on production, `staging` on the staging platform / `-dev`; selects channel-matched docs + release notes                                     |
| `FOURIA_DOCS_BASE_URL`          | (docs channel)               | `https://fouria.io/<channel>`                                                                                                                        |
| `FOURIA_DOCS_FALLBACK_URL`      | (docs channel)               | `https://fouria-docs.pages.dev/<channel>` (public mirror)                                                                                            |
| `FOURIA_VERSION`                | (build identity)             | The Worker bundle's component CalVer (`src/build-info.ts`)                                                                                           |
| `OPENCLAW_RESTORED`             | (zero-data-loss signal)      | `true` when the launch runs on a container that already holds tenant state (`hasRestoredTenantState`); makes `start-openclaw.sh` skip the setup body |
| `TELEGRAM_BOT_TOKEN`            | `channels.telegram.botToken` | **@deprecated** — E2E only. Use integrations dashboard in production.                                                                                |
| `DISCORD_BOT_TOKEN`             | `channels.discord.token`     | **@deprecated** — E2E only. Use integrations dashboard in production.                                                                                |
| `SLACK_BOT_TOKEN`               | `channels.slack.botToken`    | **@deprecated** — E2E only. Use integrations dashboard in production.                                                                                |
| `SLACK_APP_TOKEN`               | `channels.slack.appToken`    | **@deprecated** — E2E only. Use integrations dashboard in production.                                                                                |

## OpenClaw Config Schema

OpenClaw has strict config validation. Common gotchas:

- `agents.defaults.model` must be `{ "primary": "model/name" }` not a string
- `gateway.mode` must be `"local"` for headless operation
- No `webchat` channel - the Control UI is served automatically
- `gateway.bind` is not a config option - use `--bind` CLI flag

See [OpenClaw docs](https://docs.openclaw.ai/) for full schema.

## Common Tasks

### Adding a New API Endpoint

1. Add route handler in `src/routes/api.ts`
2. Add types if needed in `src/types.ts`
3. Update client API in `src/client/api.ts` if frontend needs it
4. Add tests

### Adding a New Environment Variable

1. Add to `MoltbotEnv` interface in `src/types.ts`
2. If passed to container, add to `buildEnvVars()` in `src/gateway/env.ts`
3. Update `.dev.vars.example`
4. Document in README.md secrets table

### Debugging

```bash
# View live logs
npx wrangler tail

# Check secrets
npx wrangler secret list
```

Enable debug routes with `DEBUG_ROUTES=true` and check `/debug/processes`.

## R2 Backup Encryption (F4E1)

When `BACKUP_ENCRYPTION_KEY` (base64 32-byte secret) is set, the Worker post-encrypts every
snapshot after `sandbox.createBackup()`: `backups/<id>/data.sqsh` is replaced by
`data.sqsh.enc` (chunked AES-256-GCM, per-object HKDF key) and `meta.json` is flagged
`encrypted: true` (kept readable — the SDK needs it). Restores decrypt transiently and delete
the plaintext in a `finally` block. Config versions under `configs/` are encrypted with
`customMetadata.encrypted='true'`. Legacy plaintext backups remain restorable.
Set `BACKUP_ENCRYPTION_REQUIRED=true` to fail closed. Crypto lives in `src/crypto/`.

workerd's `R2Bucket.put()` rejects `ReadableStream` bodies of unknown length, so uploads wrap
the ciphertext/plaintext stream in a `FixedLengthStream` whose size is computed from the source
object (`encryptedLength`) or the F4E1 header (`decryptedLength`). This keeps backup/restore
memory-bounded (no whole-archive buffering) while satisfying R2. See `src/crypto/r2-backup.ts`.
See `docs/plans/r2-encryption-at-rest.md`.

## R2 Storage Notes

Persistence uses the Sandbox SDK `createBackup()` / `restoreBackup()` API with
the `BACKUP_BUCKET` binding. The snapshot root is `/home/openclaw`; it is not an
s3fs mount and there is no `/data/moltbot` runtime path.

Never delete tenant backup objects outside the retention or decommission
procedure. Hard restarts must checkpoint before killing the gateway.

- **Process status**: The sandbox API's `proc.status` may not update immediately after a process completes. Instead of checking `proc.status === 'completed'`, verify success by checking for expected output (e.g., timestamp file exists after sync).

- **R2 prefix migration**: Backups are now stored under `openclaw/` prefix in R2 (was `clawdbot/`). The startup script handles restoring from both old and new prefixes with automatic migration.

- **Snapshot lease + admin snapshot mode (F5):** the shared R2 `backup-lock` TTL is
  `STEP_DEADLINE_MS` (12 minutes), with a serialized heartbeat every quarter-TTL
  and explicit renewals around slow snapshot/restore boundaries. Renewal events are
  structured; renewal failure is best-effort and does not abort work. The longer
  TTL intentionally favors avoiding false expiry during a live durable step, at the
  cost of a longer stale-holder takeover window. `POST /api/admin/storage/sync`
  enqueues a durable job by default when `ACTIVITY` is available; use `?inline=1`
  only when synchronous completion is required (`?async=1` keeps its prior behavior).
  Without `ACTIVITY`, the route falls back to inline work.

## Cloudflare Tunnel (removed)

Tenant Cloudflare Tunnel support was removed. `start-openclaw.sh` no longer
starts `cloudflared`, no tenant Worker receives a `CLOUDFLARE_TUNNEL_TOKEN`
binding, and Lerma no longer provisions one. The `obsidian-sync` skill still
runs a local sync server on port 18790; exposing it publicly is an operator
decision outside the platform.

---

FourIA offers Private Cloud only (Business On Premise is not offered at this time). Pricing is credit-based: Credits are the exclusive billable unit, consumed by completed messages and automations. Credits do NOT include AI costs — AI expenses are managed separately via BYOL or AI Gateway billing. Reader users are unlimited and free. Builder seats (users who can create and edit agents) are plan-limited. The v1.0-beta is delivered through the Design Partner Program with a single engaged partner. Partners pay a fixed credit fee negotiated with sales. See docs/VISION.md for the full pricing definition.

## Active Technologies

- OpenTofu 1.6+ (HCL) + `cloudflare` provider (Terraform/OpenTofu) (001-opentofu-deploy)
- Cloudflare R2 (for FourIA data), OpenTofu Local/Remote State (for infrastructure tracking) (001-opentofu-deploy)

## Change History

`AGENTS.md` documents how to work on this project, not what changed. Every change
is recorded in the owning product's changelog:

- **The Agent Garage is now an all-in-one, RPC-only Agent Studio.** `/_admin/agents` replaced the character builder with a workbench (roster + Overview, Capabilities, MCP, Memory, Automations, Connections, Health). Every read goes through Gateway RPC — `config.get`, `agents.list`, `agents.files.*`, `agents.workspace.*`, `tools.catalog`/`tools.effective`, `skills.*`, `plugins.list`, `hooks.status`, `cron.*`, `sessions.*`, `memory.status`, `system.info` — with **no `openclaw` CLI and no shell**. Core reads prefer Admin HTTP RPC; the richer surfaces share **one** `withGatewayRpcClient` connection per request, and each call is guarded so one unsupported method only degrades its pane. Reads are **non-waking**: the route bounds `isRuntimeActive()` (3 s, same as the passive health probe) and returns `{ available: false, reason: "sleeping" }` before opening a socket. Writes use `agents.files.set` with the `agents.files.get` hash (`409 agent_file_conflict` on stale content), `config.patch` for bindings (no `openclaw agents bind`), `cron.run`/`cron.update` for automations, and `plugins.setEnabled`/`plugins.refresh` (admin scope). RPC gaps are explicit: there is no MCP probe or binary-presence method, so MCP health is the last `tools.effective` inventory + notices and the Health pane lists `skills.bins`/missing requirements instead of claiming presence. Tests: `src/agents/studio.test.ts`, `src/routes/api/agent-studio.test.ts`, `src/client/pages/__tests__/role-isolation.test.tsx`, `test/ui/agent-studio.spec.ts`; corpus `dashboard-api-contract.txt`, `operational-modes.txt`, `tenant-isolation.txt`, `agent-visibility.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md` § Agent Studio, `apps/fouria/README.md`, `docs/TESTING.md`.
- OpenClaw migrated to **2026.11.3** (`catalog.openclaw` + the `@openclaw/gateway-client` / `@openclaw/gateway-protocol` pins in the root `package.json`; the plugin repo tracks and publishes the same release). The tracked range (2026.10.9 → 2026.11.3: 2026.10.10, 2026.11.1, 2026.11.2, 2026.11.3) has **no config-schema changes** — 2026.11.3's notes state "No new configuration options or features" — so the moltlazy `openclaw config validate` cctr contract passes unchanged and no `packages/moltlazy` source change was needed. The release does **lower the Node engine floor** from `>=24.22.1 <25 || >=26.12.1` to `>=24.16.0 <25 || >=26.1.0` (2026.10.10); the container keeps the pinned Node **v24.22.1**, which satisfies the wider range, so no image change. Relevant runtime fixes for a tenant: Gateway startup loads fewer module files and no longer stalls on long punctuation runs in abort-command detection, cron storage failures no longer exit the Gateway, plugin results stay scoped to their connection credentials, and Signal daemons/bundled language servers stop inheriting unrelated Gateway secrets. The gateway boot smoke (`boot=0 health=ok`) and the full verification gate are green against 2026.11.3. No component CalVer bump — the release train handles that.

- **Local containers no longer look like a platform bug, and the memory watchdog never races a boot.** A local restore failed with `Container failed to start` while `docker ps` showed only a `…-proxy` sidecar churning every ~20 s; the app container was never created. That is `net.ipv4.conf.all.src_valid_mark=1` (which Tailscale sets) making workerd's egress sidecar drop its own marked handshake ([cloudflare/workerd#6860](https://github.com/cloudflare/workerd/issues/6860)), not a FourIA defect. The dev server now warns before it builds images — `src/dev-container-preflight.ts` (pure `describeContainerNetnsProblem` + `readContainerNetnsSettings`, unit-tested) reads the two relevant sysctls and prints the fix (preferred: `net.core.devconf_inherit_init_net = 2`, which keeps Tailscale working; one-session alternative: `src_valid_mark=0`) plus the stale-sidecar cleanup; `FOURIA_SKIP_CONTAINER_PREFLIGHT=1` silences it for CI hosts. The same investigation exposed a real race: the durable memory watch sampled `/proc` with `sandbox.exec` — which auto-starts a stopped container — while a restore was probing it, producing `gateway.memory-read failed (launch-failed)` and a workerd `jsgInternalError` that aborted `container.start()`. `checkGatewayMemoryHealth()` now takes the bounded non-waking `isRuntimeActive()` read first and returns `{ action: 'skipped', reason: 'runtime-inactive' }` when the runtime is off, and `ActivityScheduler.runMemoryWatch()` returns before sampling while a boot/backup/maintenance job is active (queued/running/retrying), so observability never participates in a boot. The persistence lease acquire is now bounded too: a conditional `backup-lock` write that never settles is reported as `persistence.lock.write-timeout` and treated as a lost acquire, instead of holding the durable step (6 min of silence, then `job-step.interrupted`) — observed locally when taking over an expired lease. Tests: `src/dev-container-preflight.test.ts`, `src/gateway/memory.test.ts`, `src/activity/scheduler.test.ts`, `src/persistence.test.ts`; corpus `test/business-rules/operational-modes.txt`; docs `docs/DEVELOPMENT.md` § "Local containers never start", `docs/TESTING.md` § "E2E Tests (Local)", `apps/fouria/docs/WORKER-DEVELOPMENT.md` § memory watchdog.

- `apps/fouria/CHANGELOG.md` — tenant Worker (CalVer)
- `workers/fouria-dispatch/CHANGELOG.md` — dispatch Worker (CalVer)
- `apps/lerma/CHANGELOG.md` — customer dashboard (semver, Changesets)
- `packages/moltlazy/CHANGELOG.md` — config patcher and SDK (semver, Changesets)
- `packages/cost-calculator/CHANGELOG.md` — pricing helpers (semver, Changesets)

Add a changeset in every pull request (see the root `AGENTS.md`, "Semver
Releases"), and do not add a change log to this file.
