# FourIA — Development Guide

Comprehensive onboarding guide for developers and AI agents working on the FourIA platform.

> See [VISION.md](/latest/llms/platform/reference/vision/index.md) for product vision, [CONTRIBUTING.md](/latest/llms/platform/reference/contributing/index.md) for contribution rules, and [ENV-VARIABLES.md](/latest/llms/platform/reference/env-variables/index.md) for environment variable reference.

---

## Project Map

FourIA is a managed, private agent orchestration runtime that turns markdown-based AI agents into operational systems. The monorepo is organized into four top-level directories:

| Directory   | Purpose                                             |
| ----------- | --------------------------------------------------- |
| `apps/`     | Cloudflare Worker applications (Tenant + Dashboard) |
| `workers/`  | Infrastructure Workers (Dispatch, Outbound)         |
| `packages/` | Shared libraries (moltlazy, plugins)                |
| `webs/`     | Static websites (Paso4 marketing homepage)          |

---

## Component Reference

### `apps/fouria/` — FourIA Tenant Worker

| Field           | Value                                                                                                                    |
| --------------- | ------------------------------------------------------------------------------------------------------------------------ |
| **Package**     | `moltbot-sandbox`                                                                                                        |
| **Description** | Per-customer Worker managing OpenClaw in a Cloudflare Sandbox container                                                  |
| **Runtime**     | Cloudflare Workers (Hono 4)                                                                                              |
| **Frontend**    | React 19 SPA (Vite 6), served from `/_admin/`                                                                            |
| **IaC**         | `wrangler.jsonc` + OpenTofu (`iac/`)                                                                                     |
| **Key deps**    | `@cloudflare/sandbox`, `@cloudflare/puppeteer`, `moltlazy` (workspace), `openclaw` (catalog), `jose`, `react-router-dom` |

**Purpose:** This is the **core component** of FourIA. Each tenant gets a dedicated Worker instance that provisions, manages, and proxies to an OpenClaw gateway running inside a Cloudflare Sandbox container. It provides:

- OpenClaw gateway lifecycle (start container, onboard, config patch, launch)
- HTTP/WebSocket proxy to the gateway on port 18789
- Admin dashboard at `/_admin/` for device pairing, configuration, secrets, integrations
- REST API at `/api/*` wrapping OpenClaw CLI calls
- Debug endpoints at `/debug/*` (ROOT role + DEBUG_ROUTES only)
- Multi-tenant isolation via `CustomerRegistry` DO
- Cost observability via Analytics Engine + OTel metrics

**Key files:**

| File                        | Role                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `src/index.ts`              | Main Hono app: middleware pipeline, route mounting, WS proxy                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `src/gateway/process.ts`    | Container lifecycle: find, start, restart, destroy                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `src/gateway/env.ts`        | Builds env vars passed to the container                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `src/auth/`                 | Cloudflare Access JWT auth + RBAC (ROOT/ADMIN/BASE_USER)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `src/routes/api.ts`         | API route aggregator — delegates to domain modules                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `src/routes/admin-ui.ts`    | Serves React SPA from `/_admin/*`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `src/routes/debug.ts`       | Debug endpoints (process listing, CLI execution, env inspect)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `src/services/fouria-docs/` | Dev-only per-tenant client docs (R2 store, git-like versioning, backups)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `src/client/`               | React SPA (pages/, components/, hooks/)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `start-openclaw.sh`         | Container entrypoint. Production cold-start is the atomic `start` mode: it acquires the exclusive startup `flock` **before** the setup body (onboard → moltlazy patch) and holds it through the Gateway `exec`, so one owner serializes setup **and** launch; `post-start` runs dev automation after the gateway is ready. `setup`/`launch` are standalone smoke/manual modes and legacy `full` runs setup + a background gateway. `start`/`setup`/`full` serialize the setup body via the exclusive `flock` so concurrent Worker isolates cannot run it at once |
| `Dockerfile`                | Sandbox container image (Node 24 + OpenClaw + cloudflared + cctr)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |

**Development commands:**

```bash
# Frontend hot-reload (Vite dev server on port 8787)
bun run dev

# Backend Worker (Cloudflare Vite plugin, workerd on port 8788) — run in another tab
bun run dev:worker

# Full local start (build all + Vite dev server)
bun start
```

> **Note:** Frontend and Worker changes both hot-reload. The Worker dev server
> (`vite.config.ts`) runs through `@cloudflare/vite-plugin`, the same plugin the
> production build uses.

> **Tunnels:** Vite blocks unknown `Host` headers. To expose the dev server
> (e.g. for a Telegram webhook), pass the tunnel hostname:
> `DEV_ALLOWED_HOSTS=foo.trycloudflare.com bun run dev:worker`
> (comma-separated; a leading dot allows a whole domain). Point the tunnel at
> the Worker port `:8788`, not the SPA port `:8787`.

**See:** [apps/fouria/docs/WORKER-DEVELOPMENT.md](/latest/llms/platform/apps/fouria/worker-development/index.md) for deep-dive architecture guide.

---

### `apps/lerma/` — Customer Dashboard

| Field           | Value                                                  |
| --------------- | ------------------------------------------------------ |
| **Package**     | `lerma`                                                |
| **Description** | Billing/subscription dashboard and CustomerRegistry DO |
| **Runtime**     | Phoenix 1.8 (Elixir) + Hono Worker (TypeScript)        |
| **Frontend**    | Phoenix LiveView 1.1 (Tailwind CSS)                    |
| **Database**    | PostgreSQL (Ecto) + Cloudflare D1                      |
| **IaC**         | Phoenix deployment + Cloudflare Worker bindings        |

**Purpose:** Two-in-one app:

1. **Phoenix backend** — Serves the billing/subscription management UI via LiveView. Manages clients, instances, agents, channels, costs, knowledge bases.
2. **Worker portion** — Hono API Worker interacting with D1 database for CustomerRegistry operations.

The Applications catalog (`Lerma.Applications`) supports per-application logo uploads (max 1 MB, ROOT-only) stored as a binary in the `applications` table and served at `GET /dashboard/applications/:id/logo`. ROOT users can also generate, rotate, and revoke an application-scoped provisioning API key from the application detail page. Lerma stores only a digest and reveals a generated key once; see [staging promotion](/latest/llms/platform/reference/plans/staging-promotion/index.md#develop-build-promotion-to-staging).

### Development expense (non-billable cost tracking)

Dev-environment and staging-platform instances are **never billable** (no credit
consumption) but their infrastructure + AI spend is tracked as **development
expense** — the cost of building for a client. `Lerma.DevelopmentExpense` pulls
the Worker's Analytics Engine cost rows asynchronously and prices them with the
shared `Lerma.CostCalc` constants (container uptime × instance pricing, R2
Class A/B, AI `cost.usd`). Spend routed through the Cloudflare AI Gateway
(unified billing) is taken from the authoritative **AI Gateway logs** API
(`Lerma.Cloudflare.AiGateway.list_logs/3`) instead of OTel, so the two sources
never double-count. `health-check` rows yield uptime/downtime statistics.

- Module: `lib/lerma/development_expense.ex`
- Internal API (bearer `billing_ingest_token`):
  `GET /api/dev-expense/v1/clients/:client_id` (id or slug) and
  `GET /api/dev-expense/v1/summary` (global rollup). `from`/`to` = `YYYY-MM-DD`.
- The Worker tags every cost event with `platform` + `environment` and keys it
  by the base client slug — see
  [apps/fouria/docs/WORKER-DEVELOPMENT.md](/latest/llms/platform/apps/fouria/worker-development/index.md#cost-tracking-dimensions--development-expense).

**Key directories:**

| Directory             | Purpose                                               |
| --------------------- | ----------------------------------------------------- |
| `lib/lerma/`          | Business logic (billing, agents, channels, clients)   |
| `lib/lerma_web/`      | Phoenix web layer (LiveView, controllers, components) |
| `lib/lerma_web/live/` | LiveView pages (client, dashboard, instance, home)    |
| `src/`                | Worker portion (Hono routes, DO bindings)             |
| `migrations/`         | D1 SQL migrations                                     |

---

### `workers/fouria-dispatch/` — Dispatch Worker

| Field           | Value                                                           |
| --------------- | --------------------------------------------------------------- |
| **Package**     | `fouria-dispatch`                                               |
| **Description** | Routes `*.fouria.io` wildcard requests to correct tenant        |
| **Runtime**     | Cloudflare Workers (Hono 4)                                     |
| **Tech**        | KV caching, D1 tenant lookup, Dispatch Namespace, CF Access JWT |
| **IaC**         | `wrangler.jsonc`                                                |

**Purpose:** The **entry point** for all tenant traffic. Extracts tenant slug from subdomain (e.g., `acme.fouria.io` → `acme`), resolves the tenant from D1/KV, applies plan-based CPU limits, and dispatches to the tenant Worker via Cloudflare Dispatch Namespace.

**Key files:**

| File             | Role                                                       |
| ---------------- | ---------------------------------------------------------- |
| `src/index.ts`   | Main handler: slug extraction, tenant resolution, dispatch |
| `src/auth.ts`    | CF Access JWT verification for dashboard redirects         |
| `wrangler.jsonc` | KV (`TENANT_INDEX`), D1 (`DB`), Dispatch Namespace         |

**Development:**

```bash
bun run test:dispatch          # Unit tests
bun run deploy:dispatch        # Deploy to production
bun run deploy:dispatch:staging # Deploy to staging
```

Request flow: `User` → Dispatch Worker → Tenant Worker → AI Gateway → Upstream LLM

---

### `workers/fouria-outbound/` — Outbound Worker ⚠️ NOT SUPPORTED

> **Deprecated.** PII filtering is handled natively by Cloudflare AI Gateway. This worker and the Presidio container are no longer required. See [Cloudflare AI Gateway docs](https://developers.cloudflare.com/ai-gateway/).

---

### `packages/moltlazy/` — Config Patcher + Typed RPC Client

| Field           | Value                                                    |
| --------------- | -------------------------------------------------------- |
| **Package**     | `moltlazy` (v0.4.0)                                      |
| **Description** | OpenClaw config generation ($include) + typed RPC client |
| **Runtime**     | Node.js ESM (CLI tool), Bun for testing                  |
| **IaC**         | N/A — internal library                                   |

**Purpose:** Two systems:

1. **Config Module** (`src/config/` + `cli.ts`): Generates per-section include files (`moltlazy-session/tools/logging.json`) at container startup. Uses OpenClaw's `$include` directive as **per-section** single-file includes (never a root `$include`) for immutable, idempotent config injection. This is the **mandatory config system** — session, tools, logging (via per-section `$include`), plus gateway and agent defaults (inline).

2. **SDK Module** (`sdk/`): Typed RPC client over the OpenClaw Admin HTTP API. Direct gateway interaction via `POST /api/v1/admin/rpc`. **Not integrated in the Worker runtime for v1** — the Worker uses CLI calls instead.

**Key files:**

| File                      | Role                                                                 |
| ------------------------- | -------------------------------------------------------------------- |
| `cli.ts`                  | CLI entry point (`moltlazy patch` / `moltlazy validate`)             |
| `index.ts`                | Main config patching logic                                           |
| `src/config/include.ts`   | `$include` injection into `openclaw.json`                            |
| `src/config/gateway.ts`   | Gateway config section generation                                    |
| `src/config/session.ts`   | Session config generation                                            |
| `src/agents/`             | Agent definitions (defaults; `premade/prod` + `premade/dev`)         |
| `src/agents/premade/dev/` | Dev-only agents: `fouria-builder`, `fouria-qa` (`OPENCLAW_DEV_MODE`) |
| `sdk/client.ts`           | Typed RPC client over Admin API                                      |
| `sdk/`                    | Domain modules (agents, channels, config, devices, etc.)             |

**Dev-only agents and client documentation:** when the instance is a dev
environment (`OPENCLAW_DEV_MODE=true` on the container, derived by the Worker
from the instance's `-dev` hostname — it does not bypass auth, unlike `DEV_MODE`;
local development falls back to `DEV_MODE`),
`moltlazy patch` also injects **Fouria Builder** (`fouria-builder`) and
**Fouria Q&A agent** (`fouria-qa`). The Builder fetches the client proposal
(phases, acceptance criteria) from the global `paso4-leads` R2 bucket through
the `fouria-leads` skill (the sandbox `leads.r2.internal` bridge), plus platform
docs fetched from the docs site (`https://fouria.io`, fallback
`https://fouria-docs.pages.dev`, starting at `/llms.txt`), and produces a POC +
`cctr` acceptance corpus under `/home/openclaw/workspace/fouria-poc/`. Its build
method is the **`fouria-product-build` per-agent workspace skill**
(`/home/openclaw/workspace/fouria-builder/skills/fouria-product-build/SKILL.md`,
authored in `packages/moltlazy/workspace/skills/fouria-product-build.ts`), which
sequences roster framing, the tool-first restriction model, the MCP tool
surface, channel quick actions, per-client tone, the vertical slice, the corpus
and the handoff. The Q&A agent runs `cctr` against that corpus. Production ships
no premade agents (only `main`).

**Configuration flow:**

```
Worker Request → restoreIfNeeded() → buildEnvVars() → ensureGateway()
                                                          │
                                                          ▼
        start-openclaw.sh start — one supervised call (startup lock held throughout)
                          │
                          ├─ setup: onboard + moltlazy patch
                          │
                          └─ exec openclaw gateway run (supervised, lock held)
                                                          │
                                                          ▼
                     start-openclaw.sh post-start (dev automation + boot-log reset)

moltlazy patch generates per-section $include files (session/tools/logging); gateway/agents inline
```

---

### OpenClaw Plugins → [`Paso4/openclaw-plugins`](https://github.com/Paso4/openclaw-plugins)

The `cloudflare-unified-billing` OpenClaw provider plugin **no longer lives in
this repository**. Its source, tests, and release train moved to the external
[`Paso4/openclaw-plugins`](https://github.com/Paso4/openclaw-plugins) repo. Tenant
images install the published ClawHub package
(`clawhub:@paso4/cloudflare-unified-billing`); nothing in this monorepo builds or
tests it anymore.

See [OPENCLAW-PLUGINS.md](/latest/llms/platform/libraries/openclaw-plugins/index.md) for the migration record,
runtime integration details, and versioning.

---

### `webs/paso4.io/` — Paso4 Marketing Homepage

| Field           | Value                                            |
| --------------- | ------------------------------------------------ |
| **Package**     | `@paso4/homepage`                                |
| **Description** | Paso4 marketing website                          |
| **Runtime**     | React 18 (Vite 6), deployed via Cloudflare Pages |
| **IaC**         | `wrangler.jsonc` (Cloudflare Pages config)       |

**Purpose:** Public-facing website for Paso4. Uses MUI 7 + Radix UI + Tailwind CSS 4. Not part of the FourIA runtime.

---

## Cloudflare Infrastructure

All Workers use `wrangler.jsonc` for configuration. Bindings are defined per environment (dev/staging/production).

### Bindings Reference

| Binding          | Type              | Purpose                                                               |
| ---------------- | ----------------- | --------------------------------------------------------------------- |
| `Sandbox`        | Container DO      | Cloudflare Sandbox container lifecycle management                     |
| `GATEWAY_RPC`    | Durable Object    | Admin RPC proxy to OpenClaw gateway (planned — see GATEWAY_RPC issue) |
| `R2SecretsStore` | Class (in-memory) | F4E1-encrypted R2-backed secrets store (replaces UserSecretsStore DO) |
| `BACKUP_BUCKET`  | R2 Bucket         | Squashfs snapshots for container persistence                          |
| `BROWSER`        | Browser Rendering | Headless browser for CDP automation                                   |
| `ANALYTICS`      | Analytics Engine  | Cost & usage metrics (dataset: `fourai_cost_metrics`)                 |
| `TENANT_INDEX`   | KV Namespace      | Tenant slug → worker mapping cache (dispatch Worker)                  |
| `DB`             | D1 Database       | Client/tenant registry (dispatch Worker)                              |

### IaC (OpenTofu)

The `apps/fouria/iac/` directory contains OpenTofu configs for infrastructure provisioning:

- `modules/access/` — Cloudflare Access application
- `modules/ai_gateway/` — Cloudflare AI Gateway
- `modules/r2/` — R2 bucket provisioning
- `modules/worker/` — Worker deployment config
- `environments/dev/` and `environments/prod/` — Environment-specific vars

### Wrangler Commands

Local development uses the Cloudflare Vite plugin (`vite dev`, via
`bun run dev:worker`), not `wrangler dev`. Wrangler remains the deploy/config CLI.

| Command                                  | Purpose                                      |
| ---------------------------------------- | -------------------------------------------- |
| `wrangler deploy`                        | Deploy Worker to Cloudflare                  |
| `wrangler tail`                          | Stream live logs from deployed Worker        |
| `wrangler secret put KEY`                | Set an encrypted Worker secret               |
| `wrangler secret list`                   | List secret names (values not shown)         |
| `wrangler types`                         | Regenerate `worker-configuration.d.ts` types |
| `wrangler d1 execute DB --command "..."` | Execute SQL against D1 database              |

> **Skills to load:** `wrangler`, `cloudflare`, `workers-best-practices`, `durable-objects`, `sandbox-sdk`

---

## Provisioning & Release Lifecycle

> Artifacts flow from Cloudflare storage + the container registry to the edge. Lerma never builds — it lists image versions from the registry and pulls the worker bundle from the `fouria-bundles` R2 bucket, then orchestrates the Cloudflare control plane. Read [.github/workflows/test.yml](https://github.com/0xCAB0/fouria/blob/develop/.github/workflows/test.yml) (build + bundle validation + `unstable-<sha>` image push + bundle upload to R2 + e2e), [.github/workflows/lerma.yml](https://github.com/0xCAB0/fouria/blob/develop/.github/workflows/lerma.yml) (lerma CI quality gates only), [.github/workflows/deploy-dispatch.yml](https://github.com/0xCAB0/fouria/blob/develop/.github/workflows/deploy-dispatch.yml) (dispatch worker deploy) and [.github/workflows/deploy-fouria.yml](https://github.com/0xCAB0/fouria/blob/develop/.github/workflows/deploy-fouria.yml) (fouria `latest` image promotion + release bundle upload) for the CI/CD side and `apps/lerma/lib/lerma/provisioning/tenant_provisioner.ex` for the orchestration side.

### Release: one tag, two artifacts

Every GitHub Release of the fouria product produces two immutable artifacts:

1. **`fouria-bundles:<tag>/fouria.mjs`** — the multipart Worker bundle (main module + admin-dashboard assets), uploaded to the `fouria-bundles` R2 bucket (see `test.yml` / `deploy-fouria.yml`). It is also attached to the GitHub release as `fouria.mjs`.
2. **`fouria:{tag}`** — the sandbox container, pushed to the Cloudflare managed registry (`registry.cloudflare.com/{account}`).

CI validates the bundle first (every asset referenced by `index.js` must exist as a multipart part) so broken bundles never ship. Pushes to `develop` run the `build-and-push` job (build + bundle validation + `fouria:unstable-<sha>` image push + `fouria-bundles:unstable-<sha>/fouria.mjs` upload), then the `e2e` job consumes that image. On a real release, `deploy-fouria.yml` **promotes the CI-built artifacts** — it retags the `fouria:unstable-<sha>` image CI built for the release's develop commit to `fouria:{tag}` + `latest` and re-publishes the R2 bundle as `<tag>` / `latest`; nothing is rebuilt at release time. `deploy-dispatch.yml` deploys the dispatch worker. Release commits carry `[skip ci]`, so a version bump never re-runs the test suite.

#### Version calculation

There are no manual bump levels. Semver products (`packages/moltlazy`,
`packages/cost-calculator`, `apps/lerma`) are bumped by Changesets: PRs add a
changeset, the `version-packages` workflow opens a version/changelog PR, and
merging it is the versioning step. CalVer components (`apps/fouria`,
`workers/fouria-dispatch`) are bumped to today's date by
`scripts/release.mjs detect` during the manual Release workflow, which also
tags the already-bumped semver versions and builds the GitHub release body from
the per-package changelogs. See [VERSIONING.md](/latest/llms/platform/reference/versioning/index.md).

```mermaid
flowchart LR
    A[push develop] --> B[build-and-push]
    B -->|unstable-sha image| S[e2e]
    B -->|unstable-sha bundle| S
    R[GitHub Release] --> C[deploy-fouria promote]
    B -->|retag unstable-sha| C
    C --> E[(CF Container Registry<br/>fouria:tag + latest)]
    C --> G[(fouria-bundles R2<br/>tag + latest)]
    E --> I[registry.cloudflare.com]
    G --> H[release asset fouria.mjs]
```

### Stable-only production deploys

Production instances may only reference **stable release tags** (`v2026.9.15`,
`v2026.9.15-1`, `v1.2.3`) — never the mutable `unstable-<sha>` channel, which the
stale-image cleanup eventually removes. This is enforced in lerma, not just in CI:

- `Lerma.Cloudflare.ImageRegistry.stable?/1` / `stable_only?/0` classify tags and
  gate on `:cf_environment` (staging keeps the full unstable channel).
- The instance deploy UI lists stable tags only in production, and
  `Instances.update_instance_version/3` rejects unstable/prerelease versions with
  `{:error, :unstable_version_not_allowed}`.
- When a production deploy requests `latest`, `Lerma.Cloudflare.RegistryBundle`
  resolves it to the newest **stable** bundle in `fouria-bundles` (fail closed
  when no stable release has been published) instead of the newest object.

Stability is defined by the CalVer tag shape (`vYYYY.M.D` or the same-day
`vYYYY.M.D-N`) — never by the GitHub release `prerelease` flag, which release-it
sets for hyphenated CalVer tags because they parse as semver prereleases. The
release-triggered deploy workflows (`deploy-fouria.yml`, `deploy-dispatch.yml`,
`deploy-dashboard.yml`) run a `classify` job that matches the tag against
`^vYYYY.M.D(-N)?$` before promoting.

### Develop build promotion to staging

After the `Test` workflow completes on `develop`,
[`.github/workflows/promote-staging.yml`](https://github.com/0xCAB0/fouria/blob/develop/.github/workflows/promote-staging.yml)
checks the source run's `build-and-push` job conclusion through the GitHub Actions
API. A failed or skipped build never promotes. A successful build already has
matching immutable artifacts at `unstable-<sha>`:

- container image `fouria:unstable-<sha>` in Cloudflare's registry;
- worker bundle `fouria-bundles/unstable-<sha>/fouria.mjs` in R2.

The workflow sends that same version to staging Lerma's existing
`POST /api/provisioning/upgrade/client` endpoint, using the application-scoped
Fouria key and `X-FourIA-Application: fouria`. It first checks current status,
skips a completed same-version promotion, then polls until running instances
report the target version. Staging's key is generated in the staging Lerma
**Applications → Fouria** page and copied to the staging-only GitHub secret
`STAGING_PROVISIONING_API_KEY`; production credentials are not used.

> **Access bypass.** Lerma's `/api/*` paths must bypass Cloudflare Access at the
> edge (`apps/lerma/iac/tf/access.tf` creates a path-scoped `Lerma API` bypass
> application for the dashboard domain, alongside the invitation bypass). The
> app-level `LermaWeb.Plugs.CfAccessAuth` already treats `/api` as an allowed
> path, and every API route authenticates with its own bearer credential, so
> CI automation can reach the provisioning API without an Access service token.
> If this bypass is missing the promotion gets an Access login `302` instead of
> JSON.

The staging deployment applies pending Lerma migrations and runs the
`Lerma.StagingTenant` bootstrap with `CF_ENVIRONMENT=staging`. It idempotently
creates the reserved `client` tenant (renaming the legacy seeded
`fouria-staging` row if necessary). Production applies the same schema migration
with `CF_ENVIRONMENT=production` but does not create or change a `client` row.
The database's environment marker enables a trigger that prevents the staging
`client` from being deleted, renamed, or truncated.

The `.github/scripts/cleanup-stale-images.sh` cron additionally never deletes an
image (or its matching R2 bundle) that any container application still references
— current configuration _and_ every application version, so an in-progress
rollout's old image is safe. If the in-use set cannot be resolved, a live cleanup
run aborts without deleting anything.

### Provision: lerma materializes a tenant

When a client signs up, lerma runs a 9-step, idempotent pipeline (`TenantProvisioner`). Each step creates one Cloudflare resource — and creates it _only once_. If step N fails, steps 1..N−1 roll back in reverse order, best-effort and non-blocking. Health is verified **after** the run is finalized, in a background task — never as a deployment step.

The version list comes from the registry catalog (`ImageRegistry`), and the bundle is pulled from the `fouria-bundles` R2 bucket at step 4 (`RegistryBundle`), then uploaded as a WFP dispatch-namespace script together with a container application tying the `Sandbox` DO to the released image. Secrets ride as `secret_text` bindings: per-tenant ones are generated at provision time (`MOLTBOT_GATEWAY_TOKEN`, `CDP_SECRET`, `BACKUP_ENCRYPTION_KEY`); the AI credential is copied from Lerma's environment and exposed under both the native-provider and REST-transport names. No secrets ever touch the bundle.

```mermaid
flowchart TD
    A[1 · ai_gateway<br/>create fouria-slug AI Gateway] --> B[2 · access_policy<br/>per-environment Access app + closed group]
    B --> C[3 · r2_bucket<br/>create fouria-backup-slug]
    C --> D[4 · fetch_bundle<br/>pull fouria-bundles:tag from R2]
    D --> E[5 · dispatch_deploy<br/>upload Worker into dispatch namespace]
    E --> F[6 · worker_tags<br/>customer_id · plan · env]
    F --> G[7 · create_instance<br/>DB record + BACKUP_ENCRYPTION_KEY]
    G --> H[8 · invalidate_cache<br/>bust dispatch lookup]
    H --> I[9 · production_transition<br/>client to production]
    I -.-> K[async health check<br/>HealthPoller probes /api/health after finalize]

    E -.-> E1[multipart upload<br/>index.js + assets]
    E -.-> E2[container app<br/>Sandbox DO to CF registry image]
    E -.-> E3[bindings<br/>secret_text secrets · vars · R2 · Analytics · Browser]

    R1[rollback: delete gateway] -.-> A
    R2[rollback: delete access policy] -.-> B
    R3[rollback: delete r2 bucket] -.-> C
    R5[rollback: delete worker script] -.-> E
    R7[rollback: delete instance record] -.-> G

    style I fill:#d3f9d8,stroke:#2f9e44
```

The step 2 Access application is **per-environment**: the access slug carries the platform prefix (`staging-acme.fouria.io` in the staging lerma, `acme.fouria.io` in production), and each app carries two allow policies — the closed `Client Users - {access-slug}` group plus the shared `Paso4 Root Users` group. Both are required because Cloudflare matches a request to the most specific application domain: once a host has its own app, it no longer falls back to the wildcard `*.fouria.io` platform app that root users normally use. The group is seeded with the client's validated user emails, so a reprovisioned client (e.g. revived after decommissioning) keeps its existing users; users still need a fresh instance invitation for Dispatch RBAC, since instance associations cascade away with the old instances.

### Invite: verify and scope client users

Lerma's client page invites users only after an instance exists. One invitation can select multiple instances, including production and dev or instances belonging to different applications. The lifecycle is:

1. `invite_user/2` stores `status = invited`, a SHA-256 token hash, a seven-day expiry, and rows in `client_user_instances`; the raw token exists only in the email.
2. The confirmation email is delivered from a supervised background task (`Lerma.Invitations`) — never inside the request or the database transaction. A slow or failing SMTP relay can neither hold a Postgres connection, block the dashboard, nor roll the invitation back; the outcome is broadcast over PubSub and the client page alerts the operator with a toast plus a per-user delivery badge.
3. No Cloudflare Access synchronization occurs while the user is pending.
4. The public `/invitations/:token` page lets the recipient accept or reject. It is served outside Cloudflare Access — a path-scoped bypass application covers `/invitations/*` and a companion bypass covers the page's static assets (`/assets/*`, `/images/*`, `/favicon.ico`) (`apps/lerma/iac/tf/access.tf`), and the page is a plain controller, so it needs neither an Access login nor the Access-protected `/live` socket.
5. Acceptance records the verified Cloudflare Access email identity, sets `status = validated`, and syncs only selected instance groups.
6. Dispatch independently requires both `client_users.status = validated` and a matching `client_user_instances.instance_id` before forwarding an RBAC role.

Each client environment has a separate Access hostname/group: `{slug}.fouria.io` for production and `{slug}-dev.fouria.io` for dev (with the platform `staging-` prefix in staging). Access app reconciliation replaces misconfigured `everyone` policies with the matching group-only allow policy, and re-adds the shared root-group policy so root operators keep access.

### Update: swap code, keep the tenant

Upgrades are a _subset_ of the pipeline (`@upgrade_pipeline`): re-fetch → re-deploy → tags → update instance → invalidate cache → transition. They never re-run the infrastructure steps, and they preserve existing secrets via `keep_bindings` on the re-upload. Health is verified asynchronously afterwards — it is not a deployment step.

The zero-downtime trick: the running container keeps serving on the old image until its next cold start. The deploy replaces the script and the container-application image reference — live requests are unaffected. Health is probed in the background after the run finalizes: a broken release is surfaced on the instance status (and reconciled by `HealthMonitor`) without rolling the deploy back or deleting the tenant's existing worker/container. The instance page surfaces the resulting health state in its **Observability** section — an **Uptime** widget (production only) plus a **Health events** timeline — backed by the `instance_health_events` state-change log (see [apps/lerma/README.md](/latest/llms/platform/apps/lerma/index.md#observability-and-tools-sections)).

```mermaid
sequenceDiagram
    participant L as Lerma
    participant G as GitHub Releases API
    participant CF as Cloudflare API
    participant W as Tenant Worker
    participant C as Sandbox Container

    L->>G: GET latest release → fouria.mjs + manifest
    L->>CF: PUT script (keep_bindings: secret_text, plain_text)
    L->>CF: update container app image
    L->>CF: PUT worker tags
    L->>L: update instance record (DB)
    L->>CF: invalidate dispatch cache
    loop health gate
        L->>W: GET /health
        W-->>L: 200
    end
    L->>L: client → production
    Note over C: Old container keeps serving until cold start<br/>picks up the new image
```

### The big picture

![Provisioning & release architecture](/latest/assets/docs/diagrams/provisioning-architecture.png)

[Open the editable Excalidraw file](/latest/assets/docs/diagrams/provisioning-architecture.excalidraw)

---

## OpenClaw Integration

### How OpenClaw Data Structures Map to FourIA

FourIA wraps OpenClaw's concepts:

| OpenClaw Concept         | FourIA Representation            | Managed Via                  |
| ------------------------ | -------------------------------- | ---------------------------- |
| Gateway                  | Sandbox container process        | `start-openclaw.sh`          |
| Gateway token            | `MOLTBOT_GATEWAY_TOKEN` env var  | Worker Secrets               |
| Auth profiles            | Provider API keys (env vars)     | `buildEnvVars()`             |
| Config (`openclaw.json`) | Generated by `moltlazy patch`    | `packages/moltlazy/`         |
| Devices (pairing)        | `/_admin/` device management     | `routes/api/devices.ts`      |
| Agents                   | `/_admin/` agents page           | `routes/api/agents.ts`       |
| Channels                 | `/_admin/` integrations page     | `routes/api/integrations.ts` |
| Memory/Wiki              | `/_admin/` knowledge page        | `routes/api/knowledge.ts`    |
| Skills                   | `skills/` directory in container | `start-openclaw.sh`          |

### OpenClaw CLI in the Worker

The Worker wraps OpenClaw CLI calls. Example from `src/routes/api.ts`:

```typescript
// API routes delegate CLI execution to the sandbox container.
// OpenClaw 2026.8.2: commands run on the gateway host auto-connect (no --url/--token);
// multi-agent commands take an explicit --agent. Example:
// Running: openclaw devices list --json
// Output parsing: case-insensitive "Approved" detection
```

CLI calls take 10-15 seconds due to WebSocket connection overhead. Always use the `waitForProcess()` helper.

### Reading OpenClaw Docs

Before implementing any feature that touches OpenClaw configuration or APIs:

1. Load the `openclaw-memory-architect` or `documentation-lookup` skill
2. Search [docs.openclaw.ai](https://docs.openclaw.ai/) for relevant config paths
3. Map OpenClaw structures to FourIA types before coding

---

## Local Development Setup

### Prerequisites

- **Bun** (runtime, package manager, bundler) — always use Bun, never npm
- **Node.js 22** (compatibility for some deps)
- **Docker** (for building the Sandbox container image locally)
- **Wrangler CLI** (Cloudflare Workers CLI)
- **Cloudflare account** with Workers Paid plan (for Sandbox containers)

> **Optional — Nix/direnv.** The repository root ships a dev shell with Bun and
> Mix (for the root Mix workspace checks), and each app ships its own `flake.nix`
> shell, so you can skip manually installing Elixir/Node/Bun/CLIs:
>
> ```bash
> # Repo root: Bun + Mix + Node 22 (default shell)
> nix develop .
>
> # App shells
> nix develop .#lerma      # Elixir 1.20 + OTP 29 + Node 22 + psql + tofu + cloudflared
> nix develop .#fouria     # Bun + Node 22 + wrangler + cloudflared + Playwright browsers
>
> # Or activate per-app with direnv (`.envrc` present in each app dir):
> cd apps/lerma && direnv allow
> cd apps/fouria && direnv allow
> ```
>
> `wrangler`/`cctr`/`plwr` are resolved from the workspace `node_modules`
> (`bun install`), not from Nix, to avoid version drift.

### First-Time Setup

```bash
# 1. Clone and install
git clone <repo-url> fouria
cd fouria
bun install

# 2. Create .dev.vars
cp apps/fouria/.dev.vars.example apps/fouria/.dev.vars
# Edit .dev.vars with your API keys

# 3. Build all shared packages
bun run build:all

# 4. Run validation
bun run typecheck
bun run lint
bun run test

# 5. Prepare and run the Elixir workspace checks (inside `nix develop .`)
mix deps.get
mix workspace.run -t deps.get
mix check

# 6. Start development
cd apps/fouria
# Terminal 1: Frontend hot-reload
bun run dev
# Terminal 2: Worker backend (Cloudflare Vite plugin)
bun run dev:worker
```

### Recommended AI Prompts for Development

When using AI tools to work on FourIA:

```
"Read docs/DEVELOPMENT.md first, then help me understand how [COMPONENT] works."

"Follow the contributing guidelines in docs/CONTRIBUTING.md while implementing this feature."

"Check the OpenClaw docs at docs.openclaw.ai for [config path] before writing code."

"Use the wrangler and cloudflare skills to verify my Worker configuration."

"Load openclaw-memory-architect skill before suggesting OpenClaw memory configurations."
```

---

## Testing

| Scope            | Command                                   | Framework                      |
| ---------------- | ----------------------------------------- | ------------------------------ |
| All unit tests   | `bun run test`                            | Vitest                         |
| Worker unit      | `bun run --cwd apps/fouria test`          | Vitest                         |
| Worker coverage  | `bun run --cwd apps/fouria test:coverage` | Vitest + c8                    |
| Dispatch unit    | `bun run test:dispatch`                   | Vitest                         |
| Outbound unit ⚠️ | `bun run test:outbound` (deprecated)      | Vitest                         |
| Moltlazy unit    | `bun run --cwd packages/moltlazy test`    | Vitest                         |
| Moltlazy E2E     | `bun run test:moltlazy:e2e`               | cctr                           |
| Full suite       | `bun run test:all`                        | Vitest + cctr                  |
| Elixir workspace | `mix check`                               | Format + strict Credo          |
| Elixir precommit | `mix precommit`                           | Credo + license audit + ExUnit |

**AI skills for testing:** `tdd-workflow`, `ai-regression-testing`, `webapp-testing`, `e2e-testing`

---

## DevOps & Monitoring

| Tool                   | Purpose                                                  |
| ---------------------- | -------------------------------------------------------- |
| `wrangler tail`        | Stream live logs from deployed Workers                   |
| `wrangler secret list` | Verify secrets are configured                            |
| Docker                 | `docker exec -it <container_id> bash` to debug container |
| Analytics Engine       | Query `fourai_cost_metrics` dataset via GraphQL          |

### Understanding Worker Logs

When the Worker starts, you'll see boot sequence logs:

```
[api/status] existing process: proc_1781540935673_g0ymd1 running
[ERROR] [api/status] Port wait failed: Process did not become ready within 5000ms
[ERROR] Uncaught ProcessReadyTimeoutError: ... Waiting for: port 18789 (TCP)
```

**These are normal during cold start.** The gateway takes up to 3 minutes to become ready. The Worker retries automatically.
