# Testing Guide

Testing infrastructure reference for all FourIA products. See [CONTRIBUTING.md](/latest/llms/platform/reference/contributing/index.md) for contribution workflow and [DEVELOPMENT.md](/latest/llms/platform/reference/development/index.md) for onboarding.

## Overview

| Product             | Type              | Framework                  | Unit Tests              | E2E Tests                      |
| ------------------- | ----------------- | -------------------------- | ----------------------- | ------------------------------ |
| `apps/fouria`       | Cloudflare Worker | Vitest + cctr + Playwright | Colocated `*.test.ts`   | Cloud + Local + Business Rules |
| `apps/lerma`        | Phoenix Web App   | ExUnit + cctr              | Mirror `lib/` structure | Sequential cctr scenarios      |
| `packages/moltlazy` | JS Config + SDK   | Vitest + cctr              | `tests/*.test.ts`       | cctr CLI scenarios             |

> The `cloudflare-unified-billing` plugin's unit and live-integration tests now live in [Paso4/openclaw-plugins](https://github.com/Paso4/openclaw-plugins).

---

## Quick Start

```bash
# All JS/TS tests (fouria + moltlazy)
bun run test

# fouria admin SPA UI tests (Playwright, local, no credentials)
bun run --cwd apps/fouria test:ui

# TypeScript type checking (Worker, moltlazy, cost-calculator, dispatch, tail)
bun run typecheck

# Lint all TS/JS sources
bun run lint

# Full CI suite (test + typecheck + local integration + moltlazy E2E)
bun run test:all
```

See each product section below for product-specific commands.

### Release tooling tests

`scripts/changeset-changelog.test.ts`, `scripts/release-lib.test.ts`,
`scripts/sync-versions.test.ts`, and `scripts/version-packages.test.ts` cover
the Changesets changelog format, the release-notes renderer, CalVer changelog
generation, the Lerma `package.json` → `mix.exs` version sync, and the
no-changesets no-op path. They run as part of `bun run test`.

---

## apps/fouria — Cloudflare Worker

### Unit Tests

Colocated `.test.ts` files alongside source files. Run with vitest.

```bash
cd apps/fouria
bun run test                  # All unit tests
bun run test:watch            # Watch mode
bun run test:coverage         # With Istanbul coverage (alias of coverage)
bun run test:client           # SPA React component/page tests (jsdom)
```

> The Worker unit project (`vitest.config.ts`, Node pool) excludes `src/client/**`.
> The SPA component and page tests run under the dedicated jsdom project
> (`vitest.client.config.ts`) via `bun run test:client` — they need the React
> plugin and a DOM, and declare `/** @vitest-environment jsdom */` per file.

#### Authentication, backup, and billing regression coverage

- `src/auth/middleware.fallbacks.test.ts` drives the Hono authentication and role-resolution
  chain: malformed forwarded JWTs, E2E cookies, local-role fallback, and Dispatch plan metadata.
  `src/auth/routing.test.ts` covers invalid tiers, input coercion, and hostname edges.
- `src/activity/scheduler.test.ts` covers durable boot success/retry, snapshot interval and idle
  guards, deprovision checkpoint-before-stop, restore jobs, maintenance jobs, and automation wake
  lead windows. `src/backup-watch/durable-object.test.ts` covers enqueue failures without clearing
  dirty state. `src/routes/api/storage.errors.test.ts` covers backup/restore conflicts, error
  responses, background failures, and job/config read failures. The persistence and encryption
  suites remain the owners of archive, lease, retention, and recovery-point guarantees.
- `src/routes/api/billing.freeze.test.ts` exercises persisted freeze/unfreeze, acting-user
  attribution, and the OR of stored and Dispatch freeze flags. `src/billable-actions.emit.test.ts`
  covers producer chunking, single-send fallback, delivery failures, frozen-customer exclusion,
  duplicate suppression, and fail-closed outcome normalization. A producer with neither `sendBatch`
  nor `send` must reject, never report unsent actions as enqueued.

Run `bun run test:coverage` from the repository root for comparable cross-package coverage.
The root `typecheck` delegates to the five workspace configurations listed above rather than
recursively checking vendored Elixir dependencies and marketing-site sources under one Node-only
configuration. Moltlazy has a package-local strict `tsconfig.json`; it no longer inherits the root
configuration. These runs do not prove deployed Cloudflare Access policies or live Stripe delivery;
unit line coverage is not a claim of complete end-to-end or branch coverage.

> **Billing-surface coverage.** `src/client/pages/__tests__/HomePage.test.tsx`
> renders the **Usage & Billing** widget with cost data present (total plus the
> billable action/credit rollup) and asserts the empty (`$0.00`) and role-hidden
> cases. `src/client/pages/__tests__/AccountPage.test.tsx` covers the **Plan &
> Billing** card (tier, seat/credit usage, past-due alert, non-admin hidden). The
> `/api/admin/costs/v2/summary` aggregation (AE SQL query → summary) is unit-tested
> in `src/cost-summary.test.ts` and `src/routes/api/costs.test.ts`.

### UI Tests (Playwright, local)

The admin SPA has a Playwright suite that drives the **real** dashboard served by
the Vite dev server (`bun run dev`), with every `/api/*` call mocked per test.
No Worker, sandbox container, or Cloudflare credentials are required, so it runs
fully offline and is the fastest way to exercise the UI end-to-end.

```bash
cd apps/fouria
bun run test:ui           # headless Chromium
bun run test:ui:headed    # watch the browser
```

Specs live in [`apps/fouria/test/ui/`](../apps/fouria/test/ui/) and the config is
[`apps/fouria/playwright.config.ts`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/playwright.config.ts). The
shared mock helper (`test/ui/support/api.ts`) stubs `GET /api/admin/whoami`
(role/dev/demo flags), the page data endpoints, and the `/api/health-ws` +
`/api/admin/devices/ws` WebSocket streams. Coverage:

- **App shell** — sidebar, notification bell, ADMIN navigation, dev-only nav.
- **Role access** — ROOT → Diagnostics redirect, ADMIN blocked from Diagnostics,
  BASE_USER limited nav + route guard.
- **Pages** — every admin route renders for its role.
- **Notification bell** — pending-device badge + dropdown, empty state.
- **Persistence lock indicator** (`PersistenceLockIndicator`) — the lock widget
  beside the bell reflects the shared `PersistenceLockContext` state, which the
  `BackupHistoryPanel` raises/clears around a backup or restore (no polling):
  open/green when idle, closed/red while an operation is in flight, and hidden
  for BASE_USERs. Unit coverage in
  `src/client/components/__tests__/PersistenceLockIndicator.test.tsx` and
  `src/client/components/__tests__/BackupHistoryPanel.test.tsx`.
- **Gateway circuit breaker** (`gateway-circuit-breaker.spec.ts`) — with the
  health stream `down`, gateway queries are parked (no `/api/admin/agents`
  request) and the waiting indicator shows; flipping the mocked health stream to
  `up` flushes the queued queries and renders the data. The mock helper exposes
  `installApiMocks(...).sendHealth(snapshot)` and the `HEALTH_UP` / `HEALTH_DOWN`
  fixtures for this. Unit coverage lives in `src/client/gateway-gate.test.ts`,
  `src/client/GatewayHealthContext.test.tsx`, and `src/client/api.test.ts`.
- **Diagnostics dashboard (skip-not-park)** — unit coverage in
  `src/client/pages/__tests__/DiagnosticDashboard.test.tsx` and
  `src/client/components/__tests__/ConfigSection.test.tsx`: with the gateway
  down the page still renders but its container reads (`/api/admin/config*`,
  devices) are never dispatched, gateway-only debug probes are skipped with an
  inline explanation, and the Configuration Editor stays visible with a
  non-blocking waiting notice instead of a blocking spinner.
- **Billing widget** (`billing-widget.spec.ts`) — the Home "Usage & Billing"
  widget shows the billed total and billable action/credit rollup when the cost
  summary carries data, `$0.00` when it is empty, and is hidden for BASE_USERs.
  The mock helper exposes a `costSummary` option.
- **Cloudflare Components widget** (`cloudflare-components.spec.ts`) — the ROOT
  diagnostics widget renders each component's status/latency/href, hides the AI
  Gateway when the API omits it, reveals the binding-derived `facts` panel on
  hover **and** keyboard focus, opens each card's Cloudflare dashboard logs page
  in a new tab (offline dashboard stub), renders a plain non-clickable card when
  a component has no `logsUrl`, re-fetches on Refresh, and reflects
  `sleeping`/`down` states. The mock helper's `HEALTH_COMPONENTS` fixture carries
  the enriched `facts` + `logsUrl` shape returned by `GET /api/health/components`.
- **Diagnostics responsive layout** (`diagnostics-responsive.spec.ts`) — at a
  phone viewport the Cloudflare Components strip collapses into a compact status
  widget and expands on tap, the gateway + persistence status widgets sit
  parallel to it (the topbar omits them on the diagnostics page), the Fouria
  DevOps options render as a vertical icon + name menu (collapsed by default,
  tap to open), the Configuration Editor reflows to the viewport, and the page
  never scrolls sideways (checked at 320px). The mock helper exposes
  `debugRoutes` (to render the Configuration Editor) and a `config` fixture for
  the parsed sections + raw JSON.

> **NixOS / Nix dev shell.** `apps/fouria/flake.nix` provides the Playwright
> browsers via `PLAYWRIGHT_BROWSERS_PATH` (`pkgs.playwright-driver.browsers`)
> plus the `PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS` and
> `PLAYWRIGHT_HOST_PLATFORM_OVERRIDE` NixOS workarounds. Because the Nix browser
> bundle can lag the project's npm `playwright` version, `playwright.config.ts`
> resolves the newest `chromium-*` under `PLAYWRIGHT_BROWSERS_PATH` and passes it
> as an explicit `executablePath`; outside Nix the variable is unset and
> Playwright's default resolution applies.

**Key test files** (see [WORKER-DEVELOPMENT.md](/latest/llms/platform/apps/fouria/worker-development/index.md) for full list):

| File                           | Tests                                                                                                                                              |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `auth/jwt.test.ts`             | JWT decoding and validation                                                                                                                        |
| `auth/jwks.test.ts`            | JWKS fetching and caching                                                                                                                          |
| `auth/middleware.test.ts`      | Auth middleware, DEV_MODE, E2E_TEST_MODE, dev-instance ROOT→ADMIN role preview                                                                     |
| `gateway/env.test.ts`          | Container environment variable building                                                                                                            |
| `gateway/process.test.ts`      | Process finding, lifecycle, stuck-start watchdog, and structured gateway-start failure correlation                                                 |
| `gateway/classify.test.ts`     | `classifyGatewayStart` running-only stuck-start verdict                                                                                            |
| `gateway/sandbox-exec.test.ts` | Sandbox command results and structured process failure logs                                                                                        |
| `gateway/utils.test.ts`        | Process wait lifecycle and non-zero exit reporting                                                                                                 |
| `index.test.ts`                | Worker middleware, proxy routes, and request failure trace context                                                                                 |
| `gateway/r2.test.ts`           | R2 mounting logic                                                                                                                                  |
| `auth/rbac.test.ts`            | RBAC permission matrix                                                                                                                             |
| `metrics.test.ts`              | Cost/usage Analytics Engine instrumentation                                                                                                        |
| `persistence.test.ts`          | R2 backup/restore, retryable cold-restore intent, scheduled-snapshot safety, cross-isolate lease, crash-consistent catalog + orphan reconciliation |

### E2E Tests (Cloud)

Deploys a real worker to `*.workers.dev` via Terraform + `wrangler deploy`. Requires Cloudflare credentials.

```bash
cd apps/fouria

# Full E2E with browser
bun run test:e2e

# CLI-only (no browser)
bun run test:e2e:cli
```

**Setup** (`test/e2e/_setup.txt`):

1. Provisions cloud infrastructure via Terraform (R2 bucket) — the content-hash worker cache is **temporarily disabled** (issue #297), so every run provisions fresh
2. Deploys worker via `wrangler deploy --var E2E_TEST_MODE:true`
3. Starts Playwright browser via `plwr`
4. Polls `/api/health` for cold start up to the startup budget — 30s on a warm cache hit, otherwise `STARTUP_TIMEOUT_SECS` (600s default)
5. Opens browser to worker URL, waits for "Device pairing required"

> **Gateway readiness — shared helper.** After provisioning, the cctr
> `_setup.txt` waits for gateway readiness through
> [`apps/fouria/test/lib/e2e-helpers.sh`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/lib/e2e-helpers.sh).
> Its `e2e_wait_for_health` probes `/api/health` **immediately** (the first probe
> also triggers the cold boot — there is no fixed 180s pre-poll grace) and
> requires **both** `startup.ok` and `readiness.ok`, polling every 5s. The wait
> stays bounded by the existing warm/cold startup budget: 30s on a warm cache
> hit, otherwise `STARTUP_TIMEOUT_SECS` (600s default). The same helper backs
> `test/e2e-agent/_setup.txt` and the parallel driver's top-level readiness wait.
> It does **not** back every readiness check: the parallel runner's generated
> per-group `_setup.txt` and `ab_gateway_ws_origin.txt` keep their own
> grace-free, bounded `/api/health` loops (the latter also uses the shared
> `e2e_classify_ws_upgrade` verdict helper). The helpers are unit-tested in
> `apps/fouria/src/e2e-helpers.test.ts`.

> **E2E cost safety (issue #297):** the container idle timeout is **derived** from the
> instance hostname (staging 15m / dev 2h — the Sandbox SDK 1.0 `setInactivityTimeout()`
> platform cap); there is no `SANDBOX_SLEEP_AFTER` knob. Billable production instances
> (production platform, production environment) do **not** auto-sleep. A container that
> outlives its run (failed teardown, workflow timeout) therefore stops billing at the
> derived idle instead of running 24/7, and the workflow teardown deletes the workers
> outright. Worker caching is disabled until the feature is reworked to reuse only
> within working hours and to delete on eviction.

> **Boot profiles:** Both profiles provision the same **production-profile
> image** (`apps/fouria/Dockerfile`, `BUILD_PROFILE=production`, the slim
> default) **and the same `standard-2` instance type** (1 vCPU / 6 GiB — the
> smaller `basic` 1 GiB / `standard-1` 0.5 vCPU instances stall the gateway under
> the suite's CLI load). The optional PDF/document tooling (`pandoc` +
> `weasyprint`) ships only in the `dev` build profile used by the dev/staging
> environment — see `apps/fouria/docs/WORKER-DEVELOPMENT.md`. PR runs use the
> light `basic` profile (`E2E_TEST_MODE=true`, short plugin timeouts) for fast
> feedback; develop runs use the `production` profile (real plugin timeouts,
> container OTLP export + Cloudflare Tunnel). The profiles differ by env only —
> not the image or instance type. Both profiles bind the worker OTLP receiver
> with a **random per-run `OTEL_SHARED_SECRET`** (no GitHub secret required), so
> the OTel billing-queue test runs in either profile.

> **Single-worker parallel E2E:** the `e2e` workflow provisions one isolated
> worker and runs dependency-grouped cctr suites against it. Independent phase A
> and B groups run concurrently; config/lifecycle/restore groups remain serial.
> Pull requests run boot-only by default. `develop` and a full manual dispatch
> run the corpus; selective retries link only the suites recorded in the prior
> run's `e2e-failed-*` cache. The runner primes the CLI/device path once per
> group so device-admin suites do not each pay the cold first-connect cost.

> **E2E run statistics:** `run-parallel.sh` writes `e2e-stats.md` with initial
> and per-group Gateway readiness times, cctr group outcomes and case durations,
> and per-group counts of Worker-launched OpenClaw CLI calls, Gateway WebSocket
> connections, and Admin HTTP RPC calls. The report is uploaded beside debug
> logs to the `e2e-artifacts-*` branch. Transport counters require an explicit
> `X-FourIA-E2E-Stats: 1` request, `DEBUG_ROUTES=true`, and an explicitly bound
> `E2E_TEST_MODE` profile marker (`true` for `basic`, `false` for `production`).
> The long-lived Gateway process is excluded from CLI totals. CI pins cctr to
> `0.31.0` for stable report parsing.

> **Failed-only reruns:** when a run reaches the gateway (`gateway-ready`) but
> some suites fail, the job records those suite names and caches them
> (`e2e-failed-<ref>-<shard>`). The next run restores that list and passes it to
> `run-parallel.sh`, which links only those suites into the generated cctr groups
> (the PR already passed unit + `validate-secrets`, enforced by `needs`). A full pass clears the list
> and the following run is a full pass again. A committed seed
> (`.github/e2e-rerun-suites.txt`, only when present) bootstraps a subset before
> any cache exists; it is not required in steady state.

**Teardown** (`test/e2e/_teardown.txt`):

1. Dumps gateway logs + doctor output
2. Stops browser (saves video)
3. Destroys cloud resources (worker, container app, R2 bucket, Terraform)

**Test files** (alphabetical order — naming determines execution order):

| File                                 | What it tests                                                                                              |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| `b_cli_devices.txt`                  | `openclaw devices list` through `/debug/cli`                                                               |
| `c_cli_models.txt`                   | `openclaw models list`, model catalogs                                                                     |
| `d_cli_channels.txt`                 | Channel probe, config inspection                                                                           |
| `e_cli_config.txt`                   | Config validate, set, unset via CLI                                                                        |
| `g_cli_secrets.txt`                  | Secrets audit, reload via CLI                                                                              |
| `h_cli_knowledge.txt`                | Wiki init, status, get via CLI                                                                             |
| `i_cli_cron.txt`                     | Cron list, add dream job                                                                                   |
| `j_cli_integrations.txt`             | Integration config verification                                                                            |
| `k_secrets_store.txt`                | Secrets Store (gateway token, CDP, container config, device pairing)                                       |
| `l_cli_billing.txt`                  | Pricing constants, costs, instance types, billable events                                                  |
| `m_cli_agent_tokens.txt`             | Agent turns, token usage metrics                                                                           |
| `provision_and_dispatch.txt`         | **Cloud-only** — dispatch worker + Supabase tenant lifecycle                                               |
| `a_rbac_admin_isolation.txt`         | ADMIN role isolation (management API, cross-tenant)                                                        |
| `a_rbac_api_matrix.txt`              | ROOT/ADMIN/BASE_USER matrix + deny/allow decisions over dashboard + loading-page endpoints                 |
| `a_rbac_base_user_isolation.txt`     | BASE_USER isolation (whoami, blocked from management/secrets/writes)                                       |
| `a_rbac_device_pairing.txt`          | Device pairing permissions for all 3 roles (with retry on timeout)                                         |
| `a_rbac_root_isolation.txt`          | ROOT isolation (diagnostics access, blocked from management/agents/secrets)                                |
| `a_rbac_ui_pages.txt`                | UI page access for all 3 roles (25 test cases; every nav page route served)                                |
| `aa_health_gate.txt`                 | Gateway health check between RBAC and CLI test groups                                                      |
| `ac_gateway_startup_concurrency.txt` | Cross-isolate cold-start serialization after destroying the container                                      |
| `ad_observability_canary.txt`        | **Local-only** — proxy header redaction + live-gateway-log RPC-stream warning (Local Explorer SQL API)     |
| `o_opencode_credits_task.txt`        | OpenCode credits automation executed in the sandbox (direct consumer)                                      |
| `o_leads_proxy.txt`                  | Leads R2 proxy over the Worker WebSocket (self-skips without `cdp-secret.txt`)                             |
| `p_tasks_queue.txt`                  | TASKS queue producer → consumer → sandbox round trip (self-skips w/o key)                                  |
| `p_otel_billing_queue.txt`           | OTLP traces receiver → billable reconcile → BILLING_QUEUE producer (self-skips on a non-billable instance) |
| `test/e2e-billing/`                  | **Local-only** — billable worker emits to BILLING_QUEUE (observed via Local Explorer) + freeze drop        |
| `p_scheduled_cron.txt`               | **Local-only** — both crons fire via `/cdn-cgi/local/scheduled`                                            |
| `z_cli_staging.txt`                  | API staging tests (costs, storage, users)                                                                  |
| `zzz_cron_wake.txt`                  | **Cloud-only** — container destroy, cron wake cycle                                                        |

> **Backup/restore suites are excluded from this corpus.** They live in
> [`apps/fouria/test/e2e-backup/`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/e2e-backup/README.md) so a
> default `bun run test:e2e` / `test:e2e:local` run does not trigger their
> multi-minute cold boots and ~250 MB R2 snapshots. Run them with
> `bun run test:e2e:backup` (local) or through `phaseE_restore` in the parallel
> runner. Files: `zz_reprovision_guard`, `zzz_backup_cold_restore`,
> `zzz_backup_restore_skip_setup`, `zzz_backup_semantics`, and the legacy
> `zzz_persistence_lifecycle`.

> **Retry diagnostics in cctr tests.** cctr merges a test's stdout **and stderr**
> into the value matched by `{{ … }}`. A retry/progress line printed to `>&2`
> therefore breaks a strict matcher (`json object`, `json bool`, a bare number,
> …) once the retry path is taken — the flake only appears when the first probe
> is unhealthy. Write such diagnostics to a side log instead:
> `echo "…" >>"$CCTR_WORK_DIR/e2e-retry.log" 2>&1`.

> **`o_leads_proxy` requires `cdp-secret.txt`.** The test self-skips (and passes
> in ~0s) when the fixture file is missing, so a regression in the fixture wiring
> looks like a green test. `test/e2e/fixture/server/start` persists
> `$CDP_SECRET` to `cdp-secret.txt` on **both** the cache-hit and fresh-provision
> paths; if the test reports `skipped: true`, the fixture is the place to look.

> **`p_otel_billing_queue` self-skips on a non-emitting instance.** The OTLP
> traces receiver gates emission on `isBillableEventsEnabled` (`src/dev-mode.ts`):
> billable production **and the staging platform always emit**, while a
> dev-environment instance accepts + ignores approved spans (HTTP 200) and
> reports `billable: false`, never emitting to `BILLING_QUEUE`. The local E2E
> worker runs with `DEV_MODE=true` and no routable `WORKER_URL`, so it is a dev
> instance — the test skips (like it already does for a missing secret / `401`)
> instead of asserting an emit that cannot happen locally. The emit path is
> asserted by the cloud profile (a production `workers.dev` hostname / the
> staging platform) and by `src/routes/otel-traces.test.ts`. A dev instance can
> opt in explicitly with `BILLABLE_EVENTS=true`; the override changes emission
> only, so the sleep policy and the cost-row dimensions are untouched.
> `provision_and_dispatch` likewise self-skips in local mode (no dispatch worker
>
> - Supabase).

> **Worker billing-emit E2E (`test/e2e-billing/`, local-only).** The
> producer half of the billing workflow is covered locally by opting the worker
> into a **billable** identity (`E2E_BILLABLE=true` → `start-local` passes a
> production-looking `WORKER_URL`/`CUSTOMER_SLUG`). The suite POSTs a synthetic
> OTLP span, asserts `emitted >= 1`, observes the `BILLING_QUEUE` producer via
> the Local Explorer `queue*` span, then freezes the customer and asserts
> `emitted == 0` with no new queue span. Analytics Engine writes are not
> observable locally (no AE binding in Local Explorer; no `writeDataPoint`
> span), which is why the lerma consumer suite uses a fake AE. Run with
> `bun run --cwd apps/fouria test:e2e:billing:local`; see
> `apps/fouria/test/e2e-billing/README.md`.

> **`h_cli_knowledge` caches the owner agent id.** The corpus resolves the owner
> agent id **once per suite** through `test/e2e/fixture/get-agent-id`, which
> caches it in `$CCTR_FIXTURE_DIR/agent-id.txt` and reuses it across the
> sequential `wiki init` / `wiki status` / `wiki get` cases. The helper reads
> `/api/admin/agents` through Admin HTTP RPC, then Gateway WebSocket RPC, with
> the route's CLI fallback for compatibility; caching avoids repeating that
> roster lookup per case while preserving the plugin/wiki assertions. The cache
> is suite/worker-run scoped (cctr copies `fixture/` into
> `$CCTR_FIXTURE_DIR` once and reuses that directory across cases) and is
> discarded when cctr cleans up the per-suite temporary fixture/work directory,
> not by `_teardown.txt`. The helper fails closed: an empty or malformed roster
> exits nonzero instead of silently defaulting to `main`.

### E2E Tests (Local)

Runs tests against the local Worker dev server (Cloudflare Vite plugin, `vite dev`) with containers via Docker. Avoids 5-10 minute Terraform provisioning. Cloud-only tests (dispatch worker + Supabase, container destroy/recreate) self-skip in local mode.

```bash
cd apps/fouria

# Local E2E with Docker + cctr (auto-builds Docker image)
bun run test:e2e:local

# CLI-only (no browser)
bun run test:e2e:local:cli
```

**Setup** (`fixture/server/start-local`):

1. Sources `test/e2e/.dev.vars` and installs it as the Worker's `.dev.vars` for the run (the Cloudflare Vite plugin has no `--env-file`; `stop-local` restores the developer's file)
2. Finds unused port (8787-8797)
3. Starts `vite dev --port <PORT> --strictPort` (also generates `CDP_SECRET`/`OTEL_SHARED_SECRET`, injected as vars through the plugin's `config` hook, and uses `/cdn-cgi/local/scheduled` for the cron test). The outer `test:e2e:local*` scripts run `bun run build:docker` first (chained with `&&`, so it completes before the dev server starts); `vite dev` then builds/launches the container image through Docker.
4. Polls `GET /api/health?passive=1` until the Worker answers **any** HTTP status (`200`/`503`/…) — a cold boot legitimately answers **503** while the container/gateway is still starting, so the gate must not require a `200` and must not use `curl -f`. It probes the fast, **non-waking** passive endpoint so a poll does not wake the container or re-trigger the restore/`ensureGateway` boot on every attempt. The window defaults to 120s and is overridable with `E2E_STARTUP_TIMEOUT_SECS`. The real gateway readiness wait (`startup.ok` + `readiness.ok`) is the separate `wait for gateway API readiness` setup step.
5. Writes fixture files (dummy CF Access creds — ignored via `DEV_MODE=true`)

> **Resource headroom.** The local run drives the Docker daemon (container image +
> container), `workerd` (Vite dev server + Miniflare R2/DOs), and the `bun`/`cctr`
> drivers at once. On a memory-constrained host the whole process group can be
> killed by `SIGABRT` (OOM) mid-cold-boot — observed during
> `_setup: start moltworker server` after the image built and the dev server
> answered 503. Give Docker/the host at least ~8 GB and avoid running other
> containers alongside the suite. The readiness gate above fails fast when
> the dev server itself dies, but it cannot distinguish an OOM kill from a crash,
> so check `dmesg`/Docker logs when the process group dies during startup.

**Teardown** (`fixture/server/stop-local`):

1. Kills the Vite dev server process
2. Restores the app's `.dev.vars`
3. Cleans up fixture files

**Prerequisites:**

```bash
# test/e2e/.dev.vars must contain:
DEV_MODE=true
E2E_TEST_MODE=true
DEBUG_ROUTES=true
MOLTBOT_GATEWAY_TOKEN=<value>
CF_ACCOUNT_ID=<id>
CLOUDFLARE_AI_GATEWAY_API_KEY=<key>   # plus CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID
# The REST transport uses CLOUDFLARE_API_TOKEN, derived from the same AI-scoped key.

# Docker must be running
# wrangler must be authenticated via OAuth (wrangler login)
```

**Cloud-only tests** (self-skip when `E2E_LOCAL=true`):

| File                         | Reason                              |
| ---------------------------- | ----------------------------------- |
| `provision_and_dispatch.txt` | Requires dispatch worker + Supabase |
| `zzz_cron_wake.txt`          | Requires container destroy/recreate |

> **`ad_observability_canary` asserts on the local worker's own logs.** The
> Local Explorer exposes the same `logs` / `spans` tables as the Cloudflare
> Workers Observability API, scoped to the local dev worker, via
> `POST /cdn-cgi/local/explorer/api/local/observability/query` (wrapped by the
> `fixture/observability-query` helper). The canary uses it to (1) drive the
> catch-all proxy's request log with sentinel Cookie / Authorization /
> CF-Access-Jwt-Assertion / tenant-metadata values and assert **none** reach the
> log store, and (2) fetch the running gateway's logs and assert the fetch adds
> no `ReadableStream received over RPC disconnected prematurely` warning. Both
> cases are **local-only** (the API does not exist on a deployed worker), so
> they self-skip in cloud E2E; they **fail loudly** if the local API is
> unavailable rather than silently skipping, so the coverage cannot rot. These
> are the observability-validation canaries for the credential-logging and
> live-process-stream fixes.

### Admin UI Render Timing (local-only, `plwr`)

`test/e2e/ui_components_render.txt` is a **local-only** browser suite that
validates the fouria admin (root) UI components and reports how long each takes
to render. It drives the `plwr` session started by `_setup.txt` (the same one the
pairing step uses), sets the `X-Mock-Role` cookie, loads each admin route, waits
for the component's root selector, and measures, in milliseconds relative to the
page navigation start:

- `openMs` — navigation start → SPA loaded (`performance.now()` sampled right
  after `plwr open` returns)
- `componentMs` — navigation start → the component's selector is present
  (sampled right after `plwr wait` returns); this is the data-driven render
- `waitMs` — `componentMs − openMs` (client render + API round-trip after load)
- `nav.*` — Navigation Timing (`domInteractive`, `domContentLoaded`, `load`) and
  First Contentful Paint

Coverage mirrors the Playwright UI suite (`test/ui/pages.spec.ts`): the shell
(`.admin-layout`, `.sidebar`, gateway status LED, persistence lock, notification
bell) and every ADMIN page, plus the ROOT-only diagnostics page and its Fouria
DevOps + Cloudflare Components widgets.

The file also carries one **behavioral** case (driven directly through `plwr`)
that loads the diagnostics page against the local worker and asserts the
Cloudflare Components widget links the container card to its Cloudflare
dashboard page (`href` contains `dash.cloudflare.com`) and reveals its
binding-derived detail panel on hover (`visibility` flips `hidden` → `visible`,
panel text contains the `Runtime` fact). It self-skips when browser tests are
disabled or `plwr` is absent.

The file self-skips unless `E2E_LOCAL=true` (file-level `%skip`) and also
self-skips when `E2E_SKIP_BROWSER_TESTS` is set, so cloud runs stay green. The
`fixture/ui-render-measure` helper appends every record to
`ui-render-times.jsonl`; the final `ui-render-report` case writes a table to
`ui-render-report.txt` and prints a JSON rollup (stdout) with `maxComponentMs`
and the slowest component.

```bash
cd apps/fouria

# Just this suite — _setup.txt still starts the worker + browser:
bun run test:ui:render:local   # build:docker + E2E_LOCAL=true cctr -vv test/e2e/ -p 'ui render'

# Or run it as part of the whole local corpus:
bun run test:e2e:local
```

### E2E Agent Subset (OpenClaw migration smoke)

`apps/fouria/test/e2e-agent/` is a focused, browser-free subset of the CLI
surface used to validate an OpenClaw version migration. It reuses the shared
`test/e2e/fixture` (via symlink) and the same local-mode runner, but only runs
the commands the Worker shells out to:

- `b_cli_devices.txt` — `openclaw devices`
- `c_cli_models.txt` — `openclaw models`
- `d_cli_channels.txt` — `openclaw channels`
- `e_cli_config.txt` — `openclaw config validate`
- `g_cli_secrets.txt` — `openclaw secrets`
- `h_cli_knowledge.txt` — `openclaw wiki`
- `m_cli_agent_tokens.txt` — `openclaw agents` / token flows
- `n_gateway_soft_reload.txt` — gateway soft reload (`SIGUSR2`; asserts the Node inspector stays closed and proves the restart with a new `http server listening` line plus a stable `processId`)

> **Bounded agent turns.** `m_cli_agent_tokens.txt` bounds its Gateway turn with
> `openclaw agent --timeout 40`. The CLI's default turn deadline is 600s, which
> outlives the e2e HTTP budget (`debug-cli` waits up to 80s server-side and 90s
> via `curl-auth`; `/debug/cli` uses `timeout_ms`). A bounded turn returns its
> JSON envelope (result, timeout, or provider error) within the budget, and an
> empty response is reported as `{"status":"timeout"}` instead of failing the
> JSON-object assertion on an empty `jq` result.

```bash
cd apps/fouria
bun run test:e2e:agent   # build:docker + E2E_LOCAL=true + browser skipped
```

The [openclaw-migration](https://github.com/0xCAB0/fouria/blob/develop/skills/openclaw-migration/SKILL.md) skill runs this
subset after bumping `catalog.openclaw` to confirm the CLI commands still
behave.

### Gateway Boot Smoke Test (local)

Validates that the fouria sandbox image actually **boots a healthy OpenClaw
gateway** on the local machine — no Worker, no cloud e2e, no third-party
machines. It builds the exact production image (`apps/fouria/Dockerfile` +
repo-root context, same base `cloudflare/sandbox` image wrangler uses) and runs
the real boot path (`start-openclaw.sh` → onboard → plugin installs → moltlazy
patch/validate → gateway launch → health poll) inside Docker. The container's
exit code is the verdict: the in-container wrapper runs `start-openclaw.sh`
(whose own readiness loop already fails if the gateway dies or never binds),
then asserts `/health` returns `{ok: true}` while the gateway is still alive.

```bash
cd apps/fouria

# Default cold-boot smoke (no real credentials needed):
#   onboard runs --auth-choice skip, tunnel + OTEL steps self-skip.
bun run test:gateway:boot

# Standalone native phases in the smoke container: `start-openclaw.sh setup
# → start-openclaw.sh launch → post-start`, then assert /health. This is NOT
# the Worker's production cold start, which is separately `restore → atomic
# start (setup + Gateway under one lock) → post-start` (see the table below).
bun run test:gateway:boot --native

# Same, plus the real cloudflare-ai-gateway onboard path and an assertion
# that the provider plugin is installed + enabled. Sources credentials from
# apps/fouria/.dev.vars.e2e (never echoed) — generate it from SECRETS.yaml
# with `bun run secrets:generate`.
bun run test:gateway:boot:ai

# Leave the container running after the assertion, with port 18789
# published to 127.0.0.1 for manual inspection (Control UI / WS).
bun run test:gateway:boot --keep

# Skip build:docker + docker build and validate an existing image.
bash scripts/gateway-boot-smoke.sh --no-build --image <tag>
```

**Prerequisites:** Docker running, network access (the image build pulls the
ClawHub plugins; boot-time installs are a fallback only, bounded by
`start-openclaw.sh` timeouts and non-fatal for boot).

**What it validates vs. what it does not:**

| Validated locally                                                                                                                                                                                               | Still needs local e2e / cloud e2e                                 |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| Image builds from the production Dockerfile                                                                                                                                                                     | Sandbox SDK supervisor + Worker RPC layer                         |
| onboard (skip or real AI Gateway) completes                                                                                                                                                                     | RBAC / API routes / device pairing                                |
| Build-time plugin pre-install + enablement                                                                                                                                                                      | Dispatch worker routing                                           |
| moltlazy `patch` + `validate` (config contract)                                                                                                                                                                 | Cost observability / R2 backup flows                              |
| gateway binds `:18789`, `/health` returns ok                                                                                                                                                                    | Browser-driven Control UI flows                                   |
| `--native`: standalone `start-openclaw.sh setup` → `launch` → `post-start` phases in the smoke container (not the Worker's production `restore` → atomic start (setup + Gateway under one lock) → `post-start`) | Worker's `restore` + `sandbox.exec` supervision (handle/relaunch) |

### E2E Parallel (single-worker parallel runner)

`apps/fouria/test/e2e-parallel/` is the **parallel replacement for the serial
`test/e2e` run**. The `e2e` CI job no longer shards the corpus across N workers
(one per shard); it runs `run-parallel.sh`, which provisions **ONE** worker and
runs dependency-grouped cctr suites **concurrently** against it.

```bash
cd apps/fouria
bun run test:e2e:parallel        # == bash test/e2e-parallel/run-parallel.sh
```

**Suite scope (`E2E_BOOT_ONLY`).** By default the `e2e` CI job sets
`E2E_BOOT_ONLY=1` for **pull requests**: the runner provisions the worker, waits
for `/api/health` to report `startup.ok + readiness.ok`, and then tears down
**without running any cctr group** — a PR only proves the gateway boots. The full
corpus runs on `develop` (merge gate) and on a manual `workflow_dispatch` with
the **`full_tests`** input checked (the default for dispatch). Locally, set
`E2E_BOOT_ONLY=1` to reproduce the PR check.

Phases (see the suite README for the full grouping):

- `phaseA_devices` ∥ `phaseA_rbac` ∥ `phaseA_health` — parallel (the RBAC mock
  role is a per-request cookie, so the RBAC suites are independent; the two
  device-store suites share `phaseA_devices` to avoid racing the approve path)
- `phaseB_readonly` ∥ `phaseB_async` — parallel
- `phaseC_stateful` — config/secrets/cron (serial)
- `phaseD_lifecycle` — plugin + reload/restart + dispatch (serial)
- `phaseD_startup` — cross-isolate cold-start concurrency regression and the
  declarative software-provisioning suite (`zzz_software_provisioning.txt`:
  flag/report wiring, manifest validation + upload, cold reprovision installs the
  declared software, boot-terminal ticks, and an **on-demand** run that installs
  a newly declared package while skipping the already-installed one), serial,
  each with a separate startup-sized budget
- `phaseE_restore` — reprovision guard followed by the backup/restore lifecycle
  suites, resolved from `test/e2e-backup/` (serial, startup-sized budget).
  **Skipped in GitHub Actions by default** because it drives multi-minute cold
  boots and ~250 MB R2 snapshots; run it locally, or force it in CI with
  `E2E_RUN_RESTORE_PHASE=1`. For a standalone local run of just these suites use
  `bun run test:e2e:backup`.

> **Phase caps.** `phaseD_lifecycle` is bounded by `PHASE_D_TIMEOUT_SECS`
> (default **420s / 7 min**). The cold-start concurrency group has its own
> `PHASE_D_STARTUP_TIMEOUT_SECS` budget (default `STARTUP_TIMEOUT_SECS + 180s`);
> backup/restore suites use `PHASE_E_TIMEOUT_SECS` (default
> `STARTUP_TIMEOUT_SECS + 300s`). The full run's initial readiness budget is
> 1200s (`basic`) or 1500s (`production`). On a group timeout cctr fails
> (`exit 124`) and teardown still runs.

> **cctr 0.31.0 has no `#` comments.** cctr silently drops a corpus file's tests
> at the first top-level `#` line and reports the tests collected so far (often
> zero) with no error. Corpus files must begin with `===` (or a lone file-level
> `%skip`/`%shell`/`%platform` followed by the first `===`), and a `#` comment
> must never sit before the first `===`, between a file-level `%skip` and the
> first `===`, or between two tests. **Put rationale in test descriptions or docs,
> never in a `#` line** (comments _inside_ a test body/expected/`where` block are
> fine). `node scripts/check-corpus-comments.mjs` — run by the `unit` CI job —
> fails on any violating file, and `cctr -l <file>` prints the parsed test count.

> **Selective retry (`retry_failed`).** A slow container can make a single backup
> call exceed its curl budget, failing a suite that would pass on a warm run.
> Rather than inflate the curl timeouts (which hides real hangs), the workflow
> records the failed suites — the `Record failed e2e suites` step writes
> `✗ <group>/<suite>` markers to `/tmp/moltworker-e2e-failed.txt`, saved to the
> `e2e-failed-<ref>-<shard>-<run_id>` actions cache. Re-run **only those suites**
> with a manual `workflow_dispatch` and the **`retry_failed`** input checked:
> `run-parallel.sh` reads the restored file, links only the failed suites into
> their groups, skips every other group, and still provisions a single worker.
> The flag forces the suite to run (it overrides `full_tests`/boot-only). A
> `phaseX_group/*` entry (written when a group is killed by its phase cap)
> retries every suite in that group. The cache is saved only when something
> failed, so the restore always resolves to the most recent failing run.

> **Single-owner startup concurrency regression.**
> `ac_gateway_startup_concurrency.txt` is a cloud-only regression for the OpenClaw
> 2026.9.8 cross-isolate launch race: it checkpoints tenant state, destroys the
> container, fires several concurrent cold **non-passive** `/api/health` probes
> (the cold-boot fan-out across Worker isolates), then asserts the gateway reaches
> `startup.ok + readiness.ok`, that exactly **one** _running_ Gateway owner
> exists — the process may appear as `start-openclaw.sh start` before the atomic
> wrapper `exec`s, or as `openclaw gateway run` after — and that the observable
> gateway output never shows **either**
> `OpenClaw migration inputs changed during startup` **or**
> `state.sqlite is undergoing offline maintenance`. Because it destroys the
> shared worker's container it runs in `phaseD_startup`, after the ordinary
> lifecycle group and before restores, with its own startup-sized budget. It runs
> **serially** — never alongside the parallel phase A/B suites. Run it directly
> with `cctr apps/fouria/test/e2e/ac_gateway_startup_concurrency.txt`; it self-skips
> under `E2E_LOCAL=true`.

Each group is a generated cctr suite whose test files **symlink** the originals
in `test/e2e/` and whose `_setup.txt` **attaches** to the shared worker (copy
credentials → wait readiness → warm the CLI/device identity). It forces
`E2E_NO_CACHE=1` (full teardown, no container left billing) and self-skips under
`E2E_LOCAL=true`.

> **Readiness wait.** The driver's top-level wait uses the shared
> `e2e_wait_for_health` helper (immediate first probe, no fixed grace, bounded by
> `STARTUP_TIMEOUT_SECS`). The runner defaults that budget to **1200s**; the
> workflow overrides it per profile (basic **1200s**, production **1500s**). The
> generated per-group `_setup.txt` do **not** call the shared helper — each keeps
> its own grace-free, budget-bounded `/api/health` loop that attaches to the
> already-warm worker. Because the phases run concurrently against one worker,
> the per-group readiness savings do **not** sum to the same job-time reduction —
> wall time tracks the slowest phase, not the total of the individual waits.

`parallel-probe` + `a_parallel_spike.txt` remain as a raw-concurrency
**diagnostic** (how many concurrent CLI calls one worker sustains); run the
probe suite directly with `cctr -vv test/e2e-parallel/`.

**Logging policy.** Verbose transcripts — the provisioning output
(terraform/wrangler), each group's `cctr -vv` output, and the teardown/gateway
dumps — are written to `/tmp/moltworker-e2e-logs/*.log` and uploaded to the
`e2e-artifacts-<profile>` branch, so the CI job console only shows concise
`[parallel]` progress plus per-suite pass/fail markers and the summary. The
serial `test/e2e` `_teardown.txt` follows the same rule: gateway/doctor dumps
are written to disk only (never `tee`'d into the console).

See [`apps/fouria/test/e2e-parallel/README.md`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/e2e-parallel/README.md)
for the per-file parallelizability classification, the ordering graph, and the
observed concurrency results.

### Backup Semantics E2E

`apps/fouria/test/e2e-backup/zzz_backup_semantics.txt` and
`.../zz_reprovision_guard.txt` prove the backup invariants against a real
container (local Docker or cloud) and real R2. They are registered in
`phaseE_restore` (serial; reprovision guard runs first) and are the E2E counterpart of the route/unit tests
(`src/routes/public.test.ts`, `src/routes/debug.test.ts`, `src/index.test.ts`) and
the dispatch fan-out unit test (`workers/fouria-dispatch/test/unit/cron-fanout.test.ts`).

Run the local-safe cases standalone (excluded from the default `test/e2e` corpus):

```bash
cd apps/fouria
bun run test:e2e:backup   # build:docker + E2E_LOCAL=true + browser skipped
```

See [`apps/fouria/test/e2e-backup/README.md`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/e2e-backup/README.md).

| Case                                                             | Invariant                       | Mode           |
| ---------------------------------------------------------------- | ------------------------------- | -------------- |
| health probes leave the catalog unchanged                        | I1 health is observability-only | local + cloud  |
| `POST /control` `X-FourIA-Control: backup` on a running instance | I2                              | local + cloud  |
| async sync commits a durable job and promotes a stable pointer   | I12 durable job + stable point  | local + cloud  |
| idle instance returns `skipped:'not-running'` without waking     | I2/I3                           | cloud          |
| control-backup → destroy → cold restore preserves a sentinel     | I4                              | cloud          |
| encryption at rest (`data.sqsh.enc`, no plaintext)               | I8                              | cloud (opt-in) |
| retention prunes catalog + objects to `BACKUP_MAX_VERSIONS`      | I9                              | cloud          |
| worker is bound only to its per-run bucket                       | I11                             | local + cloud  |
| reprovision 409 / `force` / running checkpoint                   | I5/I6/I7                        | cloud          |

### Agent Push (promotion) E2E

`apps/fouria/test/e2e-push/` validates the **dev → production agent push**: the
promotable manifest (config + agent workspace, dev-only agents and runtime state
excluded) is pushed from a tenant's `dev` instance to its `production` instance
over HTTP as a custom backup that keeps the production instance's runtime state.

It provisions **two** local Workers side by side — a `dev` instance whose
`WORKER_URL` carries the `-dev` hostname suffix (ships `fouria-builder`/
`fouria-qa`) and a `production` instance whose `WORKER_URL` does not — on two
ports, then asserts that a generated agent propagates while the dev-only agents,
their personas, and all runtime state do not, and that the production instance
stays healthy across the deferred gateway reload.

> The dev/production axis is the `-dev` **hostname** (`isDevAgentsEnabled`), not
> `DEV_MODE`. `DEV_MODE` is only the local Cloudflare Access bypass (from
> `.dev.vars`); a tenant's real `dev` instance is a normal production-platform
> instance and is not in `DEV_MODE`.

```bash
cd apps/fouria
bun run test:e2e:push:local   # build:docker + E2E_LOCAL=true cctr -vv test/e2e-push/
```

Local-only (the first `%require` case self-skips outside `E2E_LOCAL=true`); it is
excluded from `test/e2e` and the parallel runner because it boots two full
gateways. See
[`apps/fouria/test/e2e-push/README.md`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/e2e-push/README.md).

Cloud helpers: `test/e2e/fixture/r2-list` and `r2-clear` list/empty the per-run
bucket via the R2 REST API (needs `CLOUDFLARE_TEARDOWN_API_TOKEN` or
`CLOUDFLARE_API_TOKEN`). The E2E deploy sets deterministic
`BACKUP_INTERVAL_MINUTES=1` / `BACKUP_MAX_VERSIONS=2`; set
`E2E_BACKUP_ENCRYPTION=true` to run the encrypted-at-rest case. Note that the
configured interval is a **floor**: the Worker stretches the effective scheduled
cadence to `max(BACKUP_INTERVAL_MINUTES, 2 × the last snapshot duration)`, so a
slow snapshot cannot run back-to-back. Explicit snapshots (`POST
/api/admin/storage/sync`, checkpoints) bypass that gate and stay immediate.

The same invariants are asserted at the fast layer by
`workers/fouria-dispatch/test/unit/cron-fanout.test.ts` (the cron forwards
`backup` only to non-frozen tenants) and `src/persistence.test.ts` (the “stable
recovery point promotion” block: the recovery pointer advances only, fails
closed on a malformed pointer, and retention skips the prune when the pointer is
unreadable). Manual-restore hardening is unit-tested in `src/persistence.test.ts`
too: a `setRecoveryPoint: false` restore leaves the pointer untouched, an
explicit restore opens the maintenance window and stops the gateway while it
runs, and `restoreIfNeeded` defers while that window is active. The route’s
`?setCurrent=0` mapping is covered by `src/routes/api/storage.test.ts`, the
durable-flag round-trip by `src/activity/backup-jobs.test.ts` /
`src/activity/scheduler.test.ts`, and the Backups-page checkbox by
`src/client/components/__tests__/BackupHistoryPanel.test.tsx`, plus the
Cloudflare `@cloudflare/vitest-plugin` route tests planned in
[`apps/fouria/test/e2e/BACKUP-E2E-DESIGN.md`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/e2e/BACKUP-E2E-DESIGN.md).

Restore legibility (which backup/date is loading) is covered by
`src/persistence.test.ts` (`readRecoveryState`; the restore target is written
during `sandbox.restoreBackup` and cleared on success), `src/assets/loading.test.ts`
(the loading page consumes `data.recovery` / `recovery.restoring` /
`recovery.committed`), and `src/client/components/__tests__/BackupHistoryPanel.test.tsx`
(the recovery-point badge + summary). The local observability canary
`test/e2e/ad_observability_canary.txt` shows how to query the local dev server's
Local Explorer SQL API (`test/e2e/fixture/observability-query`) to confirm the
restore-loop guards (`Restore deferred — tenant maintenance in progress`,
`GatewayMaintenanceError`) fire from the worker's own logs.

> **Vitest 5 caveat.** The Cloudflare route-integration layer is deferred: both
> `@cloudflare/vitest-pool-workers` and the renamed `@cloudflare/vitest-plugin`
> peer `vitest ^4.1.0`, and under Vitest 5 the pool fails with
> `SyntaxError: Unexpected identifier 'file'` (known open issue
> [cloudflare/workers-sdk#15618](https://github.com/cloudflare/workers-sdk/issues/15618);
> workaround: pin `vitest ~4.1.11`).

> **Restore sequencing.** Cold‑restore survival (I4) is **opt‑in** via
> `E2E_BACKUP_RESTORE=1` (`zzz_backup_cold_restore.txt`) because restoring the
> ~250 MB local‑binding archive is far too slow for CI (observed >13 min without
> completing). The default `phaseE_restore` cases (catalog, health‑observability,
> running‑backup, idle‑skip, bucket isolation, encryption, retention) do not
> restore and use `fixture/wait-ready` (one warm request, then `?passive=1` polls)
> so overlapping restores cannot abort each other with
> `OperationInterruptedError: … runtime changed` (root cause:
> [`BACKUP-E2E-DESIGN.md` §9](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/e2e/BACKUP-E2E-DESIGN.md)).

> **cctr corpus gotcha.** cctr 0.31 silently registers **zero tests** for a
> `*.txt` corpus file whose first line is a `#` comment, or when comments sit
> between a file-level `%skip` and the first `===`. Corpus files must begin with
> `===` (or a lone directive followed by `===`); keep prose in test bodies or at
> the end of the file. `cctr -l <file>` prints the parsed test count.

### Business Rules Corpus

Contract tests at `apps/fouria/test/business-rules/` that validate cross-cutting concerns. These form the "T0 validation gate" per issue #137.

```bash
cd apps/fouria
cctr test/business-rules/
```

| Corpus File                  | Validates                                                                                                                                                                                                           |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pricing.txt`                | Per-action billing, success-only, BYOL discounts, VISION.md alignment                                                                                                                                               |
| `rbac.txt`                   | ROOT/ADMIN/BASE_USER permission matrix, route guards, and dashboard + loading-page endpoint coverage                                                                                                                |
| `tenant-isolation.txt`       | Cross-tenant access controls, `canAccessCustomer`                                                                                                                                                                   |
| `operational-modes.txt`      | DEV_MODE, E2E_TEST_MODE, DEBUG_ROUTES, DEMO_MODE, running-only backups, health observability-only, late-ready gateway adoption, health is not a routing gate (ROOT-revivable frozen), reprovision fail-closed guard |
| `cost-tracking.txt`          | 6 Analytics Engine event types, field schemas                                                                                                                                                                       |
| `dashboard-api-contract.txt` | Admin API endpoint existence                                                                                                                                                                                        |
| `agent-visibility.txt`       | BASE_USER agent visibility rules                                                                                                                                                                                    |
| `credits-and-plans.txt`      | Credit model, plan limits, BYOL exclusion, p95 pricing                                                                                                                                                              |

---

## apps/lerma — Phoenix Web App

### Unit Tests

Mirror `lib/` structure under `test/lerma/` and `test/lerma_web/`. Run with ExUnit via mix.

```bash
cd apps/lerma

# All unit tests (creates database, runs migrations)
mix test

# Single file
mix test test/lerma/clients_test.exs

# From the repository root: compile + deps.unlock + format + Credo + tests
cd ../..
mix precommit
```

**Key test files:**

| File                                                          | Tests                                                                                                         |
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `lerma/clients/client_test.exs`                               | Client CRUD, validation, Ecto schema                                                                          |
| `lerma/clients/user_test.exs`                                 | User model, invitation lifecycle fields                                                                       |
| `lerma/clients_test.exs`                                      | Invite/accept/reject, instance-scoped Access sync, reserved staging-client protections                        |
| `lerma/applications_test.exs`                                 | Application CRUD, one-time provisioning key generation/rotation/revocation and digest-only storage            |
| `lerma/staging_tenant_test.exs`                               | Staging-only `client` bootstrap, production isolation, and database delete/rename/truncate guards             |
| `lerma/emails_test.exs`                                       | Confirmation email content and links                                                                          |
| `lerma/invitations_test.exs`                                  | Background invitation email delivery and PubSub alert                                                         |
| `lerma/instances/instance_test.exs`                           | Instance lifecycle                                                                                            |
| `lerma/instances_upgrade_test.exs`                            | Upgrade version guards, per-application filtering, and repeat-promotion idempotency                           |
| `lerma/provisioning/tenant_provisioner_test.exs`              | Tenant provisioning logic                                                                                     |
| `lerma/billing_test.exs`                                      | Billing calculations                                                                                          |
| `lerma/billing_plans_test.exs`                                | Global plan templates, assign/clone, autobuy toggle                                                           |
| `lerma/billing_rules_corpus_test.exs`                         | Billing business rules (a–m) incl. production gating                                                          |
| `lerma/billing_local_workflow_test.exs`                       | Local workflow: AE rows → reconcile → consume; configured/unconfigured/duplicate events, watermark ownership  |
| `lerma/billing_becaria_trace_test.exs`                        | BecarIA flow: AE rows → reconcile → exact per-action credits + sums                                           |
| `lerma/credits/credit_pool_test.exs`                          | Credit pool management                                                                                        |
| `lerma_web/live/billing_plans_live_test.exs`                  | Global plan library + designer                                                                                |
| `lerma_web/components/billing_plan_component_test.exs`        | Client board plan assignment + summary                                                                        |
| `lerma/analytics/cf_analytics_test.exs`                       | Cloudflare Analytics Engine queries                                                                           |
| `lerma/observability/otlp_test.exs`                           | OTLP/HTTP JSON parsing, allowlist privacy guard, child-span drop, caps, stable dedupe keys                    |
| `lerma/observability/telemetry_test.exs`                      | Idempotent destination ingestion, uptime/failure ratios (unknown ≠ downtime), window helpers, retention prune |
| `lerma/cloudflare/observability_test.exs`                     | Worker upload metadata: destination names, per-signal sampling, `persist: false`, disabled switch             |
| `lerma/cloudflare/observability_destinations_test.exs`        | Destination reconcile: create/PATCH staging + production pairs, 403/500 handling, env-specific names          |
| `lerma_web/controllers/api/otlp_ingest_controller_test.exs`   | Destination ingest auth (fail closed), `415` for protobuf, gzip delivery, privacy drop, replay idempotency    |
| `lerma_web/plugs/otlp_body_reader_test.exs`                   | Gzip/JSON body decoding, size caps, malformed-body 400, path/method passthrough                               |
| `lerma_web/controllers/api/observability_controller_test.exs` | Uptime endpoint ratios/status/windows, unknown when no observations, sampling rate surfaced                   |
| `lerma_web/components/worker_telemetry_widget_test.exs`       | Worker telemetry widget unknown/healthy/degraded render                                                       |
| `lerma_web/components/health_events_widget_test.exs`          | Health events timeline empty state, recorded transitions, dev-instance rendering                              |
| `lerma_web/controllers/api/instance_controller_test.exs`      | Instance API endpoints                                                                                        |
| `lerma_web/controllers/api/provisioning_controller_test.exs`  | Global/application-key auth, application-scoped upgrade and status                                            |
| `lerma_web/controllers/api/ai_gateway_controller_test.exs`    | Global provisioning-key auth; fails closed when unset                                                         |
| `lerma_web/controllers/api/credit_usage_controller_test.exs`  | Billing reconcile + credit-usage API (bearer auth)                                                            |
| `lerma_web/live/instance_live_test.exs`                       | Instance LiveView                                                                                             |
| `lerma/client_docs_test.exs`                                  | Client docs sources (proposal + per-env stores), R2 listing/reading, default document formats, path traversal |
| `lerma/client_docs/markdown_test.exs`                         | Safe Markdown preview: frontmatter, headings, tables, lists, code, relative images, unsafe links, HTML escape |
| `lerma_web/controllers/client_docs_controller_test.exs`       | Raw docs responses: content type, inline/download disposition, sandbox CSP, not-found/invalid-path errors     |
| `lerma_web/live/client_live_test.exs`                         | Invitation UI, public acceptance flow, and the lazy documentation widget tree/preview/live                    |
| `lerma_web/live/application_live_test.exs`                    | ROOT-only application key generation and one-time reveal                                                      |
| `lerma_web/plugs/cf_access_auth_test.exs`                     | CF Access auth plug                                                                                           |

**Per-test tags:**

- `@tag cf_auth: %{email: "...", name: "..."}` — injects CF Access authentication
- `@tag :integration` — excluded by default, must run explicitly via `mix test --only integration`

### Staging Fouria build promotion

`.github/workflows/promote-staging.yml` listens for completed `Test` runs on
`develop`, then queries the source run's Actions jobs and proceeds only when
`build-and-push` concluded `success`. The script promotes the immutable
`unstable-<sha>` image/bundle pair, no-ops when the staging `client` tenant is
already running that version, and polls Lerma's status endpoint until the
running application instances converge. The GitHub secret
`STAGING_PROVISIONING_API_KEY` must match the one-time generated key for the
Fouria application in staging Lerma. This cloud workflow is not run by local
`mix test`; the application-scope/auth/idempotency contracts are covered by the
ExUnit tests above.

### E2E Tests (cctr)

Sequential cctr tests against real Cloudflare dispatch namespace infrastructure.

```bash
cd apps/lerma
mix test.e2e
```

**Setup** (`test/e2e/_setup.txt`):

1. Validates `jq`, `uuidgen` tools
2. Sources `.dev.vars` for Cloudflare credentials
3. Validates `CLOUDFLARE_API_TOKEN`, `CF_ACCOUNT_ID`, `WORKERS_SUBDOMAIN`, `CF_ENVIRONMENT`
4. Computes dispatch namespace (`fouria-tenants` vs `fouria-tenants-staging`)
5. Starts Phoenix on port 4002 via `mix phx.server`

**Teardown** (`test/e2e/_teardown.txt`):

1. Reads client/instance IDs from fixture directory
2. Deletes instance (tears down CF worker + container application) via API
3. Deletes client via API — fully decommissions the tenant (remaining
   instances and container applications, Access app/group, dedicated AI
   Gateways, R2 backup buckets) and archives the client record
4. Kills Phoenix process

**Test files:**

| File                      | Tests                                                                                          |
| ------------------------- | ---------------------------------------------------------------------------------------------- |
| `client_provisioning.txt` | Create client → create instance → wait for running → verify gateway → verify CF metadata       |
| `env_isolation.txt`       | Correct namespace, wrong namespace rejection, environment tags, Analytics Engine dataset names |

### Staging Billing-Trace E2E (BecarIA)

`test/e2e_billing_trace/` (opt-in, run separately from `mix test.e2e`) validates
the complete credit-billing trace against **staging**: a deployed Fouria worker
emits the five BecarIA priced actions via `POST /api/otel/v1/traces`, they land
in Cloudflare Analytics Engine (`fourai_cost_metrics_staging`), Lerma reconciles
them (`POST /api/credits/v1/reconcile`), and the exact per-action credits
(1 + 1 + 3 + 5 + 10 = 20) plus the summed total are asserted through
`GET /api/credits/v1/usage/:client_id`.

```bash
cd apps/lerma
export RUN_STAGING_BILLING_E2E=1
export OTEL_SHARED_SECRET=...
export CF_ACCESS_CLIENT_ID=... CF_ACCESS_CLIENT_SECRET=...
mix test.e2e.billing
```

See `apps/lerma/test/e2e_billing_trace/README.md` for prerequisites, the worker
requirements it surfaces, and failure triage. The deterministic core also runs
in CI as `test/lerma/billing_becaria_trace_test.exs` with a stubbed AE stream.

The five actions are **semantic** (`telegram_query`, `queue_reevaluation`,
`signal_analysis`, `email_draft`, `review_analysis`), emitted through the
explicit `openclaw.billing.action_type` marker with span-based inference as the
fallback. The deterministic test also covers trace precedence
(`email_draft`/`review_analysis` suppress `telegram_query`), success-only
billing (`error`/`blocked` consume nothing), `action_key` idempotency across
retried traces, 0-credit fixed-service actions, and the plan price winning over
the row's advisory `credits`.

### Local Billing-Workflow E2E (fake Analytics Engine)

`test/e2e_billing_local/` (opt-in, run separately from `mix test.e2e`) exercises
the billing consume path entirely locally — **no Cloudflare account, no staging
DB, no deployed worker**:

```
fake Analytics Engine (bun)
  └─ POST /accounts/:id/analytics_engine/sql   ← BillingReconcile reads here
       └─ lerma (mix phx.server, MIX_ENV=test)
            └─ POST /api/credits/v1/reconcile → BillingConsumption → GET /usage
```

It uses a **mock billing definition** (`fixture/seed_plan.exs`, not BecarIA) and
feeds synthetic `billable-action` AE rows directly (OpenClaw's OTel emission is
owned/tested by the OpenClaw team). Cases: configured + `match_rule` events,
unconfigured/unmatched/blank events, duplicate `action_id` redelivery, AE outage
(502, no billing), and tenant isolation.

```bash
cd apps/lerma
mix test.e2e.billing:local
```

**Why a fake AE and not Cloudflare Local Explorer?** Local Explorer
([docs](https://developers.cloudflare.com/workers/local-development/local-explorer/))
does not support Analytics Engine — it is absent from the supported-bindings
table and from the Local Explorer OpenAPI, and local traces emit no
`writeDataPoint` span. A fake of the AE SQL API is therefore the only local
observability surface for the billable-event stream. Local Explorer _is_ used on
the Worker side for logs/spans (`apps/fouria/test/e2e/ad_observability_canary.txt`).
See `apps/lerma/test/e2e_billing_local/README.md`.

### Test Infrastructure Configuration

**`apps/lerma/mix.exs`:**

```elixir
test: ["ecto.create --quiet", "ecto.migrate --quiet", "test"]
"test.e2e": ["cmd cctr test/e2e/ --sequential -v"]
"test.e2e.billing": ["cmd cctr test/e2e_billing_trace/ --sequential -v"]
```

**Root `mix.exs`:** The `credo` alias forwards all arguments to each Elixir
workspace project, so `mix credo --strict` works from the repository root without
running Credo against the root workspace shell project.

```elixir
credo: &run_credo/1,
check: [
  "workspace.run -t format -- --check-formatted",
  "credo --strict"
],
precommit: [
  "workspace.run -t compile -- --warnings-as-errors",
  "workspace.run -t deps.unlock -- --unused",
  "workspace.run -t format",
  "credo --strict",
  "license_audit",
  "workspace.run -t test"
]

# Private alias helper:
defp run_credo(args) do
  args = if "--strict" in args, do: args, else: ["--strict" | args]
  Mix.Task.run("workspace.run", ["-t", "credo", "--" | args])
end
```

**`test_helper.exs`:** Excludes `:integration` tests, starts SQL sandbox.

**Mocking:** Uses `Bypass` for HTTP mocking and `Meck` for Elixir mocking. Not available outside `:test` env.

---

## packages/moltlazy — Config Module + SDK

### Unit Tests (Vitest)

```bash
cd packages/moltlazy
bun run test          # vitest run
bun run test:watch    # watch mode
```

**Config module tests** (`tests/config/`):

| File               | Tests                                                               |
| ------------------ | ------------------------------------------------------------------- |
| `gateway.test.ts`  | Gateway config section (port, mode, trustedProxies, allowedOrigins) |
| `session.test.ts`  | Session config (dmScope)                                            |
| `tools.test.ts`    | Tool search configuration                                           |
| `logging.test.ts`  | Logging section (level, format, transports)                         |
| `validate.test.ts` | CLI validation-error formatting (object-shaped `openclaw` errors)   |
| `include.test.ts`  | `$include` directive injection                                      |

**SDK module tests** (`tests/`):

| File                            | Tests                                 |
| ------------------------------- | ------------------------------------- |
| `sdk.test.ts`                   | SDK client initialization             |
| `sdk-cli.test.ts`               | SDK CLI fallback mechanisms           |
| `sdk-gateway-status-ws.test.ts` | Gateway WebSocket status checks       |
| `sdk-ws-client.test.ts`         | WebSocket client connection lifecycle |

**Agent tests** (`tests/`):

| File                    | Tests                                      |
| ----------------------- | ------------------------------------------ |
| `agents.test.ts`        | Agent defaults, list generation            |
| `index.test.ts`         | Main module (patch flow, section ordering) |
| `schema.test.ts`        | Zod schema validation for moltlazy config  |
| `obsidian-sync.test.ts` | Obsidian sync skill configuration          |
| `coverage-gaps.test.ts` | Coverage gap tracking across modules       |

### E2E Tests (cctr)

```bash
cd packages/moltlazy
bun run test:e2e        # Build + cctr
```

| Test Suite                | Scenarios                                                           |
| ------------------------- | ------------------------------------------------------------------- |
| `cli/basic.txt`           | `moltlazy patch` from empty, `--help` displays man-style docs, `-h` |
| `cli/flags.txt`           | `--file` override, `--help`, unknown commands                       |
| `cli/errors.txt`          | Validate pass/fail, nonexistent file, filesystem errors             |
| `full-flow/roundtrip.txt` | Full patch → verify output (gateway, `$include`, agents, session)   |
| `skills/discovery.txt`    | Skills directory discovery, empty skills dir, re-patching           |
| `workspace/seeding.txt`   | SOUL.md creation, fouria-builder workspace, safe mode, immutability |

**Setup:** Validates that `dist/cli.js` exists (pre-build check).

> **OpenClaw CLI version:** the `moltlazy-integration` CI job validates the
> generated config with the OpenClaw CLI **pinned to the version baked into the
> container** (root `package.json` catalog, currently `2026.7.1-2`) instead of
> `openclaw@latest`. This keeps the config-schema contract aligned with what is
> actually deployed and avoids flaky failures from upstream schema drift. Bump
> the pin in `.github/workflows/test.yml`
> and this doc together whenever the container's OpenClaw version changes.

### Smoke Tests (Vitest)

```bash
cd packages/moltlazy
bun run test:all       # Unit + E2E
```

---

## OpenClaw plugins (external)

The `cloudflare-unified-billing` plugin — its unit tests, live AI Gateway
integration tests (`infer-smoke`, `provider-smoke`, `cf-cost-api`), and CI —
moved to [Paso4/openclaw-plugins](https://github.com/Paso4/openclaw-plugins).
Run them from a checkout of that repo:

```bash
cd plugins/provider/cloudflare-unified-billing
bun run test              # unit
bun run test:integration  # live (requires AI Gateway creds + PLUGIN_LIVE_TESTS=true)
```

See [OPENCLAW-PLUGINS.md](/latest/llms/platform/libraries/openclaw-plugins/index.md).

---

## CI Pipeline

The `.github/workflows/test.yml` workflow defines the CI pipeline (see reference table in [AGENTS.md](https://github.com/0xCAB0/fouria/blob/develop/AGENTS.md) for the full job graph):

```
unit ──┬─ build-and-push → e2e   (PR: single shard · develop: sharded matrix)
       └─ moltlazy-integration   (only when packages/moltlazy changed)

changes ── iac-test              (only when apps/*/iac changed)
```

The `changes` job git-diffs the event SHAs and gates `iac-test` and
`moltlazy-integration`, so unrelated changes no longer spin them up. The
`local-integration` job was removed.

**The Changesets version pull request runs no CI.** The "Version Packages" PR is
opened from the `changeset-release/develop` branch by
`.github/workflows/version-packages.yml`; it only bumps versions and writes
changelogs, so every PR-triggered workflow exits immediately for that head
branch — `test.yml` (`changes`, `unit`, `validate-secrets`, `build-and-push`),
`lerma.yml` (`quality`, `test`), `staging-validation-ci.yml` and
`changeset-status.yml`. The guard is the head-ref prefix
`!startsWith(github.head_ref, 'changeset-release/')`, which is empty on
`push`/`workflow_dispatch`, so those paths are unaffected. Skipped jobs report as
**skipped** (not pending) to branch protection, and no CD workflow can be
affected: both `promote-staging.yml` and `deploy-dashboard.yml` only ever act on
a successful run whose `head_branch` is `develop`.

`build-and-push` builds the worker, validates the wrangler multipart bundle via
`scripts/validate-bundle.mjs`, then builds the container image and pushes it to
the Cloudflare registry as `fouria:unstable-<sha>` and uploads the worker
bundle to the `fouria-bundles` R2 bucket as `unstable-<sha>/fouria.mjs`. On
develop, `e2e` consumes that image (via `IMAGE_REF`) instead of rebuilding it.
The `deploy-dispatch.yml` workflow also deploys `workers/fouria-dispatch`
(staging on push to develop, production on a stable CalVer release or a manual
`workflow_dispatch`). `deploy-fouria.yml` is the fouria CD: on a stable CalVer
release (`vYYYY.M.D` / same-day `vYYYY.M.D-N`, classified from the tag name, not
the GitHub `prerelease` flag) it **promotes the CI-built `unstable-<sha>`
artifacts** — retagging that image to `latest` + the release tag and copying the
bundle to R2 under the release tag + `latest`. Nothing is rebuilt. Release
commits are pushed with `[skip ci]`, so they never
re-trigger this CI pipeline.

Production deploys are stable-tag-only: lerma filters the deployable versions to
stable release tags and `Instances.update_instance_version/3` rejects unstable
ones (staging keeps the unstable channel). The `cleanup-stale-images.yml` cron
never deletes an image (or matching R2 bundle) still referenced by a container
application — including in-progress rollout versions — and fails closed if that
in-use set cannot be resolved. Covered by
`apps/lerma/test/lerma/cloudflare/image_registry_test.exs`,
`apps/lerma/test/lerma/cloudflare/registry_bundle_test.exs`,
`apps/lerma/test/lerma/instances_test.exs`, and
`apps/lerma/test/lerma_web/live/instance_live_test.exs`.

Container image upgrades roll out with `strategy: "rolling"` (a single
`step_percentage: 100` step) — see `Lerma.Cloudflare.Containers`. The tenant
container is Durable-Object-managed, and a `new_instances` rollout does **not**
replace a DO-bound instance: the platform stages the target as a rollout version
at 0% and the application keeps its previous image, so `new_instances` silently
left promoted tenants running the old image. Promotion convergence is also
sourced from the **live** container image tag (`Containers.fetch_deployed_config/1`)
in `Instances.at_requested_version?/2`, not the recorded `deployed_version`, so a
promotion cannot report success while the container still runs the previous
image (and a genuine miss can be re-promoted instead of being skipped). Covered
by `apps/lerma/test/lerma/cloudflare/containers_test.exs` and
`apps/lerma/test/lerma/instances_upgrade_test.exs`.

### Plugin Integration (unified billing e2e) — moved

The `cloudflare-unified-billing` e2e tests and their dedicated workflow moved to
[Paso4/openclaw-plugins](https://github.com/Paso4/openclaw-plugins). That repo
runs live AI Gateway integration tests on `develop` only (they cost real
credits), decoupled from this monorepo's `test.yml`.

**Key env vars for CI:**

- `SKIP_CLI_E2E=true` — used in `unit` job to skip CLI-dependent tests
- `E2E_SKIP_BROWSER_TESTS=1` — skip Playwright browser tests
- `RUN_REASONING_TESTS`, `RUN_VISION_TESTS` — gate expensive integration tests

---

## Writing Tests

### TypeScript (Vitest)

- **Colocate:** Place `*.test.ts` next to source files in `apps/fouria/src/` and `packages/moltlazy/tests/`
- **Strict mode:** Tests use `strict: true` TypeScript
- **Mocking:** Use vitest's `vi.mock()` for module-level mocking
- **Lifecycle:** `beforeAll`/`afterAll` for setup/teardown; `beforeEach`/`afterEach` for per-test state
- **Pattern:** Arrange → Act → Assert (AAA)
- **Coverage:** ≥80% for new code (≥95% for API routes in `apps/fouria`)

### Elixir (ExUnit)

- **Mirror structure:** Test files mirror `lib/` paths under `test/`
- **Database:** Use `Ecto.Adapters.SQL.Sandbox` for concurrent DB tests
- **HTTP mocking:** Use `Bypass` for external HTTP requests
- **Auth helpers:** Use `@tag cf_auth` in `conn_case.ex` to inject CF Access headers
- **Integration tests:** Tag with `@tag :integration` (excluded by default)
- **Fixtures:** Define in `test/support/fixtures/`
- **Convention:** `describe "function_name/arity"` blocks; test names as strings

### cctr (Corpus Contract Tests)

- **Format:** Plain text test files with `===` (description) → script → `---` → assertion sections
- **%require:** First test case in a file with `%require` cascades failures to skip remaining tests
- **Variables:** Capture dynamic output with `{{ name }}` — used for tokens, URLs, etc.
- **Assertions:** Use `where` clauses (`* s contains "ready"`, `* status == "200"`)
- **Fixtures:** The `fixture/` directory is mirrored to `$CCTR_FIXTURE_DIR`
- **Setup/teardown:** `_setup.txt` runs before the suite, `_teardown.txt` after
- **Shared helpers:** `apps/fouria/test/lib/e2e-helpers.sh` provides
  `e2e_wait_for_health` (immediate, bounded readiness probe) and
  `e2e_classify_ws_upgrade` (WS-upgrade verdict), unit-tested in
  `apps/fouria/src/e2e-helpers.test.ts`. `e2e_wait_for_health` is called by the
  serial (`test/e2e`) and agent (`test/e2e-agent`) `_setup.txt` corpora and by
  the parallel driver's top-level wait (`test/e2e-parallel/run-parallel.sh`).
  `ab_gateway_ws_origin.txt` sources the library only for
  `e2e_classify_ws_upgrade` and keeps its own health poll; the parallel driver's
  generated group `_setup.txt` also keeps its own grace-free bounded health poll.
- **Local mode guard:** Tests that require cloud infrastructure use:
  ```bash
  if [ "${E2E_LOCAL:-}" = "true" ]; then
      echo "SKIP: ..." >&2
      exit 1
  fi
  ```

### Business Rules

- **Location:** `apps/fouria/test/business-rules/`
- **Format:** cctr `.txt` files
- **Purpose:** Cross-cutting behavioral contracts (pricing, RBAC, tenant isolation, etc.)
- **When to update:** Any change affecting pricing, RBAC, tenant isolation, operational modes, cost tracking, dashboard API, agent visibility, or credits/plans
- **Run:** `cctr apps/fouria/test/business-rules/`

---

## Validation Checklist

Before submitting changes that modify test infrastructure or add new tests:

- [ ] `bun run typecheck` passes
- [ ] `bun run lint` passes
- [ ] `bun run test` passes (all JS/TS packages)
- [ ] For `apps/fouria`: `bun run test:coverage` passes
- [ ] For `apps/lerma`: `mix test` passes
- [ ] New code has ≥80% unit test coverage (≥95% for API routes)
- [ ] If adding routes: add tests
- [ ] If changing env vars: update `.dev.vars.example`, [ENV-VARIABLES.md](/latest/llms/platform/reference/env-variables/index.md), and `apps/fouria/src/gateway/env.ts`
- [ ] If changing business rules: update corresponding corpus file and run `cctr apps/fouria/test/business-rules/`
- [ ] For `apps/fouria` local E2E: `bun run test:e2e:local:cli` passes (cloud-only tests self-skip)
