Testing Guide

On this page

Overview
Quick Start
apps/fouria — Cloudflare Worker
- Unit Tests
- E2E Tests (Cloud)
- E2E Tests (Local)
- E2E Agent Subset (OpenClaw migration smoke)
- Gateway Boot Smoke Test (local)
- Business Rules Corpus
apps/lerma — Phoenix Web App
- Unit Tests
- E2E Tests (cctr)
- Test Infrastructure Configuration
packages/moltlazy — Config Module + SDK
- Unit Tests (Vitest)
- E2E Tests (cctr)
- Smoke Tests (Vitest)
packages/plugins/cloudflare-unified-billing — OpenClaw Plugin
- Unit Tests
- Integration Tests
CI Pipeline
- Plugin Integration (unified billing e2e) — independent CI
Writing Tests
- TypeScript (Vitest)
- Elixir (ExUnit)
- cctr (Corpus Contract Tests)
- Business Rules
Validation Checklist

Testing infrastructure reference for all FourIA products. See CONTRIBUTING.md for contribution workflow and DEVELOPMENT.md for onboarding.

Overview

Product	Type	Framework	Unit Tests	E2E Tests
`apps/fouria`	Cloudflare Worker	Vitest + cctr + Playwright	Colocated `*.test.ts`	Cloud + Local + Business Rules
`apps/lerma`	Phoenix Web App	ExUnit + cctr	Mirror `lib/` structure	Sequential cctr scenarios
`packages/moltlazy`	JS Config + SDK	Vitest + cctr	`tests/*.test.ts`	cctr CLI scenarios
`packages/plugins/cloudflare-unified-billing`	OpenClaw Plugin	Vitest	Unit on `src/`	Integration (requires AI Gateway)

Quick Start

# All JS/TS tests (fouria + moltlazy + plugin)
bun run test

# TypeScript type checking (all packages)
bun run typecheck

# Lint all TS/JS sources
bun run lint

# Full CI suite (test + typecheck + local integration + moltlazy E2E)
bun run test:all

See each product section below for product-specific commands.

apps/fouria — Cloudflare Worker

Unit Tests

Colocated .test.ts files alongside source files. Run with vitest.

cd apps/fouria
bun run test                  # All unit tests
bun run test:watch            # Watch mode
bun run test:coverage         # With Istanbul + V8 coverage

Key test files (see WORKER-DEVELOPMENT.md for full list):

File	Tests
`auth/jwt.test.ts`	JWT decoding and validation
`auth/jwks.test.ts`	JWKS fetching and caching
`auth/middleware.test.ts`	Auth middleware, DEV_MODE, E2E_TEST_MODE
`gateway/env.test.ts`	Container environment variable building
`gateway/process.test.ts`	Process finding and lifecycle
`gateway/r2.test.ts`	R2 mounting logic
`auth/rbac.test.ts`	RBAC permission matrix
`metrics.test.ts`	Cost/usage Analytics Engine instrumentation

E2E Tests (Cloud)

Deploys a real worker to *.workers.dev via Terraform + wrangler deploy. Requires Cloudflare credentials.

cd apps/fouria

# Full E2E with browser
bun run test:e2e

# CLI-only (no browser)
bun run test:e2e:cli

Setup (test/e2e/_setup.txt):

Provisions cloud infrastructure via Terraform (R2 bucket) — the content-hash worker cache is temporarily disabled (issue #297), so every run provisions fresh
Deploys worker via wrangler deploy --var E2E_TEST_MODE:true
Starts Playwright browser via plwr
Polls /api/health up to 5 minutes for cold start
Opens browser to worker URL, waits for “Device pairing required”

E2E cost safety (issue #297): E2E containers deploy with SANDBOX_SLEEP_AFTER=5m, so a container that outlives its run (failed teardown, workflow timeout) stops billing within ~5 minutes instead of running 24/7. Worker caching is disabled until the feature is reworked to reuse only within working hours and to delete on eviction.

Teardown (test/e2e/_teardown.txt):

Dumps gateway logs + doctor output
Stops browser (saves video)
Destroys cloud resources (worker, container app, R2 bucket, Terraform)

Test files (alphabetical order — naming determines execution order):

File	What it tests
`b_cli_devices.txt`	`openclaw devices list` through `/debug/cli`
`c_cli_models.txt`	`openclaw models list`, model catalogs
`d_cli_channels.txt`	Channel probe, config inspection
`e_cli_config.txt`	Config validate, set, unset via CLI
`g_cli_secrets.txt`	Secrets audit, reload via CLI
`h_cli_knowledge.txt`	Wiki init, status, get via CLI
`i_cli_cron.txt`	Cron list, add dream job
`j_cli_integrations.txt`	Integration config verification
`k_secrets_store.txt`	Secrets Store (gateway token, CDP, container config, device pairing)
`l_cli_billing.txt`	Pricing constants, costs, instance types, billable events
`m_cli_agent_tokens.txt`	Agent turns, token usage metrics
`provision_and_dispatch.txt`	Cloud-only — dispatch worker + Supabase tenant lifecycle
`a_rbac_admin_isolation.txt`	ADMIN role isolation (management API, cross-tenant)
`a_rbac_api_matrix.txt`	Full ROOT/ADMIN/BASE_USER permission matrix across 40+ endpoints
`a_rbac_base_user_isolation.txt`	BASE_USER isolation (whoami, blocked from management/secrets/writes)
`a_rbac_device_pairing.txt`	Device pairing permissions for all 3 roles (with retry on timeout)
`a_rbac_root_isolation.txt`	ROOT isolation (diagnostics access, blocked from management/agents/secrets)
`a_rbac_ui_pages.txt`	UI page access for all 3 roles (22 test cases)
`aa_health_gate.txt`	Gateway health check between RBAC and CLI test groups
`z_cli_staging.txt`	API staging tests (costs, storage, users)
`zzz_cron_wake.txt`	Cloud-only — container destroy, cron wake cycle

Runs tests against wrangler dev (local mode) with containers via Docker. Avoids 5-10 minute Terraform provisioning. 95 of 97 tests pass in local mode (the 2 skipped tests require dispatch worker + Supabase or container destroy/recreate).

cd apps/fouria

# Local E2E with Docker + cctr (auto-builds Docker image)
bun run test:e2e:local

# CLI-only (no browser)
bun run test:e2e:local:cli

Setup (fixture/server/start-local):

Sources test/e2e/.dev.vars
Finds unused port (8787-8797)
Runs bun run build:docker (builds container image)
Starts wrangler dev --port <PORT> --env-file test/e2e/.dev.vars
Polls /api/health up to 120 seconds
Writes fixture files (dummy CF Access creds — ignored via DEV_MODE=true)

Teardown (fixture/server/stop-local):

Kills wrangler dev process
Cleans up fixture files

Prerequisites:

# test/e2e/.dev.vars must contain:
DEV_MODE=true
E2E_TEST_MODE=true
DEBUG_ROUTES=true
MOLTBOT_GATEWAY_TOKEN=<value>
CF_ACCOUNT_ID=<id>
CLOUDFLARE_AI_GATEWAY_API_KEY=<key>   # plus CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID
# The REST transport uses CLOUDFLARE_API_TOKEN, derived from the same AI-scoped key.

# Docker must be running
# wrangler must be authenticated via OAuth (wrangler login)

Cloud-only tests (self-skip when E2E_LOCAL=true):

File	Reason
`provision_and_dispatch.txt`	Requires dispatch worker + Supabase
`zzz_cron_wake.txt`	Requires container destroy/recreate

E2E Agent Subset (OpenClaw migration smoke)

apps/fouria/test/e2e-agent/ is a focused, browser-free subset of the CLI surface used to validate an OpenClaw version migration. It reuses the shared test/e2e/fixture (via symlink) and the same local-mode runner, but only runs the commands the Worker shells out to:

b_cli_devices.txt — openclaw devices
c_cli_models.txt — openclaw models
d_cli_channels.txt — openclaw channels
e_cli_config.txt — openclaw config validate
g_cli_secrets.txt — openclaw secrets
h_cli_knowledge.txt — openclaw wiki
m_cli_agent_tokens.txt — openclaw agents / token flows
n_gateway_soft_reload.txt — gateway soft reload (SIGUSR1)

cd apps/fouria
bun run test:e2e:agent   # build:docker + E2E_LOCAL=true + browser skipped

The openclaw-migration skill runs this subset after bumping catalog.openclaw to confirm the CLI commands still behave.

Gateway Boot Smoke Test (local)

Validates that the fouria sandbox image actually boots a healthy OpenClaw gateway on the local machine — no Worker, no cloud e2e, no third-party machines. It builds the exact production image (apps/fouria/Dockerfile + repo-root context, same base cloudflare/sandbox image wrangler uses) and runs the real boot path (start-openclaw.sh → onboard → plugin installs → moltlazy patch/validate → gateway launch → health poll) inside Docker. The container’s exit code is the verdict: the in-container wrapper runs start-openclaw.sh (whose own readiness loop already fails if the gateway dies or never binds), then asserts /health returns {ok: true} while the gateway is still alive.

cd apps/fouria

# Default cold-boot smoke (no real credentials needed):
#   onboard runs --auth-choice skip, tunnel + OTEL steps self-skip.
bun run test:gateway:boot

# Same, plus the real cloudflare-ai-gateway onboard path and an assertion
# that the provider plugin is installed + enabled. Sources credentials from
# .dev.vars.e2e (never echoed).
bun run test:gateway:boot:ai

# Leave the container running after the assertion, with port 18789
# published to 127.0.0.1 for manual inspection (Control UI / WS).
bun run test:gateway:boot --keep

# Skip build:docker + docker build and validate an existing image.
bash scripts/gateway-boot-smoke.sh --no-build --image <tag>

Prerequisites: Docker running, network access (ClawHub/npm installs during boot are bounded by start-openclaw.sh timeouts and non-fatal for boot).

What it validates vs. what it does not:

Validated locally	Still needs local e2e / cloud e2e
Image builds from the production Dockerfile	Sandbox SDK supervisor + Worker RPC layer
onboard (skip or real AI Gateway) completes	RBAC / API routes / device pairing
Pinned plugin installs + allowlist enablement	Dispatch worker routing
moltlazy `patch` + `validate` (config contract)	Cost observability / R2 backup flows
gateway binds `:18789`, `/health` returns ok	Browser-driven Control UI flows

Business Rules Corpus

Contract tests at apps/fouria/test/business-rules/ that validate cross-cutting concerns. These form the “T0 validation gate” per issue #137.

cd apps/fouria
cctr test/business-rules/

Corpus File	Validates
`pricing.txt`	Per-action billing, success-only, BYOL discounts, VISION.md alignment
`rbac.txt`	ROOT/ADMIN/BASE_USER permission matrix, route guards
`tenant-isolation.txt`	Cross-tenant access controls, `canAccessCustomer`
`operational-modes.txt`	DEV_MODE, E2E_TEST_MODE, DEBUG_ROUTES, DEMO_MODE
`cost-tracking.txt`	6 Analytics Engine event types, field schemas
`dashboard-api-contract.txt`	Admin API endpoint existence
`agent-visibility.txt`	BASE_USER agent visibility rules
`credits-and-plans.txt`	Credit model, plan limits, BYOL exclusion, p95 pricing

apps/lerma — Phoenix Web App

Unit Tests

Mirror lib/ structure under test/lerma/ and test/lerma_web/. Run with ExUnit via mix.

cd apps/lerma

# All unit tests (creates database, runs migrations)
mix test

# Single file
mix test test/lerma/clients_test.exs

# From the repository root: compile + deps.unlock + format + Credo + tests
cd ../..
mix precommit

Key test files:

File	Tests
`lerma/clients/client_test.exs`	Client CRUD, validation, Ecto schema
`lerma/clients/user_test.exs`	User model, invitation lifecycle fields
`lerma/clients_test.exs`	Invite/accept/reject and instance-scoped Access sync
`lerma/emails_test.exs`	Confirmation email content and links
`lerma/invitations_test.exs`	Background invitation email delivery and PubSub alert
`lerma/instances/instance_test.exs`	Instance lifecycle
`lerma/provisioning/tenant_provisioner_test.exs`	Tenant provisioning logic
`lerma/billing_test.exs`	Billing calculations
`lerma/credits/credit_pool_test.exs`	Credit pool management
`lerma/analytics/cf_analytics_test.exs`	Cloudflare Analytics Engine queries
`lerma_web/controllers/api/instance_controller_test.exs`	Instance API endpoints
`lerma_web/live/instance_live_test.exs`	Instance LiveView
`lerma_web/live/client_live_test.exs`	Invitation UI and public acceptance flow
`lerma_web/plugs/cf_access_auth_test.exs`	CF Access auth plug

Per-test tags:

@tag cf_auth: %{email: "...", name: "..."} — injects CF Access authentication
@tag :integration — excluded by default, must run explicitly via mix test --only integration

E2E Tests (cctr)

Sequential cctr tests against real Cloudflare dispatch namespace infrastructure.

cd apps/lerma
mix test.e2e

Setup (test/e2e/_setup.txt):

Validates jq, uuidgen tools
Sources .dev.vars for Cloudflare credentials
Validates CLOUDFLARE_API_TOKEN, CF_ACCOUNT_ID, WORKERS_SUBDOMAIN, CF_ENVIRONMENT
Computes dispatch namespace (fouria-tenants vs fouria-tenants-staging)
Starts Phoenix on port 4002 via mix phx.server

Teardown (test/e2e/_teardown.txt):

Reads client/instance IDs from fixture directory
Deletes instance (tears down CF worker + container application) via API
Deletes client via API — fully decommissions the tenant (remaining instances and container applications, Access app/group, dedicated AI Gateways, R2 backup buckets) and archives the client record
Kills Phoenix process

Test files:

File	Tests
`client_provisioning.txt`	Create client → create instance → wait for running → verify gateway → verify CF metadata
`env_isolation.txt`	Correct namespace, wrong namespace rejection, environment tags, Analytics Engine dataset names

Test Infrastructure Configuration

apps/lerma/mix.exs:

test: ["ecto.create --quiet", "ecto.migrate --quiet", "test"]
"test.e2e": ["cmd cctr test/e2e/ --sequential -v"]

Root mix.exs: The credo alias forwards all arguments to each Elixir workspace project, so mix credo --strict works from the repository root without running Credo against the root workspace shell project.

credo: &run_credo/1,
check: [
  "workspace.run -t format -- --check-formatted",
  "credo --strict"
],
precommit: [
  "workspace.run -t compile -- --warnings-as-errors",
  "workspace.run -t deps.unlock -- --unused",
  "workspace.run -t format",
  "credo --strict",
  "license_audit",
  "workspace.run -t test"
]

# Private alias helper:
defp run_credo(args) do
  args = if "--strict" in args, do: args, else: ["--strict" | args]
  Mix.Task.run("workspace.run", ["-t", "credo", "--" | args])
end

test_helper.exs: Excludes :integration tests, starts SQL sandbox.

Mocking: Uses Bypass for HTTP mocking and Meck for Elixir mocking. Not available outside :test env.

packages/moltlazy — Config Module + SDK

Unit Tests (Vitest)

cd packages/moltlazy
bun run test          # vitest run
bun run test:watch    # watch mode

Config module tests (tests/config/):

File	Tests
`gateway.test.ts`	Gateway config section (port, mode, trustedProxies, allowedOrigins)
`session.test.ts`	Session config (dmScope)
`tools.test.ts`	Tool search configuration
`logging.test.ts`	Logging section (level, format, transports)
`validate.test.ts`	CLI validation-error formatting (object-shaped `openclaw` errors)
`include.test.ts`	`$include` directive injection

SDK module tests (tests/):

File	Tests
`sdk.test.ts`	SDK client initialization
`sdk-cli.test.ts`	SDK CLI fallback mechanisms
`sdk-gateway-status-ws.test.ts`	Gateway WebSocket status checks
`sdk-ws-client.test.ts`	WebSocket client connection lifecycle

Agent tests (tests/):

File	Tests
`agents.test.ts`	Agent defaults, list generation
`index.test.ts`	Main module (patch flow, section ordering)
`schema.test.ts`	Zod schema validation for moltlazy config
`obsidian-sync.test.ts`	Obsidian sync skill configuration
`coverage-gaps.test.ts`	Coverage gap tracking across modules

E2E Tests (cctr)

cd packages/moltlazy
bun run test:e2e        # Build + cctr

Test Suite	Scenarios
`cli/basic.txt`	`moltlazy patch` from empty, `--help` displays man-style docs, `-h`
`cli/flags.txt`	`--file` override, `--help`, unknown commands
`cli/errors.txt`	Validate pass/fail, nonexistent file, filesystem errors
`full-flow/roundtrip.txt`	Full patch → verify output (gateway, `$include`, agents, session)
`skills/discovery.txt`	Skills directory discovery, empty skills dir, re-patching
`workspace/seeding.txt`	SOUL.md creation, fouria-builder workspace, safe mode, immutability

Setup: Validates that dist/cli.js exists (pre-build check).

OpenClaw CLI version: the moltlazy-integration CI job validates the generated config with the OpenClaw CLI pinned to the version baked into the container (root package.json catalog, currently 2026.7.1-2) instead of openclaw@latest. This keeps the config-schema contract aligned with what is actually deployed and avoids flaky failures from upstream schema drift. Bump the pin in .github/workflows/test.yml, .github/workflows/plugin-integration.yml, and this doc together whenever the container’s OpenClaw version changes.

Smoke Tests (Vitest)

cd packages/moltlazy
bun run test:all       # Unit + E2E

packages/plugins/cloudflare-unified-billing — OpenClaw Plugin

Unit Tests

cd packages/plugins/cloudflare-unified-billing
bun run test           # vitest on src/ only
bun run test:coverage  # With Istanbul

Tests all exported modules: plugin initialization, model catalog, onboarding flow, streaming.

Integration Tests

Requires a real Cloudflare AI Gateway instance. Tests validate the full auth → baseUrl → CF AI Gateway REST API (/ai/v1) → provider pipeline.

cd packages/plugins/cloudflare-unified-billing
bun run test:integration

File	Tests
`infer-smoke.test.ts`	Model inference (3 Gemini smoke models, reasoning, vision, dynamic model resolution, provider contract validation, model catalog)
`provider-smoke.test.ts`	Per-provider connectivity (google-ai-studio, openai, grok)
`cf-cost-api.test.ts`	AI Gateway Logs API, GraphQL Analytics (workersInvocationsAdaptive, aiGatewayRequestsAdaptive)

Skip gates: RUN_REASONING_TESTS, RUN_VISION_TESTS env vars. Credential checks via canRunTests().

Integration test files run sequentially (fileParallelism: false in vitest.config.ts) because they share one OpenClaw config, one gateway on :18789, and one installed plugin; parallel execution races the onboard/install step against the gateway start and model calls.

cf-cost-api.test.ts reads CLOUDFLARE_API_TOKEN from the plugin .dev.vars, which secrets:generate derives from the AI-scoped CLOUDFLARE_AI_GATEWAY_API_KEY. Its account-read check is skipped for that AI-scoped credential (no Account:Read); supply an account-scoped token to exercise it. The suite is skipped entirely in CI (the workflow passes no CLOUDFLARE_API_TOKEN).

CI Pipeline

The .github/workflows/test.yml workflow defines the CI pipeline (see reference table in AGENTS.md for the full job graph):

unit → build-and-push → e2e
  │                      │
  └──── local-integration ├── (PR artifacts)
  │                      └── (video + logs)
  └──── iac-test (parallel)

build-and-push builds the worker, validates the wrangler multipart bundle via scripts/validate-bundle.mjs, then builds the container image and pushes it to the Cloudflare registry as fouria:unstable-<sha> and uploads the worker bundle to the fouria-bundles R2 bucket as unstable-<sha>/fouria.mjs. On develop, e2e consumes that image (via IMAGE_REF) instead of rebuilding it. The deploy-dispatch.yml workflow also deploys workers/fouria-dispatch (staging on push to develop, production on a stable release or a manual workflow_dispatch). deploy-fouria.yml is the fouria CD: on a stable release it promotes the image to latest and uploads the bundle to R2 under the release tag + latest. Release commits are pushed with [skip ci], so they never re-trigger this CI pipeline.

Production deploys are stable-tag-only: lerma filters the deployable versions to stable release tags and Instances.update_instance_version/3 rejects unstable ones (staging keeps the unstable channel). The cleanup-stale-images.yml cron never deletes an image (or matching R2 bundle) still referenced by a container application — including in-progress rollout versions — and fails closed if that in-use set cannot be resolved. Covered by apps/lerma/test/lerma/cloudflare/image_registry_test.exs, apps/lerma/test/lerma/cloudflare/registry_bundle_test.exs, apps/lerma/test/lerma/instances_test.exs, and apps/lerma/test/lerma_web/live/instance_live_test.exs.

Plugin Integration (unified billing e2e) — independent CI

The cloudflare-unified-billing e2e tests run in their own independent workflow, .github/workflows/plugin-integration.yml, decoupled from the push/PR pipeline. It is scheduled asynchronously every Monday at 08:00 UTC (cron: '0 8 * * 1') and can also be triggered manually via workflow_dispatch. It validates secrets (ai_gateway_valid gate) and then runs bun run test:plugin:integration against the real Cloudflare AI Gateway. Because these tests cost real AI credits, they are intentionally not part of the per-commit test.yml pipeline.

Key env vars for CI:

SKIP_CLI_E2E=true — used in unit job to skip CLI-dependent tests
E2E_SKIP_BROWSER_TESTS=1 — skip Playwright browser tests
RUN_REASONING_TESTS, RUN_VISION_TESTS — gate expensive integration tests

Writing Tests

TypeScript (Vitest)

Colocate: Place *.test.ts next to source files in apps/fouria/src/ and packages/moltlazy/tests/
Strict mode: Tests use strict: true TypeScript
Mocking: Use vitest’s vi.mock() for module-level mocking
Lifecycle: beforeAll/afterAll for setup/teardown; beforeEach/afterEach for per-test state
Pattern: Arrange → Act → Assert (AAA)
Coverage: ≥80% for new code (≥95% for API routes in apps/fouria)

Elixir (ExUnit)

Mirror structure: Test files mirror lib/ paths under test/
Database: Use Ecto.Adapters.SQL.Sandbox for concurrent DB tests
HTTP mocking: Use Bypass for external HTTP requests
Auth helpers: Use @tag cf_auth in conn_case.ex to inject CF Access headers
Integration tests: Tag with @tag :integration (excluded by default)
Fixtures: Define in test/support/fixtures/
Convention: describe "function_name/arity" blocks; test names as strings

cctr (Corpus Contract Tests)

Format: Plain text test files with === (description) → script → --- → assertion sections
%require: First test case in a file with %require cascades failures to skip remaining tests
Variables: Capture dynamic output with {{ name }} — used for tokens, URLs, etc.
Assertions: Use where clauses (* s contains "ready", * status == "200")
Fixtures: The fixture/ directory is mirrored to $CCTR_FIXTURE_DIR
Setup/teardown: _setup.txt runs before the suite, _teardown.txt after

Local mode guard: Tests that require cloud infrastructure use:

if [ "${E2E_LOCAL:-}" = "true" ]; then
    echo "SKIP: ..." >&2
    exit 1
fi

Business Rules

Location: apps/fouria/test/business-rules/
Format: cctr .txt files
Purpose: Cross-cutting behavioral contracts (pricing, RBAC, tenant isolation, etc.)
When to update: Any change affecting pricing, RBAC, tenant isolation, operational modes, cost tracking, dashboard API, agent visibility, or credits/plans
Run: cctr apps/fouria/test/business-rules/

Validation Checklist

Before submitting changes that modify test infrastructure or add new tests:

bun run typecheck passes
bun run lint passes
bun run test passes (all JS/TS packages)
For apps/fouria: bun run test:coverage passes
For apps/lerma: mix test passes
New code has ≥80% unit test coverage (≥95% for API routes)
If adding routes: add tests
If changing env vars: update .dev.vars.example, ENV-VARIABLES.md, and apps/fouria/src/gateway/env.ts
If changing business rules: update corresponding corpus file and run cctr apps/fouria/test/business-rules/
For apps/fouria local E2E: bun run test:e2e:local:cli passes (95/97, 2 cloud-only skips expected)