Testing Guide
On this page
Testing infrastructure reference for all FourIA products. See CONTRIBUTING.md for contribution workflow and DEVELOPMENT.md for onboarding.
Overview
| Product | Type | Framework | Unit Tests | E2E Tests |
|---|---|---|---|---|
apps/fouria | Cloudflare Worker | Vitest + cctr + Playwright | Colocated *.test.ts | Cloud + Local + Business Rules |
apps/lerma | Phoenix Web App | ExUnit + cctr | Mirror lib/ structure | Sequential cctr scenarios |
packages/moltlazy | JS Config + SDK | Vitest + cctr | tests/*.test.ts | cctr CLI scenarios |
The
cloudflare-unified-billingplugin’s unit and live-integration tests now live in Paso4/openclaw-plugins.
Quick Start
# All JS/TS tests (fouria + moltlazy)
bun run test
# fouria admin SPA UI tests (Playwright, local, no credentials)
bun run --cwd apps/fouria test:ui
# TypeScript type checking (Worker, moltlazy, cost-calculator, dispatch, tail)
bun run typecheck
# Lint all TS/JS sources
bun run lint
# Full CI suite (test + typecheck + local integration + moltlazy E2E)
bun run test:all
See each product section below for product-specific commands.
Release tooling tests
scripts/changeset-changelog.test.ts, scripts/release-lib.test.ts,
scripts/sync-versions.test.ts, and scripts/version-packages.test.ts cover
the Changesets changelog format, the release-notes renderer, CalVer changelog
generation, the Lerma package.json → mix.exs version sync, and the
no-changesets no-op path. They run as part of bun run test.
Sandbox v1 migration plan
The v1-sandbox track carries a machine-readable evaluation plan at
docs/plans/sandbox-v1-evaluation-plan.json
(human summary: sandbox-v1-evaluation-plan.md).
It maps every implemented V1 capability to a domain, a milestone and a v1
disposition, and encodes the five-step protocol each milestone executes.
bun run plan:sandbox-v1 # regenerate from docs/releases/V1-RELEASE-NOTES.md
bun run plan:sandbox-v1:check # stale-plan + structural + coverage gate
bun run plan:sandbox-v1:sync # merge origin/develop into v1-sandbox
The plan’s e2e_suitability block is the source of truth for which existing
suites apply to the v1 runtime and which new apps/fouria/test/e2e-v1/**
scenarios a milestone must add. Staging validation targets the Lerma client
client-sandbox-v1 (id 20), never the reserved client slug or production.
Sandbox v1 live drills
apps/fouria/test/e2e-v1/ is a corpus of live drills against a deployed
tenant (default skip unless E2E_V1_LIVE=1). Each suite drives the shared
fixture/v1-drill (one JSON object per mode) and self-skips when the live
environment is absent, so collection is always safe:
| Suite | Covers |
|---|---|
00_runtime_selection.txt | Served lane + lifecycle both report v1 |
01_fresh_boot_and_migration.txt | Real readiness after a boot and a committed recovery point (C8) |
02_concurrency_and_serving_owner.txt | A probe burst folds into exactly one gateway owner |
03_passive_health_and_wake.txt | Non-waking passive probe, idle stop → automation wake, keep-alive policy |
04_channels_and_rpc_ws.txt | Agent/model/channel CLI over the v1 process shim |
05_freeze_and_control.txt | Freeze snapshots then destroys; a probe wakes; unfreeze is a no-op |
06_backup_restore_retention.txt | A record taken before a destroy is the record restored |
07_canary_gates.txt | The served bundle identity matches the promoted build; not degraded |
05/06 destroy the container and require E2E_V1_DESTRUCTIVE=1. Run it with
bun run --cwd apps/fouria test:e2e:v1:live; see
apps/fouria/test/e2e-v1/README.md
for prerequisites, per-suite timing knobs and the temporary billable procedure.
Use a dev instance: Lerma’s HealthMonitor polls production clients every
2 minutes and a passive probe is still a Durable Object request that prevents
the v1 inactivity stop.
The business-rules corpus also covers the v1 admin/billing/cost surface without a
live tenant: apps/fouria/test/business-rules/sandbox-v1-{agents,billing,cost}.txt
pin the runtime-independent route suites against the v1 transport tests
(process shim, DO entrypoint, single owner), the OTLP/billable emit path, the
freeze semantics and the cost-summary query. src/sandbox-v1-outbound.test.ts
pins the v1 outbound host set and the “a host never throws” contract (an
escaping exception is an opaque ECONNRESET in the container).
The gateway proxy attribution fallback is unit-tested without a live tenant:
apps/fouria/src/gateway/proxy-attribution.test.ts covers the proxy-shaped
header set and the 403 + proxy_attribution_required payload classifier, and
the gateway proxy attribution block in src/sandbox-v1.test.ts covers the
transport (direct fallback once the probe is refused, cached verdict, the
forwarded headers kept when attribution is accepted or the probe cannot reach a
booting gateway, no probe for a request without forwarded headers, and the
re-probe after v1Destroy).
Sandbox v1 adversarial / fuzz suites
Two seeded suites attack the Sandbox v1 critical pipelines (zero data loss,
secrets, memory, boot). apps/fouria/src/sandbox-migration/fuzz.test.ts covers
the pure state machines (tri-state reads, validators, the C8 recovery-point
barrier, retention, the migration lease, runtime-marker authority and the
runtime intent gate); apps/fouria/src/sandbox-v1-adversarial.test.ts covers the
SandboxV1 Durable Object (encryption fail-closed matrix, secret non-leakage,
failed-snapshot recovery-point safety, post-commit retention, cadence clock
skew, boot refusals, the process-map bound, the transport port guard and
randomized catalog integrity). A third, deterministic suite —
apps/fouria/src/sandbox-v1-critical-adversarial.test.ts — attacks the three
workflows that must not be a security or consistency risk at merge time:
billing (one OTLP identity authority for metrics/traces and the cost summary,
cross-tenant claims, freeze isolation, replay dedupe, Analytics Engine SQL
injection), client isolation (sentinel/affix resolution, the fixed
sandbox-id fallback on the global leads bucket, lookalike-slug prefix
disjointness) and data consistency (record-id percent-encoding, retention
never pruning the committed point, traversal rejection). All three run as part of
bun run test. The attack matrix, the defects the runs found, and the remaining
observations are in
docs/plans/sandbox-v1-adversarial-fuzz.md.
Docs channel tests
webs/fouria-docs/scripts/channel.test.ts covers the versioned docs plumbing:
channel selection/validation, base-path normalization, channel-relative link
prefixing, and the versions.json manifest (latest/staging paths). It runs
as part of the root bun run test (vitest.config.ts includes
webs/fouria-docs/scripts/**/*.test.ts).
apps/fouria — Cloudflare Worker
Unit Tests
Colocated .test.ts files alongside source files. Run with vitest.
cd apps/fouria
bun run test # All unit tests
bun run test:watch # Watch mode
bun run test:coverage # With Istanbul coverage (alias of coverage)
bun run test:client # SPA React component/page tests (jsdom)
The Worker unit project (
vitest.config.ts, Node pool) excludessrc/client/**. The SPA component and page tests run under the dedicated jsdom project (vitest.client.config.ts) viabun run test:client— they need the React plugin and a DOM, and declare/** @vitest-environment jsdom */per file.
Authentication, backup, and billing regression coverage
src/auth/middleware.fallbacks.test.tsdrives the Hono authentication and role-resolution chain: malformed forwarded JWTs, E2E cookies, local-role fallback, and Dispatch plan metadata.src/auth/routing.test.tscovers invalid tiers, input coercion, and hostname edges.src/activity/scheduler.test.tscovers durable boot success/retry, snapshot interval and idle guards, deprovision checkpoint-before-stop, restore jobs, maintenance jobs, and automation wake lead windows.src/backup-watch/durable-object.test.tscovers enqueue failures without clearing dirty state.src/routes/api/storage.errors.test.tscovers backup/restore conflicts, error responses, background failures, and job/config read failures. The persistence and encryption suites remain the owners of archive, lease, retention, and recovery-point guarantees.src/restore-trigger-adversarial.test.tsand the restore-trigger section ofsrc/sandbox-v1-adversarial.test.tsencode the destructive-operation invariant for both runtimes: a restore may run only when an administrator explicitly requests one, when the durable debug/hard-restart path armedrestore-needed, or when the container positively reports no tenant state. A gateway that is merely down, crashed, or being rebooted keeps its container filesystem, so the launch must never apply an older archive over it. Inconclusive signals (control-channel error, malformed probe, unreadable pointer) fail closed, and a request owner never runs the restore inline (the durable boot job owns it). The same suites pin the two hardening rules this matrix relies on: a container-state probe that is unavailable (no exists()) or failed (success: false, or a v1 probe exit code other than1) is never read as empty, and the legacy stuck-start watchdog checkpoints and verifies the recovery point advanced before it is allowed to reprovision a container.src/routes/api/billing.freeze.test.tsexercises persisted freeze/unfreeze, acting-user attribution, and the OR of stored and Dispatch freeze flags.src/billable-actions.emit.test.tscovers producer chunking, single-send fallback, delivery failures, frozen-customer exclusion, duplicate suppression, and fail-closed outcome normalization. A producer with neithersendBatchnorsendmust reject, never report unsent actions as enqueued.
Run bun run test:coverage from the repository root for comparable cross-package coverage.
The root typecheck delegates to the five workspace configurations listed above rather than
recursively checking vendored Elixir dependencies and marketing-site sources under one Node-only
configuration. Moltlazy has a package-local strict tsconfig.json; it no longer inherits the root
configuration. These runs do not prove deployed Cloudflare Access policies or live Stripe delivery;
unit line coverage is not a claim of complete end-to-end or branch coverage.
Billing-surface coverage.
src/client/pages/__tests__/HomePage.test.tsxrenders the Usage & Billing widget with cost data present (total plus the billable action/credit rollup) and asserts the empty ($0.00) and role-hidden cases.src/client/pages/__tests__/AccountPage.test.tsxcovers the Plan & Billing card (tier, seat/credit usage, past-due alert, non-admin hidden). The/api/admin/costs/v2/summaryaggregation (AE SQL query → summary) is unit-tested insrc/cost-summary.test.tsandsrc/routes/api/costs.test.ts.
UI Tests (Playwright, local)
The admin SPA has a Playwright suite that drives the real dashboard served by
the Vite dev server (bun run dev), with every /api/* call mocked per test.
No Worker, sandbox container, or Cloudflare credentials are required, so it runs
fully offline and is the fastest way to exercise the UI end-to-end.
cd apps/fouria
bun run test:ui # headless Chromium
bun run test:ui:headed # watch the browser
Specs live in apps/fouria/test/ui/ and the config is
apps/fouria/playwright.config.ts. The
shared mock helper (test/ui/support/api.ts) stubs GET /api/admin/whoami
(role/dev/demo flags), the page data endpoints, and the /api/health-ws +
/api/admin/devices/ws WebSocket streams. Coverage:
- App shell — sidebar, notification bell, ADMIN navigation, dev-only nav.
- Role access — ROOT → Diagnostics redirect, ADMIN blocked from Diagnostics, BASE_USER limited nav + route guard.
- Pages — every admin route renders for its role.
- Agent Studio API regression (
src/routes/api/agent-studio.test.ts) — the dedicated automations endpoint normalizes cron and interval schedule objects to display strings (including thekind/everyMs/anchorMscrash case).GatewayStatusLed.test.tsxverifies non-ROOT users never query debug status. - Agent Studio (
agent-studio.spec.ts) — the roster + overview render, all seven tabs (Capabilities, MCP, Memory, Automations, Connections, Health) open over the mocked RPC payloads,AGENTS.mdopens in the RPC editor, and a sleeping instance shows the wake affordance without any read waking it. The mock helper exposesstudio/studioDetailoptions plusSAMPLE_STUDIO_ROSTER/SAMPLE_STUDIO_DETAILfixtures. Unit coverage insrc/agents/studio.test.tsandsrc/routes/api/agent-studio.test.ts. - Health plugin pagination (
agent-studio.spec.ts) — 10 plugins per page, navigation to the final partial page and back, disabled boundary controls, and no pagination controls for a single-page list. - Capabilities pagination (
agent-studio.spec.ts) — independent 10-item pages for allowed/denied tools, subagents, skills, available/effective tools across groups, and notices; filtering resets pages before displaying matching items. - Releases (
releases.spec.ts) — the promotion page stages a release on a dev instance (with a commit message), applies it on production, and warns when production lacks thePROMOTION_SOURCE_BUCKETbinding. The mock helper exposesreleaseEnvironmentandpromotionSourceBindingforGET /api/admin/releases/statusand an emptyGET /api/admin/releases/history. Unit coverage insrc/client/pages/__tests__/ReleasesPage.test.tsx. - Notification bell — pending-device badge + dropdown, empty state.
- Persistence lock indicator (
PersistenceLockIndicator) — the lock widget beside the bell reflects the sharedPersistenceLockContextstate, which theBackupHistoryPanelraises/clears around a backup or restore (no polling): open/green when idle, closed/red while an operation is in flight, and hidden for BASE_USERs. Unit coverage insrc/client/components/__tests__/PersistenceLockIndicator.test.tsxandsrc/client/components/__tests__/BackupHistoryPanel.test.tsx. - Gateway circuit breaker (
gateway-circuit-breaker.spec.ts) — with the health streamdown, gateway queries are parked (no/api/admin/agentsrequest) and the waiting indicator shows; flipping the mocked health stream toupflushes the queued queries and renders the data. The mock helper exposesinstallApiMocks(...).sendHealth(snapshot)and theHEALTH_UP/HEALTH_DOWNfixtures for this. Unit coverage lives insrc/client/gateway-gate.test.ts,src/client/GatewayHealthContext.test.tsx, andsrc/client/api.test.ts. - Diagnostics dashboard (skip-not-park) — unit coverage in
src/client/pages/__tests__/DiagnosticDashboard.test.tsxandsrc/client/components/__tests__/ConfigSection.test.tsx: with the gateway down the page still renders but its container reads (/api/admin/config*, devices) are never dispatched, gateway-only debug probes are skipped with an inline explanation, and the Configuration Editor stays visible with a non-blocking waiting notice instead of a blocking spinner. - Billing widget (
billing-widget.spec.ts) — the Home “Usage & Billing” widget shows the billed total and billable action/credit rollup when the cost summary carries data,$0.00when it is empty, and is hidden for BASE_USERs. The mock helper exposes acostSummaryoption. - Cloudflare Components widget (
cloudflare-components.spec.ts) — the ROOT diagnostics widget renders each component’s status/latency/href, hides the AI Gateway when the API omits it, reveals the binding-derivedfactspanel on hover and keyboard focus, opens each card’s Cloudflare dashboard logs page in a new tab (offline dashboard stub), renders a plain non-clickable card when a component has nologsUrl, re-fetches on Refresh, and reflectssleeping/downstates. The mock helper’sHEALTH_COMPONENTSfixture carries the enrichedfacts+logsUrlshape returned byGET /api/health/components. - Diagnostics responsive layout (
diagnostics-responsive.spec.ts) — at a phone viewport the Cloudflare Components strip collapses into a compact status widget and expands on tap, the gateway + persistence status widgets sit parallel to it (the topbar omits them on the diagnostics page), the Fouria DevOps options render as a vertical icon + name menu (collapsed by default, tap to open), the Configuration Editor reflows to the viewport, and the page never scrolls sideways (checked at 320px). The mock helper exposesdebugRoutes(to render the Configuration Editor) and aconfigfixture for the parsed sections + raw JSON.
NixOS / Nix dev shell.
apps/fouria/flake.nixprovides the Playwright browsers viaPLAYWRIGHT_BROWSERS_PATH(pkgs.playwright-driver.browsers) plus thePLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTSandPLAYWRIGHT_HOST_PLATFORM_OVERRIDENixOS workarounds. Because the Nix browser bundle can lag the project’s npmplaywrightversion,playwright.config.tsresolves the newestchromium-*underPLAYWRIGHT_BROWSERS_PATHand passes it as an explicitexecutablePath; outside Nix the variable is unset and Playwright’s default resolution applies.
Key test files (see WORKER-DEVELOPMENT.md for full list):
| File | Tests |
|---|---|
auth/jwt.test.ts | JWT decoding and validation |
auth/jwks.test.ts | JWKS fetching and caching |
auth/middleware.test.ts | Auth middleware, DEV_MODE, E2E_TEST_MODE, dev-instance ROOT→ADMIN role preview |
gateway/env.test.ts | Container environment variable building |
gateway/process.test.ts | Process finding, lifecycle, stuck-start watchdog, and structured gateway-start failure correlation |
gateway/classify.test.ts | classifyGatewayStart running-only stuck-start verdict |
gateway/sandbox-exec.test.ts | Sandbox command results and structured process failure logs |
gateway/utils.test.ts | Process wait lifecycle and non-zero exit reporting |
index.test.ts | Worker middleware, proxy routes, and request failure trace context |
gateway/r2.test.ts | R2 mounting logic |
auth/rbac.test.ts | RBAC permission matrix |
metrics.test.ts | Cost/usage Analytics Engine instrumentation |
persistence.test.ts | R2 backup/restore, retryable cold-restore intent, scheduled-snapshot safety, cross-isolate lease, crash-consistent catalog + orphan reconciliation |
restore-trigger-adversarial.test.ts | Restore-trigger matrix: gateway down/reboot never restores live state; only an explicit latch or an empty container may restore; unknown probes fail closed |
sandbox-v1-adversarial.test.ts | v1 zero-data-loss and boot attacks, including “a v1 boot restores only an empty container” (gateway down, stopped container, probe failure, operator restore) |
E2E Tests (Cloud)
Deploys a real worker to *.workers.dev via Terraform + wrangler deploy. Requires Cloudflare credentials.
cd apps/fouria
# Full E2E with browser
bun run test:e2e
# CLI-only (no browser)
bun run test:e2e:cli
Setup (test/e2e/_setup.txt):
- Provisions cloud infrastructure via Terraform (R2 bucket) — the content-hash worker cache is temporarily disabled (issue #297), so every run provisions fresh
- Deploys worker via
wrangler deploy --var E2E_TEST_MODE:true - Starts Playwright browser via
plwr - Polls
/api/healthfor cold start up to the startup budget — 30s on a warm cache hit, otherwiseSTARTUP_TIMEOUT_SECS(600s default) - Opens browser to worker URL, waits for “Device pairing required”
Fresh-deploy failures are usually the shared v1 app. The top-level (local/E2E)
wrangler.jsoncstill declares theSandboxV1container application asfouria-sandbox-v1, which belongs to the production worker;server/deployrenames it per run (<worker>-sandbox-v1). If a fresh E2E deploy reports “Durable Object-managed Container application settings were not fully applied” after five attempts, check that rename first. Teardown probes the teardown token against the account endpoint and falls back to the deploy token, then reports the real container-app delete status — a staleE2E_CLOUDFLARE_API_TOKENused to make every delete silently no-op and leak billing applications.Phase-D suite assumptions.
y_production_profilecreates its recovery point on demand (POST /debug/backup/sync) becauseBACKUP_CHANGE_WATCHis off on the E2E worker, so no scheduled snapshot exists.zz_gateway_hard_restartreadsprocessIdfrom/api/health(the/status/pollingRPC fast path deliberately omits the process scan) and waits for the post-restore boot before capturing its baseline.get-agent-idretries the roster for a bounded window becauseh_cli_knowledgeruns it right after a plugin reload.
Tenant identity is explicit. Both fixtures bind
CUSTOMER_SLUG(acme-corpfor the cloud worker, matching the corpus’ mock tenant and the admin cross-tenant guard’sE2E_EXPECTED_CUSTOMER;start-localsets it for the billable profile). Without it the Worker derives its identity fromWORKER_NAME, which a bare*.workers.devE2E worker does not set, sogetCostContext().customerfalls back to thedefaultsentinel. That is not a valid tenant slug, so the software-manifest store refuses to build andPUT /api/admin/software/manifestanswers an instant 500 — the fixture bug that failedzzz_software_provisioning, not a route defect.
Gateway readiness — shared helper. After provisioning, the cctr
_setup.txtwaits for gateway readiness throughapps/fouria/test/lib/e2e-helpers.sh. Itse2e_wait_for_healthprobes/api/healthimmediately (the first probe also triggers the cold boot — there is no fixed 180s pre-poll grace) and requires bothstartup.okandreadiness.ok, polling every 5s. The wait stays bounded by the existing warm/cold startup budget: 30s on a warm cache hit, otherwiseSTARTUP_TIMEOUT_SECS(600s default). The same helper backstest/e2e-agent/_setup.txtand the parallel driver’s top-level readiness wait. It does not back every readiness check: the parallel runner’s generated per-group_setup.txtandab_gateway_ws_origin.txtkeep their own grace-free, bounded/api/healthloops (the latter also uses the sharede2e_classify_ws_upgradeverdict helper). The helpers are unit-tested inapps/fouria/src/e2e-helpers.test.ts.
E2E cost safety (issue #297): the container idle timeout is derived from the instance hostname (staging 15m / dev 2h — the Sandbox SDK 1.0
setInactivityTimeout()platform cap); there is noSANDBOX_SLEEP_AFTERknob. Billable production instances (production platform, production environment) do not auto-sleep. A container that outlives its run (failed teardown, workflow timeout) therefore stops billing at the derived idle instead of running 24/7, and the workflow teardown deletes the workers outright. Worker caching is disabled until the feature is reworked to reuse only within working hours and to delete on eviction.
Boot profiles: Both profiles provision the same production-profile image (
apps/fouria/Dockerfile,BUILD_PROFILE=production, the slim default) and the samestandard-2instance type (1 vCPU / 6 GiB — the smallerbasic1 GiB /standard-10.5 vCPU instances stall the gateway under the suite’s CLI load). The optional PDF/document tooling (pandoc+weasyprint) ships only in thedevbuild profile used by the dev/staging environment — seeapps/fouria/docs/WORKER-DEVELOPMENT.md. PR runs use the lightbasicprofile (E2E_TEST_MODE=true, short plugin timeouts) for fast feedback; develop runs use theproductionprofile (real plugin timeouts, container OTLP export + Cloudflare Tunnel). The profiles differ by env only — not the image or instance type. Both profiles bind the worker OTLP receiver with a random per-runOTEL_SHARED_SECRET(no GitHub secret required), so the OTel billing-queue test runs in either profile.
Single-worker parallel E2E: the
e2eworkflow provisions one isolated worker and runs dependency-grouped cctr suites against it. Independent phase A and B groups run concurrently; config/lifecycle/restore groups remain serial. Pull requests run boot-only by default.developand a full manual dispatch run the corpus; selective retries link only the suites recorded in the prior run’se2e-failed-*cache. The runner primes the CLI/device path once per group so device-admin suites do not each pay the cold first-connect cost.
E2E run statistics:
run-parallel.shwritese2e-stats.mdwith initial and per-group Gateway readiness times, cctr group outcomes and case durations, and per-group counts of Worker-launched OpenClaw CLI calls, Gateway WebSocket connections, and Admin HTTP RPC calls. The report and debug transcripts are uploaded as thee2e-logs-*run artifact, and mirrored to thee2e-artifacts-*branch on a best-effort basis (the branch push never fails the job). Transport counters require an explicitX-FourIA-E2E-Stats: 1request,DEBUG_ROUTES=true, and an explicitly boundE2E_TEST_MODEprofile marker (trueforbasic,falseforproduction). The long-lived Gateway process is excluded from CLI totals. CI pins cctr to0.31.0for stable report parsing.
Failed-only reruns: when a run reaches the gateway (
gateway-ready) but some suites fail, the job records those suite names and caches them (e2e-failed-<ref>-<shard>). The next run restores that list and passes it torun-parallel.sh, which links only those suites into the generated cctr groups (the PR already passed unit +validate-secrets, enforced byneeds). A full pass clears the list and the following run is a full pass again. A committed seed (.github/e2e-rerun-suites.txt, only when present) bootstraps a subset before any cache exists; it is not required in steady state.
Teardown (test/e2e/_teardown.txt):
- Dumps gateway logs + doctor output
- Stops browser (saves video)
- Destroys cloud resources (worker, container app, R2 bucket, Terraform)
Test files (alphabetical order — naming determines execution order):
| File | What it tests |
|---|---|
b_cli_devices.txt | openclaw devices list through /debug/cli |
c_cli_models.txt | openclaw models list, model catalogs |
d_cli_channels.txt | Channel probe, config inspection |
e_cli_config.txt | Config validate, set, unset via CLI |
g_cli_secrets.txt | Secrets audit, reload via CLI |
h_cli_knowledge.txt | Wiki init, status, get via CLI |
i_cli_cron.txt | Cron list, add dream job |
j_cli_integrations.txt | Integration config verification |
k_secrets_store.txt | Secrets Store (gateway token, CDP, container config, device pairing) |
l_cli_billing.txt | Pricing constants, costs, instance types, billable events |
m_cli_agent_tokens.txt | Agent turns, token usage metrics |
provision_and_dispatch.txt | Cloud-only — dispatch worker + Supabase tenant lifecycle. Its control-plane steps need INTERNAL_DISPATCH_INVALIDATE_TOKEN (staging value) and self-skip when it is unset; there is no hardcoded fallback. |
a_rbac_admin_isolation.txt | ADMIN role isolation (management API, cross-tenant) |
a_rbac_api_matrix.txt | ROOT/ADMIN/BASE_USER matrix + deny/allow decisions over dashboard + loading-page endpoints |
a_rbac_base_user_isolation.txt | BASE_USER isolation (whoami, blocked from management/secrets/writes) |
a_rbac_device_pairing.txt | Device pairing permissions for all 3 roles (with retry on timeout) |
a_rbac_root_isolation.txt | ROOT isolation (diagnostics access, blocked from management/agents/secrets) |
a_rbac_ui_pages.txt | UI page access for all 3 roles (25 test cases; every nav page route served) |
aa_health_gate.txt | Gateway health check between RBAC and CLI test groups |
ac_gateway_startup_concurrency.txt | Cross-isolate cold-start serialization after destroying the container |
ad_observability_canary.txt | Local-only — proxy header redaction + live-gateway-log RPC-stream warning (Local Explorer SQL API) |
o_opencode_credits_task.txt | OpenCode credits automation executed in the sandbox (direct consumer) |
o_leads_proxy.txt | Leads R2 proxy over the Worker WebSocket (self-skips without cdp-secret.txt) |
r_browser_cdp.txt | Cloudflare Browser Rendering BROWSER binding over the public /cdp shim (discovery, shared-secret auth, target + DOM + PNG); self-skips when the binding is unavailable |
r_browser_remote_profile.txt | Platform browser.profiles.cloudflare remote-CDP profile → container proxy → private cdp.internal bridge (registration, open, cross-call tabs) |
p_tasks_queue.txt | TASKS queue producer → consumer → sandbox round trip (self-skips w/o key) |
p_otel_billing_queue.txt | OTLP traces receiver → billable reconcile → BILLING_QUEUE producer (self-skips on a non-billable instance) |
test/e2e-billing/ | Local-only — billable worker emits to BILLING_QUEUE (observed via Local Explorer) + freeze drop |
p_scheduled_cron.txt | Local-only — both crons fire via /cdn-cgi/local/scheduled |
z_cli_staging.txt | API staging tests (costs, storage, users) |
zzz_cron_wake.txt | Cloud-only — container destroy, cron wake cycle |
Backup/restore suites are excluded from this corpus. They live in
apps/fouria/test/e2e-backup/so a defaultbun run test:e2e/test:e2e:localrun does not trigger their multi-minute cold boots and ~250 MB R2 snapshots. Run them withbun run test:e2e:backup(local) or throughphaseE_restorein the parallel runner. Files:zz_reprovision_guard,zzz_backup_cold_restore,zzz_backup_restore_skip_setup,zzz_backup_semantics, and the legacyzzz_persistence_lifecycle.
Retry diagnostics in cctr tests. cctr merges a test’s stdout and stderr into the value matched by
{{ … }}. A retry/progress line printed to>&2therefore breaks a strict matcher (json object,json bool, a bare number, …) once the retry path is taken — the flake only appears when the first probe is unhealthy. Write such diagnostics to a side log instead:echo "…" >>"$CCTR_WORK_DIR/e2e-retry.log" 2>&1.
o_leads_proxyrequirescdp-secret.txt. The test self-skips (and passes in ~0s) when the fixture file is missing, so a regression in the fixture wiring looks like a green test.test/e2e/fixture/server/startpersists$CDP_SECRETtocdp-secret.txton both the cache-hit and fresh-provision paths; if the test reportsskipped: true, the fixture is the place to look.
p_otel_billing_queueself-skips on a non-emitting instance. The OTLP traces receiver gates emission onisBillableEventsEnabled(src/dev-mode.ts): billable production and the staging platform always emit, while a dev-environment instance accepts + ignores approved spans (HTTP 200) and reportsbillable: false, never emitting toBILLING_QUEUE. The local E2E worker runs withDEV_MODE=trueand no routableWORKER_URL, so it is a dev instance — the test skips (like it already does for a missing secret /401) instead of asserting an emit that cannot happen locally. The emit path is asserted by the cloud profile (a productionworkers.devhostname / the staging platform) and bysrc/routes/otel-traces.test.ts. A dev instance can opt in explicitly withBILLABLE_EVENTS=true; the override changes emission only, so the sleep policy and the cost-row dimensions are untouched.provision_and_dispatchlikewise self-skips in local mode (no dispatch worker
- Supabase).
Worker billing-emit E2E (
test/e2e-billing/, local-only). The producer half of the billing workflow is covered locally by opting the worker into a billable identity (E2E_BILLABLE=true→start-localpasses a production-lookingWORKER_URL/CUSTOMER_SLUG). The suite POSTs a synthetic OTLP span, assertsemitted >= 1, observes theBILLING_QUEUEproducer via the Local Explorerqueue*span, then freezes the customer and assertsemitted == 0with no new queue span. Analytics Engine writes are not observable locally (no AE binding in Local Explorer; nowriteDataPointspan), which is why the lerma consumer suite uses a fake AE. Run withbun run --cwd apps/fouria test:e2e:billing:local; seeapps/fouria/test/e2e-billing/README.md.
h_cli_knowledgecaches the owner agent id. The corpus resolves the owner agent id once per suite throughtest/e2e/fixture/get-agent-id, which caches it in$CCTR_FIXTURE_DIR/agent-id.txtand reuses it across the sequentialwiki init/wiki status/wiki getcases. The helper reads/api/admin/agentsthrough Admin HTTP RPC, then Gateway WebSocket RPC, with the route’s CLI fallback for compatibility; caching avoids repeating that roster lookup per case while preserving the plugin/wiki assertions. The cache is suite/worker-run scoped (cctr copiesfixture/into$CCTR_FIXTURE_DIRonce and reuses that directory across cases) and is discarded when cctr cleans up the per-suite temporary fixture/work directory, not by_teardown.txt. The helper fails closed: an empty or malformed roster exits nonzero instead of silently defaulting tomain.
E2E Tests (Local)
Runs tests against the local Worker dev server (Cloudflare Vite plugin, vite dev) with containers via Docker. Avoids 5-10 minute Terraform provisioning. Cloud-only tests (dispatch worker + Supabase, container destroy/recreate) self-skip in local mode.
cd apps/fouria
# Local E2E with Docker + cctr (auto-builds Docker image)
bun run test:e2e:local
# CLI-only (no browser)
bun run test:e2e:local:cli
Setup (fixture/server/start-local):
- Sources
test/e2e/.dev.varsand installs it as the Worker’s.dev.varsfor the run (the Cloudflare Vite plugin has no--env-file;stop-localrestores the developer’s file) - Finds unused port (8787-8797)
- Starts
vite dev --port <PORT> --strictPort(also generatesCDP_SECRET/OTEL_SHARED_SECRET, injected as vars through the plugin’sconfighook, and uses/cdn-cgi/local/scheduledfor the cron test). The outertest:e2e:local*scripts runbun run build:dockerfirst (chained with&&, so it completes before the dev server starts);vite devthen builds/launches the container image through Docker. - Polls
GET /api/health?passive=1until the Worker answers any HTTP status (200/503/…) — a cold boot legitimately answers 503 while the container/gateway is still starting, so the gate must not require a200and must not usecurl -f. It probes the fast, non-waking passive endpoint so a poll does not wake the container or re-trigger the restore/ensureGatewayboot on every attempt. The window defaults to 120s and is overridable withE2E_STARTUP_TIMEOUT_SECS. The real gateway readiness wait (startup.ok+readiness.ok) is the separatewait for gateway API readinesssetup step. - Writes fixture files (dummy CF Access creds — ignored via
DEV_MODE=true)
Resource headroom. The local run drives the Docker daemon (container image + container),
workerd(Vite dev server + Miniflare R2/DOs), and thebun/cctrdrivers at once. On a memory-constrained host the whole process group can be killed bySIGABRT(OOM) mid-cold-boot — observed during_setup: start moltworker serverafter the image built and the dev server answered 503. Give Docker/the host at least ~8 GB and avoid running other containers alongside the suite. The readiness gate above fails fast when the dev server itself dies, but it cannot distinguish an OOM kill from a crash, so checkdmesg/Docker logs when the process group dies during startup.
Container failed to start/ no app container. If the suite stalls at the readiness step whiledocker psshows only a…-proxysidecar that is recreated every ~20 s, the host cannot start local containers at all: on Linux hosts wherenet.ipv4.conf.all.src_valid_mark=1(Tailscale sets this), the sidecar handshake is dropped and workerd times out (cloudflare/workerd#6860). The dev server prints the fix before building images; the short version isecho 'net.core.devconf_inherit_init_net = 2' | sudo tee /etc/sysctl.d/99-docker-netns-defaults.conf && sudo sysctl -w net.core.devconf_inherit_init_net=2, thendocker rm -f $(docker ps -aq --filter name=workerd-). See DEVELOPMENT.md.
Teardown (fixture/server/stop-local):
- Kills the Vite dev server process
- Restores the app’s
.dev.vars - Cleans up fixture files
Prerequisites:
# test/e2e/.dev.vars must contain:
DEV_MODE=true
E2E_TEST_MODE=true
DEBUG_ROUTES=true
MOLTBOT_GATEWAY_TOKEN=<value>
CF_ACCOUNT_ID=<id>
CLOUDFLARE_AI_GATEWAY_API_KEY=<key> # plus CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID
# The REST transport uses CLOUDFLARE_API_TOKEN, derived from the same AI-scoped key.
# Docker must be running
# wrangler must be authenticated via OAuth (wrangler login)
Cloud-only tests (self-skip when E2E_LOCAL=true):
| File | Reason |
|---|---|
provision_and_dispatch.txt | Requires dispatch worker + Supabase |
zzz_cron_wake.txt | Requires container destroy/recreate |
The Dispatch control-plane steps in
provision_and_dispatch.txt(and the staging integration corpus) sendINTERNAL_DISPATCH_INVALIDATE_TOKENto the deployed Worker and self-skip when it is unset. Both the Worker and Lerma get that value fromSTAGING_INTERNAL_DISPATCH_INVALIDATE_TOKENin CI; the Worker fails closed (403) without it, and there is no shared hardcoded key.
ad_observability_canaryasserts on the local worker’s own logs. The Local Explorer exposes the samelogs/spanstables as the Cloudflare Workers Observability API, scoped to the local dev worker, viaPOST /cdn-cgi/local/explorer/api/local/observability/query(wrapped by thefixture/observability-queryhelper). The canary uses it to (1) drive the catch-all proxy’s request log with sentinel Cookie / Authorization / CF-Access-Jwt-Assertion / tenant-metadata values and assert none reach the log store, and (2) fetch the running gateway’s logs and assert the fetch adds noReadableStream received over RPC disconnected prematurelywarning. Both cases are local-only (the API does not exist on a deployed worker), so they self-skip in cloud E2E; they fail loudly if the local API is unavailable rather than silently skipping, so the coverage cannot rot. These are the observability-validation canaries for the credential-logging and live-process-stream fixes.
Admin UI Render Timing (local-only, plwr)
test/e2e/ui_components_render.txt is a local-only browser suite that
validates the fouria admin (root) UI components and reports how long each takes
to render. It drives the plwr session started by _setup.txt (the same one the
pairing step uses), sets the X-Mock-Role cookie, loads each admin route, waits
for the component’s root selector, and measures, in milliseconds relative to the
page navigation start:
openMs— navigation start → SPA loaded (performance.now()sampled right afterplwr openreturns)componentMs— navigation start → the component’s selector is present (sampled right afterplwr waitreturns); this is the data-driven renderwaitMs—componentMs − openMs(client render + API round-trip after load)nav.*— Navigation Timing (domInteractive,domContentLoaded,load) and First Contentful Paint
Coverage mirrors the Playwright UI suite (test/ui/pages.spec.ts): the shell
(.admin-layout, .sidebar, gateway status LED, persistence lock, notification
bell) and every ADMIN page, plus the ROOT-only diagnostics page and its Fouria
DevOps + Cloudflare Components widgets.
The file also carries one behavioral case (driven directly through plwr)
that loads the diagnostics page against the local worker and asserts the
Cloudflare Components widget links the container card to its Cloudflare
dashboard page (href contains dash.cloudflare.com) and reveals its
binding-derived detail panel on hover (visibility flips hidden → visible,
panel text contains the Runtime fact). It self-skips when browser tests are
disabled or plwr is absent.
The file self-skips unless E2E_LOCAL=true (file-level %skip) and also
self-skips when E2E_SKIP_BROWSER_TESTS is set, so cloud runs stay green. The
fixture/ui-render-measure helper appends every record to
ui-render-times.jsonl; the final ui-render-report case writes a table to
ui-render-report.txt and prints a JSON rollup (stdout) with maxComponentMs
and the slowest component.
cd apps/fouria
# Just this suite — _setup.txt still starts the worker + browser:
bun run test:ui:render:local # build:docker + E2E_LOCAL=true cctr -vv test/e2e/ -p 'ui render'
# Or run it as part of the whole local corpus:
bun run test:e2e:localE2E Agent Subset (OpenClaw migration smoke)
apps/fouria/test/e2e-agent/ is a focused, browser-free subset of the CLI
surface used to validate an OpenClaw version migration. It reuses the shared
test/e2e/fixture (via symlink) and the same local-mode runner, but only runs
the commands the Worker shells out to:
b_cli_devices.txt—openclaw devicesc_cli_models.txt—openclaw modelsd_cli_channels.txt—openclaw channelse_cli_config.txt—openclaw config validateg_cli_secrets.txt—openclaw secretsh_cli_knowledge.txt—openclaw wikim_cli_agent_tokens.txt—openclaw agents/ token flowsn_gateway_soft_reload.txt— gateway soft reload (SIGUSR2; asserts the Node inspector stays closed and proves the restart with a newhttp server listeningline plus a stableprocessId)
Bounded agent turns.
m_cli_agent_tokens.txtbounds its Gateway turn withopenclaw agent --timeout 40. The CLI’s default turn deadline is 600s, which outlives the e2e HTTP budget (debug-cliwaits up to 80s server-side and 90s viacurl-auth;/debug/cliusestimeout_ms). A bounded turn returns its JSON envelope (result, timeout, or provider error) within the budget, and an empty response is reported as{"status":"timeout"}instead of failing the JSON-object assertion on an emptyjqresult.
cd apps/fouria
bun run test:e2e:agent # build:docker + E2E_LOCAL=true + browser skipped
The openclaw-migration skill runs this
subset after bumping catalog.openclaw to confirm the CLI commands still
behave.
Gateway Boot Smoke Test (local)
Validates that the fouria sandbox image actually boots a healthy OpenClaw
gateway on the local machine — no Worker, no cloud e2e, no third-party
machines. It builds the exact production image (apps/fouria/Dockerfile +
repo-root context, same base cloudflare/sandbox image wrangler uses) and runs
the real boot path (start-openclaw.sh → onboard → plugin installs → moltlazy
patch/validate → gateway launch → health poll) inside Docker. The container’s
exit code is the verdict: the in-container wrapper runs start-openclaw.sh
(whose own readiness loop already fails if the gateway dies or never binds),
then asserts /health returns {ok: true} while the gateway is still alive.
cd apps/fouria
# Default cold-boot smoke (no real credentials needed):
# onboard runs --auth-choice skip, tunnel + OTEL steps self-skip.
bun run test:gateway:boot
# Standalone native phases in the smoke container: `start-openclaw.sh setup
# → start-openclaw.sh launch → post-start`, then assert /health. This is NOT
# the Worker's production cold start, which is separately `restore → atomic
# start (setup + Gateway under one lock) → post-start` (see the table below).
bun run test:gateway:boot --native
# Same, plus the real cloudflare-ai-gateway onboard path and an assertion
# that the provider plugin is installed + enabled. Sources credentials from
# apps/fouria/.dev.vars.e2e (never echoed) — generate it from SECRETS.yaml
# with `bun run secrets:generate`.
bun run test:gateway:boot:ai
# Leave the container running after the assertion, with port 18789
# published to 127.0.0.1 for manual inspection (Control UI / WS).
bun run test:gateway:boot --keep
# Skip build:docker + docker build and validate an existing image.
bash scripts/gateway-boot-smoke.sh --no-build --image <tag>
Prerequisites: Docker running, network access (the image build pulls the
ClawHub plugins; boot-time installs are a fallback only, bounded by
start-openclaw.sh timeouts and non-fatal for boot).
What it validates vs. what it does not:
| Validated locally | Still needs local e2e / cloud e2e |
|---|---|
| Image builds from the production Dockerfile | Sandbox SDK supervisor + Worker RPC layer |
| onboard (skip or real AI Gateway) completes | RBAC / API routes / device pairing |
| Build-time plugin pre-install + enablement | Dispatch worker routing |
moltlazy patch + validate (config contract) | Cost observability / R2 backup flows |
gateway binds :18789, /health returns ok | Browser-driven Control UI flows |
--native: standalone start-openclaw.sh setup → launch → post-start phases in the smoke container (not the Worker’s production restore → atomic start (setup + Gateway under one lock) → post-start) | Worker’s restore + sandbox.exec supervision (handle/relaunch) |
E2E Parallel (single-worker parallel runner)
apps/fouria/test/e2e-parallel/ is the parallel replacement for the serial
test/e2e run. The e2e CI job no longer shards the corpus across N workers
(one per shard); it runs run-parallel.sh, which provisions ONE worker and
runs dependency-grouped cctr suites concurrently against it.
cd apps/fouria
bun run test:e2e:parallel # == bash test/e2e-parallel/run-parallel.sh
Suite scope (E2E_BOOT_ONLY). By default the e2e CI job sets
E2E_BOOT_ONLY=1 for pull requests: the runner provisions the worker, waits
for /api/health to report startup.ok + readiness.ok, and then tears down
without running any cctr group — a PR only proves the gateway boots. The full
corpus runs on develop (merge gate) and on a manual workflow_dispatch with
the full_tests input checked (the default for dispatch). Locally, set
E2E_BOOT_ONLY=1 to reproduce the PR check.
Phases (see the suite README for the full grouping):
phaseA_devices∥phaseA_rbac∥phaseA_health— parallel (the RBAC mock role is a per-request cookie, so the RBAC suites are independent; the two device-store suites sharephaseA_devicesto avoid racing the approve path)phaseB_readonly∥phaseB_async— parallelphaseC_stateful— config/secrets/cron (serial)phaseD_lifecycle— plugin + reload/restart + dispatch (serial)phaseD_startup— cross-isolate cold-start concurrency regression and the declarative software-provisioning suite (zzz_software_provisioning.txt: flag/report wiring, manifest validation + upload, cold reprovision installs the declared software, boot-terminal ticks, and an on-demand run that installs a newly declared package while skipping the already-installed one), serial, each with a separate startup-sized budgetphaseE_restore— reprovision guard followed by the backup/restore lifecycle suites, resolved fromtest/e2e-backup/(serial, startup-sized budget). Skipped in GitHub Actions by default because it drives multi-minute cold boots and ~250 MB R2 snapshots; run it locally, or force it in CI withE2E_RUN_RESTORE_PHASE=1. For a standalone local run of just these suites usebun run test:e2e:backup.
Phase caps.
phaseD_lifecycleis bounded byPHASE_D_TIMEOUT_SECS(default 1200s / 20 min; the recovery suites plusprovision_and_dispatchnow legitimately exceed the old 7-minute budget, and killing the group mid-dispatch left a restore in flight). The cold-start concurrency group has its ownPHASE_D_STARTUP_TIMEOUT_SECSbudget (defaultSTARTUP_TIMEOUT_SECS + 180s); backup/restore suites usePHASE_E_TIMEOUT_SECS(defaultSTARTUP_TIMEOUT_SECS + 300s). The full run’s initial readiness budget is 1200s (basic) or 1500s (production). On a group timeout cctr fails (exit 124) and teardown still runs.
cctr 0.31.0 has no
#comments. cctr silently drops a corpus file’s tests at the first top-level#line and reports the tests collected so far (often zero) with no error. Corpus files must begin with===(or a lone file-level%skip/%shell/%platformfollowed by the first===), and a#comment must never sit before the first===, between a file-level%skipand the first===, or between two tests. Put rationale in test descriptions or docs, never in a#line (comments inside a test body/expected/whereblock are fine).node scripts/check-corpus-comments.mjs— run by theunitCI job — fails on any violating file, andcctr -l <file>prints the parsed test count.
Selective retry (
retry_failed). A slow container can make a single backup call exceed its curl budget, failing a suite that would pass on a warm run. Rather than inflate the curl timeouts (which hides real hangs), the workflow records the failed suites — theRecord failed e2e suitesstep writes✗ <group>/<suite>markers to/tmp/moltworker-e2e-failed.txt, saved to thee2e-failed-<ref>-<shard>-<run_id>actions cache. Re-run only those suites with a manualworkflow_dispatchand theretry_failedinput checked:run-parallel.shreads the restored file, links only the failed suites into their groups, skips every other group, and still provisions a single worker. The flag forces the suite to run (it overridesfull_tests/boot-only). AphaseX_group/*entry (written when a group is killed by its phase cap) retries every suite in that group. The cache is saved only when something failed, so the restore always resolves to the most recent failing run.
Single-owner startup concurrency regression.
ac_gateway_startup_concurrency.txtis a cloud-only regression for the OpenClaw 2026.9.8 cross-isolate launch race: it checkpoints tenant state, destroys the container, fires several concurrent cold non-passive/api/healthprobes (the cold-boot fan-out across Worker isolates), then asserts the gateway reachesstartup.ok + readiness.ok, that exactly one running Gateway owner exists — the process may appear asstart-openclaw.sh startbefore the atomic wrapperexecs, or asopenclaw gateway runafter — and that the observable gateway output never shows eitherOpenClaw migration inputs changed during startuporstate.sqlite is undergoing offline maintenance. Because it destroys the shared worker’s container it runs inphaseD_startup, after the ordinary lifecycle group and before restores, with its own startup-sized budget. It runs serially — never alongside the parallel phase A/B suites. Run it directly withcctr apps/fouria/test/e2e/ac_gateway_startup_concurrency.txt; it self-skips underE2E_LOCAL=true.
Each group is a generated cctr suite whose test files symlink the originals
in test/e2e/ and whose _setup.txt attaches to the shared worker (copy
credentials → wait readiness → warm the CLI/device identity). It forces
E2E_NO_CACHE=1 (full teardown, no container left billing) and self-skips under
E2E_LOCAL=true.
Readiness wait. The driver’s top-level wait uses the shared
e2e_wait_for_healthhelper (immediate first probe, no fixed grace, bounded bySTARTUP_TIMEOUT_SECS). The runner defaults that budget to 1200s; the workflow overrides it per profile (basic 1200s, production 1500s). The generated per-group_setup.txtdo not call the shared helper — each keeps its own grace-free, budget-bounded/api/healthloop that attaches to the already-warm worker. Because the phases run concurrently against one worker, the per-group readiness savings do not sum to the same job-time reduction — wall time tracks the slowest phase, not the total of the individual waits.
parallel-probe + a_parallel_spike.txt remain as a raw-concurrency
diagnostic (how many concurrent CLI calls one worker sustains); run the
probe suite directly with cctr -vv test/e2e-parallel/.
Logging policy. Verbose transcripts — the provisioning output
(terraform/wrangler), each group’s cctr -vv output, and the teardown/gateway
dumps — are written to /tmp/moltworker-e2e-logs/*.log and uploaded to the
e2e-artifacts-<profile> branch, so the CI job console only shows concise
[parallel] progress plus per-suite pass/fail markers and the summary. The
serial test/e2e _teardown.txt follows the same rule: gateway/doctor dumps
are written to disk only (never tee’d into the console).
See apps/fouria/test/e2e-parallel/README.md
for the per-file parallelizability classification, the ordering graph, and the
observed concurrency results.
Backup Semantics E2E
apps/fouria/test/e2e-backup/zzz_backup_semantics.txt and
.../zz_reprovision_guard.txt prove the backup invariants against a real
container (local Docker or cloud) and real R2. They are registered in
phaseE_restore (serial; reprovision guard runs first) and are the E2E counterpart of the route/unit tests
(src/routes/public.test.ts, src/routes/debug.test.ts, src/index.test.ts) and
the dispatch fan-out unit test (workers/fouria-dispatch/test/unit/cron-fanout.test.ts).
Run the local-safe cases standalone (excluded from the default test/e2e corpus):
cd apps/fouria
bun run test:e2e:backup # build:docker + E2E_LOCAL=true + browser skipped
See apps/fouria/test/e2e-backup/README.md.
Declared-software restore E2E (softwareProvisioning)
apps/fouria/test/e2e-backup/zzz_software_binaries_restore.txt is the runnable
end-to-end example of a gateway whose agent skills need binaries that are not
in the image. The manifest
(apps/fouria/test/e2e/fixture/software/gog-goplaces.json) declares gogcli
(the gog binary for the gog skill) and goplaces, the two skills are
installed with openclaw skills install skills-sh:openclaw/openclaw/{gog,goplaces} --global, and after a backup (?inline=1) → /debug/destroy-container →
restore it asserts that both binaries are back on PATH because the next
provisioning cycle re-installed them (the report startedAt advances), while the
skills come back from the snapshot rather than being reinstalled. It is
local-only (self-skips unless E2E_LOCAL=true) and needs container network
access (GitHub release downloads plus the ClawHub skills.sh resolver).
cd apps/fouria
E2E_LOCAL=true cctr -vv test/e2e-backup/zzz_software_binaries_restore.txt
The dependency-free equivalent — the same contract driven directly against the
sandbox image, without workerd/Miniflare — is
bun run --cwd apps/fouria test:software:smoke
(scripts/software-restore-smoke.sh): provision → install skills → snapshot →
fresh container → restore → re-provision, with five PASS: checks.
| Case | Invariant | Mode |
|---|---|---|
| health probes leave the catalog unchanged | I1 health is observability-only | local + cloud |
POST /control X-FourIA-Control: backup on a running instance | I2 | local + cloud |
| async sync commits a durable job and promotes a stable pointer | I12 durable job + stable point | local + cloud |
idle instance returns skipped:'not-running' without waking | I2/I3 | cloud |
| control-backup → destroy → cold restore preserves a sentinel | I4 | cloud |
encryption at rest (data.sqsh.enc, no plaintext) | I8 | cloud (opt-in) |
retention prunes catalog + objects to BACKUP_MAX_VERSIONS | I9 | cloud |
| worker is bound only to its per-run bucket | I11 | local + cloud |
reprovision 409 / force / running checkpoint | I5/I6/I7 | cloud |
Promotion (dev → production) coverage
The former test/e2e-push/ suite drove the HTTP Worker → Worker push
(POST /api/admin/releases/promote fetching /api/admin/releases/import). That
transport is gone: promotion is a private R2 stage/pull (the dev instance
writes promotion/staged/<id>.tar.gz + promotion/pending.json to its own
bucket; production reads it through PROMOTION_SOURCE_BUCKET on boot, or on
POST /api/admin/releases/apply). There is no import route to push to, and two
local Workers cannot share a miniflare R2 bucket, so the two-worker local harness
could not validate the new transport.
The transport is covered by:
src/services/promotion/r2-source.test.ts— stage → pending → apply → applied pointer, content hash + commit message, staged/applied history entries, the merged dev+production history view, plus target/environment/tenant gating, already-applied, no tenant state, parked, busy, missing bundle, and idempotency.src/routes/api/releases.test.ts—202stage (message/author pass-through), production409, apply + checkpoint, statusavailable, history, and RBAC.src/index.test.ts/src/gateway/process.test.ts— the bridge registration and the boot-time apply hook.
The dev/production axis remains the -dev hostname
(isDevAgentsEnabled), not DEV_MODE.
Cloud helpers: test/e2e/fixture/r2-list and r2-clear list/empty the per-run
bucket via the R2 REST API (needs CLOUDFLARE_TEARDOWN_API_TOKEN or
CLOUDFLARE_API_TOKEN). The E2E deploy sets deterministic
BACKUP_INTERVAL_MINUTES=1 / BACKUP_MAX_VERSIONS=2; set
E2E_BACKUP_ENCRYPTION=true to run the encrypted-at-rest case. Note that the
configured interval is a floor: the Worker stretches the effective scheduled
cadence to max(BACKUP_INTERVAL_MINUTES, 2 × the last snapshot duration), so a
slow snapshot cannot run back-to-back. Explicit snapshots (POST /api/admin/storage/sync, checkpoints) bypass that gate and stay immediate.
The same invariants are asserted at the fast layer by
workers/fouria-dispatch/test/unit/cron-fanout.test.ts (the cron forwards
backup only to non-frozen tenants) and src/persistence.test.ts (the “stable
recovery point promotion” block: the recovery pointer advances only, fails
closed on a malformed pointer, and retention skips the prune when the pointer is
unreadable). Manual-restore hardening is unit-tested in src/persistence.test.ts
too: a setRecoveryPoint: false restore leaves the pointer untouched, an
explicit restore opens the maintenance window and stops the gateway while it
runs, and restoreIfNeeded defers while that window is active. The route’s
?setCurrent=0 mapping is covered by src/routes/api/storage.test.ts, the
durable-flag round-trip by src/activity/backup-jobs.test.ts /
src/activity/scheduler.test.ts, and the Backups-page checkbox by
src/client/components/__tests__/BackupHistoryPanel.test.tsx, plus the
Cloudflare @cloudflare/vitest-plugin route tests planned in
apps/fouria/test/e2e/BACKUP-E2E-DESIGN.md.
Restore legibility (which backup/date is loading) is covered by
src/persistence.test.ts (readRecoveryState; the restore target is written
during sandbox.restoreBackup and cleared on success), src/assets/loading.test.ts
(the loading page consumes data.recovery / recovery.restoring /
recovery.committed), and src/client/components/__tests__/BackupHistoryPanel.test.tsx
(the recovery-point badge + summary). The local observability canary
test/e2e/ad_observability_canary.txt shows how to query the local dev server’s
Local Explorer SQL API (test/e2e/fixture/observability-query) to confirm the
restore-loop guards (Restore deferred — tenant maintenance in progress,
GatewayMaintenanceError) fire from the worker’s own logs.
Vitest 5 caveat. The Cloudflare route-integration layer is deferred: both
@cloudflare/vitest-pool-workersand the renamed@cloudflare/vitest-pluginpeervitest ^4.1.0, and under Vitest 5 the pool fails withSyntaxError: Unexpected identifier 'file'(known open issue cloudflare/workers-sdk#15618; workaround: pinvitest ~4.1.11).
Restore sequencing. Cold‑restore survival (I4) is opt‑in via
E2E_BACKUP_RESTORE=1(zzz_backup_cold_restore.txt) because restoring the ~250 MB local‑binding archive is far too slow for CI (observed >13 min without completing). The defaultphaseE_restorecases (catalog, health‑observability, running‑backup, idle‑skip, bucket isolation, encryption, retention) do not restore and usefixture/wait-ready(one warm request, then?passive=1polls) so overlapping restores cannot abort each other withOperationInterruptedError: … runtime changed(root cause:BACKUP-E2E-DESIGN.md§9).
cctr corpus gotcha. cctr 0.31 silently registers zero tests for a
*.txtcorpus file whose first line is a#comment, or when comments sit between a file-level%skipand the first===. Corpus files must begin with===(or a lone directive followed by===); keep prose in test bodies or at the end of the file.cctr -l <file>prints the parsed test count.
Business Rules Corpus
Contract tests at apps/fouria/test/business-rules/ that validate cross-cutting concerns. These form the “T0 validation gate” per issue #137.
cd apps/fouria
cctr test/business-rules/| Corpus File | Validates |
|---|---|
pricing.txt | Per-action billing, success-only, BYOL discounts, VISION.md alignment |
rbac.txt | ROOT/ADMIN/BASE_USER permission matrix, route guards, and dashboard + loading-page endpoint coverage |
tenant-isolation.txt | Cross-tenant access controls, canAccessCustomer |
operational-modes.txt | DEV_MODE, E2E_TEST_MODE, DEBUG_ROUTES, DEMO_MODE, running-only backups, health observability-only, late-ready gateway adoption, health is not a routing gate (ROOT-revivable frozen), reprovision fail-closed guard, automatic doctor --fix repair for a launch blocked by a corrupt state database |
cost-tracking.txt | 6 Analytics Engine event types, field schemas |
dashboard-api-contract.txt | Admin API endpoint existence |
agent-visibility.txt | BASE_USER agent visibility rules |
credits-and-plans.txt | Credit model, plan limits, BYOL exclusion, p95 pricing |
apps/lerma — Phoenix Web App
Unit Tests
Mirror lib/ structure under test/lerma/ and test/lerma_web/. Run with ExUnit via mix.
cd apps/lerma
# All unit tests (creates database, runs migrations)
mix test
# Single file
mix test test/lerma/clients_test.exs
# From the repository root: compile + deps.unlock + format + Credo + tests
cd ../..
mix precommit
Key test files:
| File | Tests |
|---|---|
lerma/clients/client_test.exs | Client CRUD, validation, Ecto schema |
lerma/clients/user_test.exs | User model, invitation lifecycle fields |
lerma/clients_test.exs | Invite/accept/reject, instance-scoped Access sync, reserved staging-client protections |
lerma/applications_test.exs | Application CRUD, one-time provisioning key generation/rotation/revocation and digest-only storage |
lerma/staging_tenant_test.exs | Staging-only client bootstrap, production isolation, and database delete/rename/truncate guards |
lerma/emails_test.exs | Confirmation email content and links |
lerma/invitations_test.exs | Background invitation email delivery and PubSub alert |
lerma/instances/instance_test.exs | Instance lifecycle |
lerma/instances_upgrade_test.exs | Upgrade version guards, per-application filtering, and repeat-promotion idempotency |
lerma/provisioning/tenant_provisioner_test.exs | Tenant provisioning logic |
lerma/billing_test.exs | Billing calculations |
lerma/billing_plans_test.exs | Global plan templates, assign/clone, autobuy toggle |
lerma/billing_rules_corpus_test.exs | Billing business rules (a–m) incl. production gating |
lerma/billing_local_workflow_test.exs | Local workflow: AE rows → reconcile → consume; configured/unconfigured/duplicate events, watermark ownership |
lerma/billing_becaria_trace_test.exs | BecarIA flow: AE rows → reconcile → exact per-action credits + sums |
lerma/credits/credit_pool_test.exs | Credit pool management |
lerma_web/live/billing_plans_live_test.exs | Global plan library + designer |
lerma_web/components/billing_plan_component_test.exs | Client board plan assignment + summary |
lerma/analytics/cf_analytics_test.exs | Cloudflare Analytics Engine queries |
lerma/observability/otlp_test.exs | OTLP/HTTP JSON parsing, allowlist privacy guard, child-span drop, caps, stable dedupe keys |
lerma/observability/telemetry_test.exs | Idempotent destination ingestion, uptime/failure ratios (unknown ≠ downtime), window helpers, retention prune |
lerma/cloudflare/observability_test.exs | Worker upload metadata: destination names, per-signal sampling, persist: false, disabled switch |
lerma/cloudflare/observability_destinations_test.exs | Destination reconcile: create/PATCH staging + production pairs, 403/500 handling, env-specific names |
lerma_web/controllers/api/otlp_ingest_controller_test.exs | Destination ingest auth (fail closed), 415 for protobuf, gzip delivery, privacy drop, replay idempotency |
lerma_web/plugs/otlp_body_reader_test.exs | Gzip/JSON body decoding, size caps, malformed-body 400, path/method passthrough |
lerma_web/controllers/api/observability_controller_test.exs | Uptime endpoint ratios/status/windows, unknown when no observations, sampling rate surfaced |
lerma_web/components/worker_telemetry_widget_test.exs | Worker telemetry widget unknown/healthy/degraded render |
lerma_web/components/health_events_widget_test.exs | Health events timeline empty state, recorded transitions, dev-instance rendering |
lerma_web/controllers/api/instance_controller_test.exs | Instance API endpoints |
lerma_web/controllers/api/provisioning_controller_test.exs | Global/application-key auth, application-scoped upgrade and status |
lerma_web/controllers/api/ai_gateway_controller_test.exs | Global provisioning-key auth; fails closed when unset |
lerma_web/controllers/api/credit_usage_controller_test.exs | Billing reconcile + credit-usage API (bearer auth) |
lerma_web/live/instance_live_test.exs | Instance LiveView |
lerma/client_docs_test.exs | Client docs sources (proposal + per-env stores), R2 listing/reading, default document formats, path traversal |
lerma/client_docs/markdown_test.exs | Safe Markdown preview: frontmatter, headings, tables, lists, code, relative images, unsafe links, HTML escape |
lerma_web/controllers/client_docs_controller_test.exs | Raw docs responses: content type, inline/download disposition, sandbox CSP, not-found/invalid-path errors |
lerma_web/live/client_live_test.exs | Invitation UI, public acceptance flow, and the lazy documentation widget tree/preview/live |
lerma_web/live/application_live_test.exs | ROOT-only application key generation and one-time reveal |
lerma_web/plugs/cf_access_auth_test.exs | CF Access auth plug |
Per-test tags:
@tag cf_auth: %{email: "...", name: "..."}— injects CF Access authentication@tag :integration— excluded by default, must run explicitly viamix test --only integration
Staging Fouria build promotion
.github/workflows/promote-staging.yml listens for completed Test runs on
develop, then queries the source run’s Actions jobs and proceeds only when
build-and-push concluded success. The script promotes the immutable
unstable-<sha> image/bundle pair, no-ops when the staging client tenant is
already running that version, and polls Lerma’s status endpoint until the
running application instances converge. The GitHub secret
STAGING_PROVISIONING_API_KEY must match the one-time generated key for the
Fouria application in staging Lerma. This cloud workflow is not run by local
mix test; the application-scope/auth/idempotency contracts are covered by the
ExUnit tests above.
E2E Tests (cctr)
Sequential cctr tests against real Cloudflare dispatch namespace infrastructure.
cd apps/lerma
mix test.e2e
Setup (test/e2e/_setup.txt):
- Validates
jq,uuidgentools - Sources
.dev.varsfor Cloudflare credentials - Validates
CLOUDFLARE_API_TOKEN,CF_ACCOUNT_ID,WORKERS_SUBDOMAIN,CF_ENVIRONMENT - Computes dispatch namespace (
fouria-tenantsvsfouria-tenants-staging) - Starts Phoenix on port 4002 via
mix phx.server
Teardown (test/e2e/_teardown.txt):
- Reads client/instance IDs from fixture directory
- Deletes instance (tears down CF worker + container application) via API
- Deletes client via API — fully decommissions the tenant (remaining instances and container applications, Access app/group, dedicated AI Gateways, R2 backup buckets) and archives the client record
- Kills Phoenix process
Test files:
| File | Tests |
|---|---|
client_provisioning.txt | Create client → create instance → wait for running → verify gateway → verify CF metadata |
env_isolation.txt | Correct namespace, wrong namespace rejection, environment tags, Analytics Engine dataset names |
Billing trace: deterministic + local
The former staging billing-trace suite (test/e2e_billing_trace/) drove a
deployed worker’s public POST /api/otel/v1/traces. That endpoint no longer
exists: OTLP ingest is the sandbox bridge (http://otel.internal/v1/traces),
resolved by the Sandbox SDK inside the Worker, so it is only reachable from
inside the tenant container. There is no public ingest route and no Access path
app to configure.
The five BecarIA actions (telegram_query, queue_reevaluation,
signal_analysis, email_draft, review_analysis, 1 + 1 + 3 + 5 + 10 = 20) are
still covered end to end by:
test/lerma/billing_becaria_trace_test.exs(deterministic, stubbed Analytics Engine stream, runs in CI), andtest/e2e_billing_local/(mix test.e2e.billing:local, fake Analytics Engine, no Cloudflare account).
For a manual replay inside a container, run
apps/fouria/scripts/emit-billing-actions.ts with
--base-url http://otel.internal (the bridge keeps OTEL_SHARED_SECRET).
ANALYTICS_ENGINE_API_TOKEN is Worker-only and irrelevant to emission — the
Worker writes the billable row with its ANALYTICS binding; the token only
powers the dashboard read-back and is never given to a container.
Local Billing-Workflow E2E (fake Analytics Engine)
test/e2e_billing_local/ (opt-in, run separately from mix test.e2e) exercises
the billing consume path entirely locally — no Cloudflare account, no staging
DB, no deployed worker:
fake Analytics Engine (bun)
└─ POST /accounts/:id/analytics_engine/sql ← BillingReconcile reads here
└─ lerma (mix phx.server, MIX_ENV=test)
└─ POST /api/credits/v1/reconcile → BillingConsumption → GET /usage
It uses a mock billing definition (fixture/seed_plan.exs, not BecarIA) and
feeds synthetic billable-action AE rows directly (OpenClaw’s OTel emission is
owned/tested by the OpenClaw team). Cases: configured + match_rule events,
unconfigured/unmatched/blank events, duplicate action_id redelivery, AE outage
(502, no billing), and tenant isolation.
cd apps/lerma
mix test.e2e.billing:local
Why a fake AE and not Cloudflare Local Explorer? Local Explorer
(docs)
does not support Analytics Engine — it is absent from the supported-bindings
table and from the Local Explorer OpenAPI, and local traces emit no
writeDataPoint span. A fake of the AE SQL API is therefore the only local
observability surface for the billable-event stream. Local Explorer is used on
the Worker side for logs/spans (apps/fouria/test/e2e/ad_observability_canary.txt).
See apps/lerma/test/e2e_billing_local/README.md.
Test Infrastructure Configuration
apps/lerma/mix.exs:
test: ["ecto.create --quiet", "ecto.migrate --quiet", "test"]
"test.e2e": ["cmd cctr test/e2e/ --sequential -v"]
Root mix.exs: The credo alias forwards all arguments to each Elixir
workspace project, so mix credo --strict works from the repository root without
running Credo against the root workspace shell project.
credo: &run_credo/1,
check: [
"workspace.run -t format -- --check-formatted",
"credo --strict"
],
precommit: [
"workspace.run -t compile -- --warnings-as-errors",
"workspace.run -t deps.unlock -- --unused",
"workspace.run -t format",
"credo --strict",
"license_audit",
"workspace.run -t test"
]
# Private alias helper:
defp run_credo(args) do
args = if "--strict" in args, do: args, else: ["--strict" | args]
Mix.Task.run("workspace.run", ["-t", "credo", "--" | args])
end
test_helper.exs: Excludes :integration tests, starts SQL sandbox.
Mocking: Uses Bypass for HTTP mocking and Meck for Elixir mocking. Not available outside :test env.
packages/moltlazy — Config Module + SDK
Unit Tests (Vitest)
cd packages/moltlazy
bun run test # vitest run
bun run test:watch # watch mode
Config module tests (tests/config/):
| File | Tests |
|---|---|
gateway.test.ts | Gateway config section (port, mode, trustedProxies, allowedOrigins) |
session.test.ts | Session config (dmScope) |
tools.test.ts | Tool search configuration |
logging.test.ts | Logging section (level, format, transports) |
validate.test.ts | CLI validation-error formatting (object-shaped openclaw errors) |
include.test.ts | $include directive injection |
SDK module tests (tests/):
| File | Tests |
|---|---|
sdk.test.ts | SDK client initialization |
sdk-cli.test.ts | SDK CLI fallback mechanisms |
sdk-gateway-status-ws.test.ts | Gateway WebSocket status checks |
sdk-ws-client.test.ts | WebSocket client connection lifecycle |
Agent tests (tests/):
| File | Tests |
|---|---|
agents.test.ts | Agent defaults, list generation |
index.test.ts | Main module (patch flow, section ordering) |
schema.test.ts | Zod schema validation for moltlazy config |
obsidian-sync.test.ts | Obsidian sync skill configuration |
coverage-gaps.test.ts | Coverage gap tracking across modules |
E2E Tests (cctr)
cd packages/moltlazy
bun run test:e2e # Build + cctr| Test Suite | Scenarios |
|---|---|
cli/basic.txt | moltlazy patch from empty, --help displays man-style docs, -h |
cli/flags.txt | --file override, --help, unknown commands |
cli/errors.txt | Validate pass/fail, nonexistent file, filesystem errors |
full-flow/roundtrip.txt | Full patch → verify output (gateway, $include, agents, session) |
skills/discovery.txt | Skills directory discovery, empty skills dir, re-patching |
workspace/seeding.txt | SOUL.md creation, fouria-builder workspace, safe mode, immutability |
Setup: Validates that dist/cli.js exists (pre-build check).
OpenClaw CLI version: the
moltlazy-integrationCI job validates the generated config with the OpenClaw CLI pinned to the version baked into the container (rootpackage.jsoncatalog, currently2026.7.1-2) instead ofopenclaw@latest. This keeps the config-schema contract aligned with what is actually deployed and avoids flaky failures from upstream schema drift. Bump the pin in.github/workflows/test.ymland this doc together whenever the container’s OpenClaw version changes.
Smoke Tests (Vitest)
cd packages/moltlazy
bun run test:all # Unit + E2E
OpenClaw plugins (external)
The cloudflare-unified-billing plugin — its unit tests, live AI Gateway
integration tests (infer-smoke, provider-smoke, cf-cost-api), and CI —
moved to Paso4/openclaw-plugins.
Run them from a checkout of that repo:
cd plugins/provider/cloudflare-unified-billing
bun run test # unit
bun run test:integration # live (requires AI Gateway creds + PLUGIN_LIVE_TESTS=true)
See OPENCLAW-PLUGINS.md.
CI Pipeline
The .github/workflows/test.yml workflow defines the CI pipeline (see reference table in AGENTS.md for the full job graph):
unit ──┬─ build-and-push → e2e (PR: single shard · develop: sharded matrix)
└─ moltlazy-integration (only when packages/moltlazy changed)
changes ── iac-test (only when apps/*/iac changed)
The changes job git-diffs the event SHAs and gates iac-test and
moltlazy-integration, so unrelated changes no longer spin them up. The
local-integration job was removed.
The Changesets version pull request runs no CI. The “Version Packages” PR is
opened from the changeset-release/develop branch by
.github/workflows/version-packages.yml; it only bumps versions and writes
changelogs, so every PR-triggered workflow exits immediately for that head
branch — test.yml (changes, unit, validate-secrets, build-and-push),
lerma.yml (quality, test), staging-validation-ci.yml and
changeset-status.yml. The guard is the head-ref prefix
!startsWith(github.head_ref, 'changeset-release/'), which is empty on
push/workflow_dispatch, so those paths are unaffected. Skipped jobs report as
skipped (not pending) to branch protection, and no CD workflow can be
affected: both promote-staging.yml and deploy-dashboard.yml only ever act on
a successful run whose head_branch is develop.
build-and-push builds the worker, validates the wrangler multipart bundle via
scripts/validate-bundle.mjs, then builds the container image and pushes it to
the Cloudflare registry as fouria:unstable-<sha> and uploads the worker
bundle to the fouria-bundles R2 bucket as unstable-<sha>/fouria.mjs. On
develop, e2e consumes that image (via IMAGE_REF) instead of rebuilding it.
The deploy-dispatch.yml workflow also deploys workers/fouria-dispatch
(staging on push to develop, production on a stable CalVer release or a manual
workflow_dispatch). deploy-fouria.yml is the fouria CD: on a stable CalVer
release (vYYYY.M.D / same-day vYYYY.M.D-N, classified from the tag name, not
the GitHub prerelease flag) it promotes the CI-built unstable-<sha>
artifacts — retagging that image to latest + the release tag and copying the
bundle to R2 under the release tag + latest. Nothing is rebuilt. Release
commits are pushed with [skip ci], so they never
re-trigger this CI pipeline.
Production deploys are stable-tag-only: lerma filters the deployable versions to
stable release tags and Instances.update_instance_version/3 rejects unstable
ones (staging keeps the unstable channel). The cleanup-stale-images.yml cron
never deletes an image (or matching R2 bundle) still referenced by a container
application — including in-progress rollout versions — and fails closed if that
in-use set cannot be resolved. Covered by
apps/lerma/test/lerma/cloudflare/image_registry_test.exs,
apps/lerma/test/lerma/cloudflare/registry_bundle_test.exs,
apps/lerma/test/lerma/instances_test.exs, and
apps/lerma/test/lerma_web/live/instance_live_test.exs.
Container image upgrades roll out with strategy: "rolling" (a single
step_percentage: 100 step) — see Lerma.Cloudflare.Containers. The tenant
container is Durable-Object-managed, and a new_instances rollout does not
replace a DO-bound instance: the platform stages the target as a rollout version
at 0% and the application keeps its previous image, so new_instances silently
left promoted tenants running the old image. Promotion convergence is also
sourced from the live container image tag (Containers.fetch_deployed_config/1)
in Instances.at_requested_version?/2, not the recorded deployed_version, so a
promotion cannot report success while the container still runs the previous
image (and a genuine miss can be re-promoted instead of being skipped). Covered
by apps/lerma/test/lerma/cloudflare/containers_test.exs and
apps/lerma/test/lerma/instances_upgrade_test.exs.
Plugin Integration (unified billing e2e) — moved
The cloudflare-unified-billing e2e tests and their dedicated workflow moved to
Paso4/openclaw-plugins. That repo
runs live AI Gateway integration tests on develop only (they cost real
credits), decoupled from this monorepo’s test.yml.
Key env vars for CI:
SKIP_CLI_E2E=true— used inunitjob to skip CLI-dependent testsE2E_SKIP_BROWSER_TESTS=1— skip Playwright browser testsRUN_REASONING_TESTS,RUN_VISION_TESTS— gate expensive integration tests
Writing Tests
TypeScript (Vitest)
- Colocate: Place
*.test.tsnext to source files inapps/fouria/src/andpackages/moltlazy/tests/ - Strict mode: Tests use
strict: trueTypeScript - Mocking: Use vitest’s
vi.mock()for module-level mocking - Lifecycle:
beforeAll/afterAllfor setup/teardown;beforeEach/afterEachfor per-test state - Pattern: Arrange → Act → Assert (AAA)
- Coverage: ≥80% for new code (≥95% for API routes in
apps/fouria)
Elixir (ExUnit)
- Mirror structure: Test files mirror
lib/paths undertest/ - Database: Use
Ecto.Adapters.SQL.Sandboxfor concurrent DB tests - HTTP mocking: Use
Bypassfor external HTTP requests - Auth helpers: Use
@tag cf_authinconn_case.exto inject CF Access headers - Integration tests: Tag with
@tag :integration(excluded by default) - Fixtures: Define in
test/support/fixtures/ - Convention:
describe "function_name/arity"blocks; test names as strings
cctr (Corpus Contract Tests)
- Format: Plain text test files with
===(description) → script →---→ assertion sections - %require: First test case in a file with
%requirecascades failures to skip remaining tests - Variables: Capture dynamic output with
{{ name }}— used for tokens, URLs, etc. - Assertions: Use
whereclauses (* s contains "ready",* status == "200") - Fixtures: The
fixture/directory is mirrored to$CCTR_FIXTURE_DIR - Setup/teardown:
_setup.txtruns before the suite,_teardown.txtafter - Shared helpers:
apps/fouria/test/lib/e2e-helpers.shprovidese2e_wait_for_health(immediate, bounded readiness probe) ande2e_classify_ws_upgrade(WS-upgrade verdict), unit-tested inapps/fouria/src/e2e-helpers.test.ts.e2e_wait_for_healthis called by the serial (test/e2e) and agent (test/e2e-agent)_setup.txtcorpora and by the parallel driver’s top-level wait (test/e2e-parallel/run-parallel.sh).ab_gateway_ws_origin.txtsources the library only fore2e_classify_ws_upgradeand keeps its own health poll; the parallel driver’s generated group_setup.txtalso keeps its own grace-free bounded health poll. - Local mode guard: Tests that require cloud infrastructure use:
if [ "${E2E_LOCAL:-}" = "true" ]; then echo "SKIP: ..." >&2 exit 1 fi
Business Rules
- Location:
apps/fouria/test/business-rules/ - Format: cctr
.txtfiles - Purpose: Cross-cutting behavioral contracts (pricing, RBAC, tenant isolation, etc.)
- When to update: Any change affecting pricing, RBAC, tenant isolation, operational modes, cost tracking, dashboard API, agent visibility, or credits/plans
- Run:
cctr apps/fouria/test/business-rules/
Validation Checklist
Before submitting changes that modify test infrastructure or add new tests:
-
bun run typecheckpasses -
bun run lintpasses -
bun run testpasses (all JS/TS packages) -
For
apps/fouria:bun run test:coveragepasses -
For
apps/lerma:mix testpasses - New code has ≥80% unit test coverage (≥95% for API routes)
- If adding routes: add tests
-
If changing env vars: update
.dev.vars.example, ENV-VARIABLES.md, andapps/fouria/src/gateway/env.ts -
If changing business rules: update corresponding corpus file and run
cctr apps/fouria/test/business-rules/ -
For
apps/fourialocal E2E:bun run test:e2e:local:clipasses (cloud-only tests self-skip)