FourIA — Environment Variables Reference
On this page
- Variable Classification
- Dynamic Secrets (F4E1-Encrypted R2 Storage)
- AI Provider Configuration
- Authentication & Access Control
- Persistence & Storage
- Browser Automation (CDP)
- Container Rescue (/debug/rescue-container)
- Billing API (Lerma, server-side only)
- Lerma Provisioning API Keys
- Feature Flags
- Stripe Billing Integration
- Communication Schema (Dispatch Worker → User Worker)
- Dashboard Tenant Provisioning (apps/lerma)
- Legacy Channel Secrets (E2E Testing Only)
- Internal Variables (Auto-Set)
- Setting Variables
Complete reference of all environment variables used across FourIA Workers. Derived from apps/fouria/src/gateway/env.ts and apps/fouria/.dev.vars.example.
Variable Classification
FourIA uses three classifications for configuration values:
| Classification | Storage | Binding Type | Managed By | Examples |
|---|---|---|---|---|
| Instance Secret | CF encrypted secret_text | secret_text | lerma provisioner | Gateway token, CDP secret, AI keys, R2 |
| Variable | wrangler.jsonc / WFP plain_text | plain_text | Provisioner / IaC | DEV_MODE, URLs, account IDs, sandbox config |
| Dynamic Secret | F4E1-encrypted R2 object in BACKUP_BUCKET | — | End-user via integrations UI | Channel tokens, OAuth, custom integrations |
Instance Secrets
These are automatically generated and deployed by the lerma provisioner as secret_text bindings on each tenant User Worker. For the full inventory of required secrets, see the secrets manifest.
| Secret | Description |
|---|---|
MOLTBOT_GATEWAY_TOKEN | Auto-generated gateway token for CLI auth. Mapped to OPENCLAW_GATEWAY_TOKEN in container. |
CF_ACCESS_PLATFORM_AUD | Platform-level Access AUD (always present, from dispatch worker). |
CF_ACCESS_CLIENT_AUD | Per-tenant Access AUD (optional, provisioned by lerma). |
Secrets reference: See docs/schemas/secrets-manifest.schema.json for the canonical credential inventory.
Variables (Non-Secret Config)
Operational configuration, safe to store in wrangler.jsonc or .dev.vars. Deployed as plain_text bindings by the provisioner.
| Variable | Container | Required | Description |
|---|---|---|---|
DEV_MODE | Yes† | No | Skips CF Access auth + bypasses device pairing. Local dev only. |
OPENCLAW_DEV_MODE | Yes | No | Explicit override for the dev-agents flag. Normally derived from the instance’s own -dev hostname suffix (WORKER_URL), which enables the dev-only Fouria Builder/Q&A agents, client-doc materialization, the dev-only ROOT “view as” role preview, and the opencode-credits-check automation. Does not bypass Cloudflare Access auth. Falls back to DEV_MODE for local development. |
E2E_TEST_MODE | Yes‡ | No | Skips CF Access auth but keeps device pairing. For automated E2E. |
BILLABLE_EVENTS | No | No | Opt-in for billable-action emission on a dev instance (the billable-action Analytics Engine rows + BILLING_QUEUE enqueue lerma reconciles into credits). Billable production and the staging platform always emit and cannot be opted out (staging mirrors production so the billing pipeline is exercised before release; its rows land in the staging Analytics Engine dataset). A dev environment is silent by default and emits only with true — useful for exercising billing end-to-end from a dev instance. Emission only: the platform/environment cost-row dimensions and the container sleep policy are unchanged, so an opted-in dev instance still auto-sleeps (and staging keeps its 15m idle timeout). |
DEBUG_ROUTES | No | No | Enables /debug/* endpoints (ROOT role only). Off in production. lerma provisions dev instances with DEBUG_ROUTES=true by default; toggle per instance from the dashboard. |
DEMO_MODE | No | No | Enables demo-specific UI behaviors (read-only, sample data). |
SANDBOX_RUNTIME | No | No | Requested container runtime: legacy (default) or v1. It is not the authority for serving: on the trusted staging platform the runtime comes from the durable conversion intent, else from the fresh-v1 marker (sandbox-migration/<slug>/runtime.json, written by Lerma at provisioning). The flag only makes an unprovisioned tenant fail closed: a staging tenant with no intent and no marker resolves to pending (503 SANDBOX_V1_CONVERSION_PENDING) instead of falling back to legacy. See apps/fouria/docs/WORKER-DEVELOPMENT.md § “Provisioning an instance directly on Sandbox v1”. The container idle timeout is derived, not configured: staging 15m / dev 2h / production 6h (the Sandbox SDK 1.0 platform cap), from the instance hostname — there is no SANDBOX_SLEEP_AFTER binding. Only non-billable instances auto-sleep. Billable production keeps the container awake with the 6h platform maximum (setInactivityTimeout) renewed by a recurring keep-alive alarm on the SandboxV1 DO (resolveInactivityPolicy → keepAlive: true); it is stopped only by an explicit lerma freeze (POST /control → sandbox.destroy()). A stopped container is never woken by a snapshot. Waking a sleeping container is the ActivityScheduler DO alarm’s job. Lerma provisions this binding per instance from instances.sandbox_runtime (legacy by default; v1 only for a dev instance on the staging platform) and writes the marker as a hard provisioning step before the script deploy — see apps/lerma/README.md. |
WORKER_PLATFORM | No | Yes | Trusted, non-secret deployment platform (local, staging, production), provisioned by lerma from its own cf_environment. Sandbox v1 migration is allowed only when this is exactly staging (contract C1). Absent/empty/whitespace/production/dev/local/STAGING all refuse before any R2 write or DO/container addressing. It is never derived from the request Host, WORKER_URL, X-FourIA-* headers, DEV_MODE, or a slug prefix/suffix. |
SANDBOX_V1_CUTOVER | No | No | Per-tenant Sandbox v1 cutover intent (contract S1), default false. Off keeps the tenant on the legacy Sandbox class byte-identically; a staging-only migration flips it through the audited /control action. No production cutover is in scope. |
SANDBOX_V1_SNAPSHOT_INTERVAL_MS | No | No | Idle-snapshot cadence for a converted (SandboxV1) tenant, in milliseconds (0 disables it). The DO alarm commits a v1 record this often while the container is running, so the data-loss window is bounded even with no traffic/automations (SDK 1.0 has no eviction hook). Defaults to the configured BACKUP_INTERVAL_MINUTES, else 1h, and never exceeds half the instance’s inactivity timeout (staging 7.5m, dev 1h; production 3h). It never wakes a stopped container and skips when the recovery point is already newer. Worker-only. |
CONTAINER_INSTANCE_TYPE | Yes | Yes | Provisioned container size (lite | basic | standard-1..standard-4), deployed by lerma from the instance type chosen at provisioning. A durable_object Sandbox v1 container selects its size in ctx.container.start() on every boot and wake, so this must carry the provisioned value; absent/invalid falls back to standard-1 and logs sandbox.v1.instance_type.invalid. |
SANDBOX_INSTANCE_TIMEOUT_MS | No | No | Bound for container VM provisioning (default: 20000). Fail-fast detection of a degraded/off container. |
SANDBOX_PORT_TIMEOUT_MS | No | No | Bound for container port readiness (default: 90000). Keep generous — gateway cold start is slow. |
SANDBOX_POLL_INTERVAL_MS | No | No | Poll interval for container readiness (default: 300). |
SANDBOX_LOG_LEVEL | No | No | Container log level: debug, info, warn, error |
SANDBOX_LOG_FORMAT | No | No | Container log format: pretty or json |
WORKER_URL | Yes | No | Public Worker URL (used for CDP and OTel endpoint) |
WORKER_NAME | Yes | No | Worker name for config bucket key resolution |
CF_AI_GATEWAY_ACCOUNT_ID | Yes | Yes* | Cloudflare account ID for AI Gateway URL construction |
CF_AI_GATEWAY_GATEWAY_ID | Yes | Yes* | AI Gateway ID for URL construction |
AI_GATEWAY_PROXY | No | No | Opt-in ('true'). Routes tenant inference through the sandbox → Worker proxy (ai.internal): the container holds a per-instance derived token instead of the account credential, and every request carries trusted tenant_id/environment/platform metadata for per-client spend attribution on a shared gateway pool. Enable only when the tenant’s primary model resolves under cloudflare-unified-billing/*; while the official cloudflare-ai-gateway provider handles the turn it ignores the custom domain and the derived token is rejected. Lerma binds it per instance from instances.ai_gateway_proxy (instance page → AI proxy), including a binding-only re-deploy of already-running instances. |
CF_AI_GATEWAY_CUSTOM_DOMAIN | No | No | Optional operator opt-out. When set, the container talks to that custom domain directly with the real credential (no proxy, no tenant metadata). It must not be Access-protected: tenant containers carry no Cloudflare Access credentials. |
CF_ACCESS_TEAM_DOMAIN | No | Yes* | Cloudflare Access team domain (e.g., myteam.cloudflareaccess.com) |
CLOUDFLARE_ACCOUNT_ID | Yes | No | Cloudflare account ID. Read by the Sandbox SDK for R2 presigned URL signing (backup/restore). Legacy CF_ACCOUNT_ID is deprecated (still mapped into the container by buildEnvVars). |
BACKUP_BUCKET_NAME | Yes§ | No | R2 backup bucket name (default: fouria-data). Required by the Sandbox SDK presigned-URL backup path; mapped to R2_BUCKET_NAME in container. |
BACKUP_MAX_VERSIONS | No | No | Max squashfs snapshot versions to retain (default: 3) |
BACKUP_INTERVAL_MINUTES | No | No | Minimum scheduled-snapshot cadence (default: 5). The effective interval is max(this, 2 × the last snapshot's end-to-end duration), so a tenant whose snapshot outlasts the cadence is not snapshotted back-to-back (which starved the automation wake and held the R2 lease continuously). See resolveEffectiveIntervalMs in src/persistence.ts. |
BACKUP_ARCHIVE_EXCLUDES | No | No | Feature flag (default: off). 'true' passes SNAPSHOT_EXCLUDES (.npm, *.log, *.tmp, .cache, workspace/*node_modules*) to mksquashfs, shrinking the archive. Off by default because it is a fidelity trade: an archive restored without those paths no longer matches the filesystem a Tier 1 workspace snapshot restores, so the recovery tier stays a complete image until an operator opts in per instance. Worker-only. |
BACKUP_CHANGE_DETECTION | No | No | Change-aware gate (default: on). Skips a scheduled snapshot when the tenant’s state has not changed since the last committed one — a snapshot re-uploads the whole archive, so an unnecessary one dominates the whole backup cost. Only an explicit 'false'/'0'/'off' disables it. The scan fails closed to changed, so a broken scan costs an extra snapshot, never a missed one. Set on lerma; deployed to tenant Workers. |
BACKUP_CHANGE_WATCH | No | No | Feature flag (default: off). 'true' runs the per-tenant BackupWatch loop. Set on lerma, which also binds BACKUP_WATCH. |
BACKUP_CHANGE_CHECK_MINUTES | No | No | BackupWatch check cadence while armed (default: 2). Worker-only. |
BACKUP_CHANGE_DEBOUNCE_MS | No | No | Quiet window before a still-changing tenant is snapshotted, in ms (default: 60000). Worker-only. |
WORKSPACE_SNAPSHOT | No | No | Feature flag (default: off). 'true' captures a Tier 1 container snapshot immediately before idle eviction and commits its handle to workspace/snapshot.json. Requires a container application with scheduling_policy: "durable_object"; on the default policy the API is unsupported and the attempt is recorded as such. Worker-only. |
BACKUP_LOCAL_BUCKET | No | No | Set to true to store R2 backups in the Sandbox local binding bucket instead of remote R2 (needed for local dev without presigned-URL support). Explicit; falls back to DEV_MODE only when unset, so a real deployment — including the cloud E2E, which sets DEV_MODE=true merely to bypass Access — must set it to false or backups land somewhere the Worker cannot restore from. |
LEGACY_STAGE_TIMEOUT_MS | No | No | Bound (ms) for the legacy-state staging shell that runs before sandbox.createBackup; the stage is retried once on a transient timeout. Default: 600000 (10 min) |
ANALYTICS_DATASET | No | No | Analytics Engine dataset holding emitted billable-action rows, read back by GET /api/admin/costs/v2/summary for the dashboard “Usage & Billing” widget. Default: fourai_cost_metrics; the staging provisioner sets fourai_cost_metrics_staging. |
ANALYTICS_ENGINE_BASE_URL | No | No | Overrides the Analytics Engine SQL API base URL for cost-summary queries (default https://api.cloudflare.com/client/v4/accounts). Local billing-workflow E2E only — points the query at a fake AE. |
ANALYTICS_ENGINE_API_TOKEN | No | No | Optional Analytics Engine SQL token (Account Analytics:Read) for the dashboard read-back only. Emission needs no credential (the Worker writes with its ANALYTICS binding) and Lerma no longer provisions this token; set it as an instance secret if a tenant should read its own rows back. No CLOUDFLARE_API_TOKEN fallback (that alias lacks AE read). |
† DEV_MODE is a local-development switch (auth bypass). The dev-only agents surface is derived from a dev instance’s -dev hostname (or an explicit OPENCLAW_DEV_MODE override) without bypassing auth.
‡ Mapped to OPENCLAW_E2E_TEST_MODE in container.
§ Mapped to R2_BUCKET_NAME in container.
Mode behavior summary:
| Mode | CF Access Auth | Device Pairing | Debug Routes | Write API |
|---|---|---|---|---|
| Production | Required | Required | Disabled | RBAC |
DEV_MODE | Bypassed | Bypassed | Configurable | ROOT |
E2E_TEST_MODE | Bypassed | Required | Configurable | ROOT |
DEMO_MODE | Required | Required | Disabled | Read-only |
Dynamic Secrets (F4E1-Encrypted R2 Storage)
Channel tokens, tool OAuth credentials, and custom integration secrets are managed by end-users through the integrations dashboard. They are stored as F4E1-encrypted JSON objects in the BACKUP_BUCKET R2 bucket under the secrets/<customerSlug>/ prefix.
Secret categories: ai_provider, channel, integration
The encryption uses the shared BACKUP_ENCRYPTION_KEY (same master key used for backup snapshots) with per-object HKDF-derived AES-256-GCM keys (domain-separated via fouria-r2-secrets/v1). The master key never leaves the Worker and is never passed to the container.
Unlike the old vault, there is no unlock/lock mechanism: secrets are always accessible when the key is configured. The /lock endpoint is a no-op kept for client compatibility. The WebSocket API (/ws) has been removed (returns 410 Gone).
See
apps/fouria/src/services/secrets/R2SecretsStore.tsfor the full implementation.
AI Provider Configuration
At least one provider must be configured. Priority: CF AI Gateway > Anthropic > OpenAI.
CI-Only Secret Mapping
The CI workflow maps GitHub secrets (prefixed with E2E_) to their corresponding env variables. See .github/workflows/test.yml for the authoritative mapping.
| GitHub Secret | Maps To |
|---|---|
E2E_CLOUDFLARE_AI_GATEWAY_API_KEY | CLOUDFLARE_AI_GATEWAY_API_KEY |
E2E_CF_AI_GATEWAY_ACCOUNT_ID | CF_AI_GATEWAY_ACCOUNT_ID |
E2E_CF_AI_GATEWAY_GATEWAY_ID | CF_AI_GATEWAY_GATEWAY_ID |
E2E_TEARDOWN_API_TOKEN | CLOUDFLARE_TEARDOWN_API_TOKEN |
E2E_TELEGRAM_BOT_TOKEN | E2E_TELEGRAM_BOT_TOKEN |
There is no GitHub secret named
CLOUDFLARE_AI_GATEWAY_API_KEY. Staging and production Lerma deploys (.github/workflows/deploy-dashboard.yml) both readE2E_CLOUDFLARE_AI_GATEWAY_API_KEYand deploy it as theCLOUDFLARE_AI_GATEWAY_API_KEYruntime env. Referencing a non-existentsecrets.CLOUDFLARE_AI_GATEWAY_API_KEYexpands to empty and ships a dashboard whose tenant provisioning fails with:missing_ai_gateway_credential— the deploy workflow fails fast when the secret is empty.
E2E_TEARDOWN_API_TOKENis used only by the E2E teardown for destructive cleanup (wrangler delete+ R2 bucket empty/delete + container/Access/service-token deletes). It must carry Workers Scripts, Workers KV Storage, Workers R2 Storage, Cloudflare Containers, and Access (Apps + Service Tokens) Edit permissions. When unset, teardown falls back toCLOUDFLARE_API_TOKEN(DEPLOY_WORKER_API_TOKENin CI).
E2E_TELEGRAM_BOT_TOKENis an optional real Telegram bot token (from@BotFather) for the Telegram webhook-ingress E2E (q_telegram_webhook.txt). The suite self-skips when it is unset. It is never passed to the container or used in production.Local cloud E2E file.
bun run secrets:generateproduces the gitignoredapps/fouria/.dev.vars.e2ewith the mandatory cloud E2E vars —CLOUDFLARE_API_TOKEN(fromLERMA_API_TOKEN, since E2E provisions real Workers/Containers/R2/Access/KV resources),CLOUDFLARE_ACCOUNT_ID,WORKERS_SUBDOMAIN,CF_ACCESS_TEAM_DOMAIN,R2_ACCESS_KEY_ID,R2_SECRET_ACCESS_KEY— plus the AI Gateway trio and Tunnel token when set. Vars the registry does not track (CLOUDFLARE_TEARDOWN_API_TOKEN,SUPABASE_*, channel tokens, …) live in the file’s preserved manual section.Secrets reference: See docs/schemas/secrets-manifest.schema.json for the canonical credential inventory.
Runtime AI Credentials
| Variable | Class | Container | Description |
|---|---|---|---|
CLOUDFLARE_AI_GATEWAY_API_KEY | Platform Secret | Yes | Real AI-scoped Cloudflare credential supplied by Lerma. In proxy mode it stays in the Worker; the container receives a per-instance derived token (HMAC(gateway token, "fouria-ai-proxy/v1")) instead. |
CLOUDFLARE_API_TOKEN | Tenant Runtime Alias | Yes | Same derived token in proxy mode (the unified-billing plugin sends it as the bearer). It only authenticates against the Worker’s ai.internal handler, never against Cloudflare. With a custom-domain opt-out it is the real key. |
The unified-billing plugin always sends CLOUDFLARE_API_TOKEN as the bearer. With AI_GATEWAY_PROXY=true the Worker sets CF_AI_GATEWAY_CUSTOM_DOMAIN=ai.internal, so the plugin calls https://ai.internal/compat/..., which the sandbox intercepts and routes to src/sandbox-ai-bridge.ts. That handler authenticates the sandbox with the derived token, derives the tenant from the Worker env (never the request), adds cf-aig-metadata (tenant_id, environment, platform, plus the plugin’s openclaw_run_id), forces cf-aig-gateway-id from the trusted binding, and forwards to https://api.cloudflare.com/client/v4/accounts/{account}/ai/v1/... with the real platform credential.
Without the proxy the container keeps the real credential and calls Cloudflare directly; AI Gateway log rows then carry no tenant metadata, so Lerma reports them as unattributed and falls back to the tenant-scoped OTel ai-query estimate instead of assigning a pool total to a client.
The proxy is what makes per-client spend attributable on a shared gateway pool (the account allows 20 gateways) and keeps the account-scoped credential out of the container. Attribution metadata is a reporting and budget dimension — Cloudflare exposes it to Analytics metadataValue(key: ...) grouping and spend-limit rules — not an authorization boundary; the proxy is the boundary. Tenant containers carry no Cloudflare Access credentials: CF_ACCESS_TOKEN/CF_ACCESS_CLIENT_ID/CF_ACCESS_CLIENT_SECRET were removed, because an Access-protected custom domain requires a per-tenant service token that still cannot produce cf.user_id.
bun run secrets:validate proves this credential by running a 2-token inference
through the account’s gateway (gateway.ai.cloudflare.com/{account}/{gateway}/workers-ai),
so a wrong CF_AI_GATEWAY_GATEWAY_ID or CF_AI_GATEWAY_ACCOUNT_ID fails the
check too. Workers AI retires model ids on a schedule, and a retired id answers a
non-auth 4xx (e.g. required properties at '/' are 'model,fields', Model has been deprecated) — so that class of rejection is re-checked with a model-free
gateway read: the credential is only reported broken when the read fails as well,
and a stale pinned model (AI_GATEWAY_TEST_MODEL in scripts/validate-secrets.ts)
is surfaced as a warning to refresh, instead of red-lighting the secrets run.
* At least one provider must be configured.
Authentication & Access Control
| Variable | Class | Container | Required | Description |
|---|---|---|---|---|
MOLTBOT_GATEWAY_TOKEN | Instance Secret | Yes† | Yes | Gateway token for CLI auth. Auto-generated per tenant by lerma provisioner. |
CF_ACCESS_PLATFORM_AUD | Instance Secret | No | Yes* | Platform-level Access AUD (always present, from dispatch worker). |
CF_ACCESS_CLIENT_AUD | Instance Secret | No | No | Per-tenant Access AUD (optional, provisioned by lerma). |
CF_ACCESS_TEAM_DOMAIN | Variable | No | Yes* | Cloudflare Access team domain (e.g., myteam.cloudflareaccess.com). |
Secrets reference: See docs/schemas/secrets-manifest.schema.json for the canonical credential inventory.
* Required for production. Not required when DEV_MODE=true or E2E_TEST_MODE=true.
† Mapped to OPENCLAW_GATEWAY_TOKEN by buildEnvVars().
Persistence & Storage
| Variable | Class | Container | Required | Description |
|---|---|---|---|---|
CLOUDFLARE_ACCOUNT_ID | Variable | Yes | No | Cloudflare account ID. Read by the Sandbox SDK for R2 presigned URL signing (backup/restore). Legacy CF_ACCOUNT_ID is deprecated (still mapped into the container by buildEnvVars). |
BACKUP_BUCKET_NAME | Variable | Yes† | No | R2 backup bucket name (default: fouria-data). Required by the Sandbox SDK presigned-URL backup path alongside the BACKUP_BUCKET binding; mapped to R2_BUCKET_NAME in container. |
R2_ACCESS_KEY_ID | Secret | No‡ | Yes | Deprecated for the container. Worker binding only: required by the Sandbox SDK presigned-URL backup path. Resolved per-instance by the lerma provisioner (shared lerma credential or a generated per-instance token). No longer forwarded into the container — sandbox R2 access uses the outboundByHost bridges (releases.r2.internal / docs.r2.internal / leads.r2.internal / software.r2.internal). |
R2_SECRET_ACCESS_KEY | Secret | No‡ | Yes | R2 secret key, paired with R2_ACCESS_KEY_ID. Worker binding only (see above); not forwarded into the container. |
BACKUP_MAX_VERSIONS | Variable | No | No | Max squashfs snapshot versions to retain (default: 3) |
BACKUP_INTERVAL_MINUTES | Variable | No | No | Minimum scheduled-snapshot cadence (default: 5). The effective interval stretches to at least twice the last observed snapshot duration so a slow tenant cannot snapshot back-to-back. See resolveEffectiveIntervalMs in src/persistence.ts. |
SANDBOX_V1_SNAPSHOT_INTERVAL_MS | Variable | No | No | Idle-snapshot cadence for a converted (SandboxV1) tenant, in ms (0 disables). Defaults to the configured BACKUP_INTERVAL_MINUTES, else 1h, capped at half the inactivity timeout; never wakes a stopped container. See resolveSnapshotIntervalMs in src/sandbox-v1.ts. |
SANDBOX_V1_ZERO_EXIT_GRACE_MS | Variable | No | No | Grace window (ms) after start-openclaw.sh start exits 0 before the gateway port opens (the “lost the container startup lock” path). After it, a boot with no gateway process anywhere fails fast instead of waiting the full cold-start budget; 0 disables the fast-fail. Default: 15000. See resolveZeroExitGraceMs in src/sandbox-v1.ts. |
BACKUP_LOCAL_BUCKET | Variable | No | No | Set to true to store backups in the Sandbox local binding bucket (local dev only). Explicit; falls back to DEV_MODE when unset. Real deployments (including the cloud E2E) set it to false so backups land in the remote BACKUP_BUCKET. |
LEGACY_STAGE_TIMEOUT_MS | Variable | No | No | Bound (ms) for the legacy-state staging shell in createSnapshot; retried once on a transient timeout. Default: 600000 (10 min) |
BACKUP_ENCRYPTION_KEY | Secret | No | Recommended | Base64 32-byte AES-256 master key for client-side R2 backup encryption (F4E1). Worker-side only, never passed to the container. Generate: openssl rand -base64 32 |
BACKUP_ENCRYPTION_REQUIRED | Variable | No | No | Set to true to fail closed (skip backup + error) when BACKUP_ENCRYPTION_KEY is missing (default: false, legacy plaintext + warning) |
† Mapped to R2_BUCKET_NAME in container.
‡ Deprecated for the container: kept as a Worker binding for the Sandbox SDK presigned-URL backup path, never exposed inside the container. Sandbox R2 access goes through the outboundByHost bridges (releases.r2.internal / docs.r2.internal / leads.r2.internal / software.r2.internal).
Secrets reference: See docs/schemas/secrets-manifest.schema.json for the canonical credential inventory.
Browser Automation (CDP)
| Variable | Class | Container | Required | Description |
|---|---|---|---|---|
WORKER_URL | Variable | Yes | No | Public Worker URL (used for CDP and OTel endpoint) |
Secrets reference: See docs/schemas/secrets-manifest.schema.json for the canonical credential inventory.
Container Rescue (/debug/rescue-container)
Recovers a container stuck in Error mode (e.g. the fouria:unstable-<sha> image it references was deleted by the stale-image cleanup cron). The tool probes the container via the Sandbox SDK, re-points the application image to the expected tag through the Cloudflare Containers API (rollout), and destroys the stuck container so the platform provisions a fresh one. See apps/fouria/src/diagnostics.ts.
| Variable | Class | Container | Required | Description |
|---|---|---|---|---|
CLOUDFLARE_API_TOKEN | Runtime Secret | Yes | No | AI Gateway REST Bearer token on tenant workers. On ROOT/ops deployments the same binding may also be used by /debug/rescue-container when it has Containers:Edit. |
CONTAINER_APPLICATION_ID | Variable | No | No | Cloudflare Containers application id for this worker’s Sandbox container. Not provisioned by Lerma: set it manually on a ROOT/ops deployment that needs /debug/rescue-container (a tenant’s AI-scoped token lacks Containers:Edit). |
CONTAINER_IMAGE_TAG | Variable | No | No | Expected container image tag (e.g. unstable-<sha>); used to derive the repair target registry.cloudflare.com/<account>/fouria:<tag>. Set manually with CONTAINER_APPLICATION_ID. |
CONTAINER_IMAGE_NAME | Variable | No | No | Container image repo name for the tenant sandbox image (default: fouria). |
Secrets reference: See docs/schemas/secrets-manifest.schema.json for the canonical credential inventory.
| Variable | Class | Container | Required | Description |
|---|---|---|---|---|
| Variable | Class | Container | Required | Description |
| ———————————–– | ————— | ——— | –––– | ———————————————————————————————————————————————————————————————–– |
OTEL_SHARED_SECRET | Instance Secret | Yes | Yes† | Per-instance bearer for OTLP ingest authentication on the sandbox bridge, generated by Lerma at provisioning and stored on the instance row. Mandatory for the billing path. |
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT | Auto | Yes | Auto | Auto-set to http://otel.internal/v1/metrics (virtual host resolved by the sandbox bridge) |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT | Auto | Yes | Auto | Auto-set to http://otel.internal/v1/traces (virtual host resolved by the sandbox bridge) |
OTEL_EXPORTER_OTLP_PROTOCOL | Auto | Yes | Auto | Auto-set to http/json (never grpc/protobuf) |
OTEL_TRACES_ENABLED | Variable | Yes | Yes† | 'true' enables diagnostics.otel.traces (default: disabled). Lerma provisions it: true for a billable production/staging instance, false for a client’s non-billable dev instance. Without it the gateway emits no spans, so no billable action exists. |
OTEL_SERVICE_NAME | Auto | Yes | Auto | Auto-set to customer slug or worker name |
ANALYTICS_ENGINE_API_TOKEN | Instance Secret | No | No | Optional Account Analytics:Read token for the dashboard read-back only. Not required for emission (the Worker writes with its ANALYTICS binding) and no longer provisioned by Lerma; set it as an instance secret if a tenant should read its own rows back. Never given to the container. |
PROMOTION_SOURCE_BUCKET | Binding | No | Auto‡ | Read-only R2 binding on a production instance to its sibling dev bucket. The private promotion transport (stage on dev, pull on boot). Never forwarded to the container. |
† Mandatory for the billing proof only: the bridge rejects an unauthenticated
export (401) and no billable traces are produced when either the secret is
unbound or traces are not enabled.
‡ Provisioned automatically for environment: "production"; dev instances do not
get it. Both buckets stay separate — production only reads the sibling’s prefix.
The OTEL_SHARED_SECRET above authenticates the tenant Worker’s OTLP ingest and
is forwarded to the container as OTEL_EXPORTER_OTLP_HEADERS. When it is unset
the bridge rejects every export and no billable traces are produced. It is
generated per instance by Lerma at provisioning, stored on the instance row
(instances.otel_shared_secret), and deployed as that tenant’s secret_text
binding. It is never read from Lerma’s environment: a leaked value is scoped to
one tenant, and rotating it is a redeploy of that tenant (the upgrade path
preserves the stored value; a legacy instance with no stored value gets a fresh
one on its next upgrade).
ANALYTICS_ENGINE_API_TOKEN is not part of the emission path and is not
provisioned by Lerma. The Worker writes every billable row with its ANALYTICS
binding (no credential), and the platform reconciles credits from its own
Analytics Engine credentials. The token only enables the optional per-tenant
dashboard read-back; when it is absent the summary reports available: false and
the SQL API is never called (there is no CLOUDFLARE_API_TOKEN fallback — that
alias lacks Analytics read and only produced 403s).
The container never touches Analytics Engine. Emission is the Worker’s job: the container exports OTLP to the sandbox bridge (
http://otel.internal) and the Worker writes thebillable-actionrow with itsANALYTICSbinding — no credential involved — and the platform reconciles credits from its own Analytics Engine credentials. The optional dashboard read-back (a per-instanceANALYTICS_ENGINE_API_TOKEN) never leaves the Worker. No Analytics Engine credential is ever forwarded into the sandbox, and no agent, skill, or manifest entry should try to reach the AE SQL API.
Private transports (no public ingest or import route)
| Feature | Transport | Authority | Public route |
|---|---|---|---|
| OTLP | container → http://otel.internal/v1/{traces,metrics} → TenantSandbox.outboundByHost | per-tenant OTEL_SHARED_SECRET (defense in depth) | none |
| Promotion | dev writes promotion/staged/* to its own bucket; production reads via PROMOTION_SOURCE_BUCKET | R2 binding (tenant-scoped) | none |
/api/otel/* and /api/admin/releases/import no longer exist. The OTLP route was
Worker-public but Access still redirected it, so the container now exports to a
virtual host that the SDK resolves inside the Worker: the traffic never leaves
the sandbox ↔ Worker control plane and Access never sees it. Promotion no longer
crosses the internet either: the dev instance stages the bundle in its own
bucket, and the production instance pulls it on boot (or on an explicit
POST /api/admin/releases/apply) under the shared persistence lease, skipping
while parked or while a maintenance window/restore is in flight.
Billing API (Lerma, server-side only)
These are Lerma-dashboard environment variables — never deployed to tenant workers.
| Variable | Class | Required | Description |
|---|---|---|---|
BILLING_INGEST_TOKEN | Runtime Secret | No | Bearer token guarding the internal billing API: L2 ingest (POST /api/credits/v1/ingest) and L5 reconcile/usage (POST /api/credits/v1/reconcile, GET /api/credits/v1/usage/:id). Unset → every request is rejected (401). |
ANALYTICS_DATASET | Variable | No | Analytics Engine dataset holding billable rows (fourai_cost_metrics default; fourai_cost_metrics_staging when CF_ENVIRONMENT=staging). |
ANALYTICS_ENGINE_BASE_URL | Variable | No | Overrides the Analytics Engine SQL API base URL for Lerma.BillingReconcile/Lerma.Analytics.CFAnalytics. Local billing-workflow E2E only — points them at the local fake AE (apps/lerma/test/e2e_billing_local/). Unset everywhere else. |
BILLING_INGEST_TOKEN permissions: none. It is not a Cloudflare API token and needs no Cloudflare scopes — it is an opaque shared bearer secret. Generate it with openssl rand -hex 32 and give the same value to every caller of the internal billing API (the billing queue consumer, operator tooling, the staging billing-trace e2e). The endpoints compare it in constant time and fail closed (401) when it is unset. Rotate by updating the value in the callers and in Lerma’s environment; no scope changes are involved.
Lerma Provisioning API Keys
Lerma supports both a platform-wide compatibility key and per-application keys:
| Name | Storage | Scope | Description |
|---|---|---|---|
PROVISIONING_API_KEY | Lerma runtime secret only | Platform-wide | Optional global bearer for /api/provisioning/* and /api/ai-gateways/*. When unset, global-only API operations fail closed (401). Not a GitHub repository secret: it is never synced by secrets:generate --github. |
STAGING_PROVISIONING_API_KEY | GitHub Actions secret only | Staging Fouria application | Used by .github/workflows/promote-staging.yml to upgrade the staging client tenant. It is not injected into Lerma’s runtime. |
Per-application keys are generated, rotated, and revoked by ROOT users on
/dashboard/applications/:id. Lerma stores only the SHA-256 digest in the
application row and reveals a newly generated key once. Copy the staging
Fouria application’s value to the GitHub secret STAGING_PROVISIONING_API_KEY;
never reuse it in production. Calls using an application key must send
X-FourIA-Application: <application-slug>, and Lerma filters the operation to
that application’s instances. The platform-wide key, when configured, remains a
full-scope compatibility credential.
Feature Flags
| Variable | Class | Container | Required | Description |
|---|---|---|---|---|
MOLTLAZY_FEATURE_FLAGS | Auto | Yes | Auto | JSON feature flags for moltlazy config generation (auto-generated). Includes the softwareProvisioning key (Worker-side: reconcile the declared software manifest at boot). |
WORKER_NAME | Variable | Yes | No | Worker name for config bucket key resolution. |
STARTUP_LOG_VERBOSITY | Variable | Yes | No | Clean boot-log verbosity for the loading page: steps | detailed (default) | verbose. |
OPENCODE_API_KEY | Secret | Yes | Dev | Platform OpenCode management credential. Lerma mints a restricted (Models: Request) key per client and binds that key to the client’s dev instances; the management key needs This account's API keys: Write. Never used for API billing. |
MOLTLAZY_AGENTS | Variable | Yes | Paso4 | Comma-separated premade agent roster to inject (moltlazy MOLTLAZY_AGENTS). Empty/unset means the platform default (all non-opt-in agents). |
MOLTLAZY_PASO4_AGENTS | Variable | Yes | Paso4 | true makes the Paso4-exclusive agent registry eligible (the opt-in paso4-dev agent). Set by lerma only on scope: "paso4" instances. |
GITHUB_TOKEN | Secret | Yes | Paso4 | GitHub token for the paso4-dev agent’s gh CLI. Deployed by the provisioner to Paso4-exclusive instances only; never present on client tenants. |
PASO4_TELEGRAM_GROUP_ID | Variable | Yes | Dev | Paso4 general Telegram group id for platform alerts (global lerma env; default -5442477312). |
TELEGRAM_INGRESS_HOST | Variable | No | No | Public Telegram webhook ingress host (hooks.fouria.io / hooks.dev.fouria.io). The Dispatch Worker matches it and routes to the tenant Worker without an Access JWT. |
TELEGRAM_INGRESS_QUEUE | Queue | No | Yes | Cloudflare Queue (fouria-telegram-ingress) buffering Telegram updates that arrive while the container is cold; the consumer wakes/heals and forwards with at-least-once delivery. |
TELEGRAM_INGRESS | DO | No | Yes | Per-tenant TelegramIngress Durable Object owning the fixed-window rate limiter for inbound webhook updates. |
Telegram webhook ingress (#364). Managed tenants switch Telegram to webhook mode. The webhook secret + bot token live in a Worker-readable F4E1 R2 registry (
telegram-ingress/<slug>/<account>); moltlazy writeschannels.telegram.accounts.<account>.webhook*from theTELEGRAM_WEBHOOK_MANIFESTenv var the Worker injects. The Dispatch Worker addsTELEGRAM_INGRESS_HOST(its own ingress routing) and the tenant Worker adds theTELEGRAM_INGRESSDO +TELEGRAM_INGRESS_QUEUEqueue bindings.
Stripe Billing Integration
The tenant User Worker has no Stripe credential binding and no Stripe checkout/portal route. Stripe billing runs in Lerma; the admin UI shows the plan, limits, and usage it receives from the Dispatch Worker headers, but plan changes and payment methods are managed in the customer dashboard.
| Variable | Class | Container | Required | Description |
|---|---|---|---|---|
BILLING_QUEUE | Queue binding | No | Yes | Shared Cloudflare Queue producer for canonical billable actions to Lerma (fouria-billing / fouria-billing-staging). Created/adopted by Lerma at provisioning. |
TASKS | Queue binding | No | No | Per-tenant task buffer (fouria-tasks-<slug>[-dev]). The tenant Worker is its own consumer; Lerma creates the queue and registers the script as consumer. |
Billing data (plan tier, status, seats, credits, Stripe customer ID) is sourced from the Supabase database via the Dispatch Worker. The Worker does not persist billing state in Durable Objects.
Communication Schema (Dispatch Worker → User Worker)
Billing metadata is passed from the Dispatch Worker to the User Worker via one of three resolution methods (in priority order):
| Priority | Method | Header / Source | Format |
|---|---|---|---|
| 1 (preferred) | Cloudflare for SaaS custom hostname metadata | request.cf.hostMetadata | JSON object — set via PATCH /zones/:id/custom_hostnames/:id with custom_metadata |
| 2 | Single metadata header | X-FourIA-Metadata | Base64url-encoded JSON of BillingMetadata |
| 3 (legacy) | Individual headers | X-FourIA-Plan, X-FourIA-SeatsUsed, etc. | String values (deprecated) |
Preferred approach (1): Set custom_metadata on each customer’s custom hostname via the Cloudflare API. The Dispatch Worker reads request.cf.hostMetadata and the User Worker reads it when trusted_workers: true is set on the dispatch namespace.
Fallback approach (2): The Dispatch Worker encodes the BillingMetadata object as base64url JSON and sets it as the X-FourIA-Metadata header when forwarding the request.
BillingMetadata Schema
interface BillingMetadata {
plan_tier: string; // "free" | "basic" | "pro" | "enterprise"
plan_name: string; // Human-readable name
billing_status: string; // "active" | "past_due" | "canceled" | "trialing" | "incomplete"
seats_used: number; // Current seat count
seats_max: number; // Plan seat limit
credits_used: number; // Credits consumed this period
credits_max: number; // Credit limit (0 = unlimited/pay-as-you-go)
stripe_customer_id: string | null; // Stripe customer for payments
period_end: string | null; // ISO 8601 — current billing period end
trial_end: string | null; // ISO 8601 — trial end (if trialing)
}RBAC Role Resolution
The user’s RBAC role is forwarded by the Dispatch Worker via the X-FourIA-Role header (one of ROOT, ADMIN, BASE_USER), when the Dispatch Worker can resolve it:
- Paso4 ROOT users are always
ROOT. A user qualifies when their JWTaudis the platform AUD (the*.fouria.iowildcard Access application) or their email is a member of the sharedPaso4 Root UsersAccess group. The group check matters because per-instance Access applications include the ROOT group policy (so ROOT operators are not locked out at{slug}.fouria.io); Cloudflare matches the most-specific application, so those tokens carry the per-instance AUD rather than the platform AUD. In staging this is the only permitted audience/identity. - Client users (per-instance Access application AUDs) get their role only when
client_users.status = validatedandclient_user_instanceslinks them to the requested instance. Pending, rejected, production-only, or dev-only assignments do not cross environment boundaries.
The User Worker’s resolveCustomerRole resolves the role in priority order:
- JWT
audmatchesCF_ACCESS_PLATFORM_AUD→ROOT X-FourIA-Roleheader (from the Dispatch Worker)- JWT
custom.roleclaim (IdP-provided) LOCAL_ROLE_MAP(local dev fallback)
Dashboard Tenant Provisioning (apps/lerma)
Variables required by the Phoenix dashboard to create and decommission tenant User Workers.
CF Access Local Development
Variables for local CF Access integration via cloudflared tunnel. See apps/lerma/bin/setup-cf-access-local.sh for one-time setup.
| Variable | Class | Required | Description |
|---|---|---|---|
CF_ACCESS_TEAM_DOMAIN | Variable | Yes* | Cloudflare Access team domain (e.g., paso4). Used by lerma’s dashboard auth and deployed to tenant workers. |
CF_ACCESS_AUD | Variable | Yes* | Dashboard’s own Access AUD (JWT audience for lerma’s dashboard). Populated by the deploy flow from CF_ACCESS_LERMA_AUD/STAGING_CF_ACCESS_LERMA_AUD. |
CF_ACCESS_PLATFORM_AUD | Variable | Yes* | Platform-level Access AUD (Paso4 Root app). Deployed to every tenant worker as the CF_ACCESS_PLATFORM_AUD secret so its auth middleware can validate JWTs forwarded by the Dispatch Worker. |
CF_ACCESS_SERVICE_TOKEN_ID | Instance Secret | No | CF Access service token Client ID for local CI/CD (non-interactive). |
CF_ACCESS_SERVICE_TOKEN_SECRET | Instance Secret | No | CF Access service token Client Secret for local CI/CD. |
CLOUDFLARED_TUNNEL_TOKEN | Instance Secret | No | Tunnel token for lerma-local (routes to port 4001). |
SKIP_AUTH | Variable | No | Set to "true" to bypass CF Access JWT verification (escape hatch). |
Lerma staging healthcheck tunnel
The staging Hetzner host runs a remotely managed Cloudflare Tunnel sidecar at
lerma-health-dev.paso4.io. Its ingress forwards only the minimal
GET /health response (ok) to Lerma; all other paths return 404. The existing
dashboard hostname and Access policies are unchanged. The tunnel does not replace
the existing staging A record or close its current origin ports.
| Variable | Class | Required | Description |
|---|---|---|---|
TF_VAR_healthcheck_tunnel_api_token | Local IaC Secret | Apply | Dedicated Cloudflare token used only by the staging OpenTofu apply; requires Account Tunnel:Edit and paso4.io Zone DNS:Edit. Do not reuse LERMA_API_TOKEN. |
LERMA_STAGING_HEALTHCHECK_TUNNEL_TOKEN | GitHub Secret | Staging | Remote-managed Tunnel run token passed through Ansible to the staging-only cloudflared sidecar. It is not passed to the Lerma Phoenix app. |
Invitation Email
Lerma delivers invitation confirmation emails through Cloudflare Email Service over authenticated SMTP submission (smtp.mx.cloudflare.net:465, implicit TLS). It reuses the existing provisioning credential as the SMTP password, so no additional secret is required.
| Variable | Class | Required | Description |
|---|---|---|---|
LERMA_API_TOKEN | Secret | Yes | Cloudflare API token. Must include the Email Sending: Edit permission; used as the SMTP password (username is the literal api_token). The From domain must be onboarded for Email Sending. |
Local development continues to use Swoosh’s local mailbox adapter; tests use the Swoosh test adapter.
Provisioning Configuration
| Variable | Class | Required | Description |
|---|---|---|---|
CLOUDFLARE_API_TOKEN | ENV only | Yes | API token with Workers:Edit, R2:Edit, Containers:Edit, and D1 edit permissions. Containers:Edit is required — the dashboard generates short-lived registry credentials (POST /containers/registries/{host}/credentials) to list versions, and uses R2:Edit to fetch the worker bundle from the fouria-bundles R2 bucket at provisioning time. Without it the calls fail with 403 Authentication error. |
CLOUDFLARE_ACCOUNT_ID | Variable | Yes | Paso4 Cloudflare account ID for provisioning calls. Must be set or the registry-health check reports it missing and registry-credentials calls fail with Could not route to /accounts/containers/registries/... — object identifier is invalid (the account ID is interpolated into the request path). On the fouria User Worker it is a plain var: the Sandbox SDK reads it for R2 presigned URL signing (backup/restore) — missing it throws InvalidBackupConfigError (Backup requires R2 presigned URL credentials). The dashboard health check also validates that CLOUDFLARE_API_TOKEN grants the required permission groups (Workers Scripts Write, DNS Write, Access: Apps and Policies Write, Workers Containers Write), flagging a missing one as “Limited access”. Legacy CF_ACCOUNT_ID is deprecated. |
CONTAINER_REGISTRY_HOST | Variable | No | Cloudflare managed container registry host (default: registry.cloudflare.com). Mirrors CLOUDFLARE_CONTAINER_REGISTRY in wrangler. |
CONTAINER_IMAGE_NAME | Variable | No | Container image repo name for the tenant sandbox image (default: fouria). |
CONTAINER_BUNDLE_R2_BUCKET | Variable | No | R2 bucket that publishes the worker bundle fouria.mjs (default: fouria-bundles). CI uploads the bundle to <version>/fouria.mjs and the dashboard fetches it from R2 at provisioning time. |
FOURIA_DISPATCH_NAMESPACE | Variable | No | Workers for Platforms dispatch namespace name (default: fouria-dispatch). |
FOURIA_R2_PUBLIC_ENDPOINT | Variable | No | R2 public endpoint template, e.g. https://pub-{bucket_id}.r2.dev. |
ANALYTICS_DATASET | Variable | No | Analytics Engine dataset name (default: fourai_cost_metrics, staging: fourai_cost_metrics_staging). |
CONTAINER_IMAGE | Variable | No | Container image reference for the tenant’s container application (linked to the Sandbox DO namespace). Example: registry.cloudflare.com/<account_id>/fouria:latest. When unset, lerma derives it from the published bundle version as registry.cloudflare.com/<account_id>/fouria:<image_tag>. |
CONTAINER_MAX_INSTANCES | Variable | No | Max container instances for tenant container apps (default: 3). |
DEV_INSTANCE_SSH_PUBLIC_KEY | Secret | No | Optional single-line ssh-ed25519 public key Lerma applies to Cloudflare Container authorized_keys only for dev instances. It is never a Worker/container environment binding; keep the private key on the operator workstation. Access uses wrangler containers ssh (Cloudflare-authenticated; no public SSH port). Existing dev apps receive key changes on their next Lerma redeploy. |
WORKER_OBSERVABILITY_ENABLED | Variable | No | Enable Workers Observability (logs + traces) on deployed tenant User Workers (default: true). Mirrors apps/fouria/wrangler.jsonc. |
WORKER_OBSERVABILITY_SAMPLING_RATE | Variable | No | Head sampling rate for traces on tenant User Workers, 0..1 (default: 1). |
WORKER_OBSERVABILITY_LOGS | Variable | No | Enable logs on tenant User Workers (default: true). |
WORKER_OBSERVABILITY_INVOCATION_LOGS | Variable | No | Enable invocation logs on tenant User Workers (default: true). |
WORKER_OBSERVABILITY_TRACES | Variable | No | Enable traces on tenant User Workers (default: true). |
WORKER_OBSERVABILITY_LOGS_DESTINATIONS | Variable | No | Comma-separated account-level Cloudflare Observability Logs destination names (Observability → Destinations), referenced by name in the worker upload metadata. Empty → no export destination. |
WORKER_OBSERVABILITY_TRACES_DESTINATIONS | Variable | No | Comma-separated account-level Cloudflare Observability Traces destination names. Empty → no export destination. |
WORKER_OBSERVABILITY_LOGS_SAMPLING_RATE | Variable | No | Logs head sampling rate, 0..1. Falls back to WORKER_OBSERVABILITY_SAMPLING_RATE. Use 0.1 for a 10% pilot. |
WORKER_OBSERVABILITY_TRACES_SAMPLING_RATE | Variable | No | Traces head sampling rate, 0..1. Falls back to WORKER_OBSERVABILITY_SAMPLING_RATE. The uptime widget/API report this rate and treat “no events” as unknown, never downtime. |
WORKER_OBSERVABILITY_LOGS_PERSIST | Variable | No | Keep the Cloudflare-side copy when an export destination is configured (default: false — Lerma is the destination of record). |
WORKER_OBSERVABILITY_TRACES_PERSIST | Variable | No | Keep the Cloudflare-side trace copy when an export destination is configured (default: false). |
CONTAINER_OBSERVABILITY_LOGS | Variable | No | Enable log capture on the tenant’s container application (default: true). |
CONTAINER_REGIONS | Variable | No | Container application placement regions (deployment zone), comma-separated (default: WEUR). Valid values: ENAM, WNAM, EEUR, WEUR, APAC, SAM, ME, OC, AFR. |
R2_TOKEN_STRATEGY | Variable | No | How lerma resolves the mandatory per-tenant R2 S3 backup credential: auto (default), shared, or per_instance. auto uses R2_TOKEN_MAX_PER_ACCOUNT: a known limit <= 250 selects shared, > 250 or unknown selects per_instance. per_instance mints a scoped R2 token per instance at provisioning. See Lerma.Provisioning.R2Credentials. |
R2_TOKEN_MAX_PER_ACCOUNT | Variable | No | Account API-token limit used by :auto (default: unknown → :per_instance). Cloudflare exposes no endpoint for the maximum, so pin a known value here if desired. |
R2_ACCESS_KEY_ID | Secret | Cond. | Lerma-level shared R2 S3 access key, required when the effective strategy is shared. Kept as a Worker binding for the SDK presigned-URL backup path (never forwarded to the container — sandbox R2 access uses the outboundByHost bridges). Generate with mix lerma.r2_token --all-buckets. |
R2_SECRET_ACCESS_KEY | Secret | Cond. | Lerma-level shared R2 S3 secret key, paired with R2_ACCESS_KEY_ID. Required when the effective strategy is shared; provisioning fails with :missing_shared_r2_credentials otherwise. Worker binding only (see above). |
Observability destination ingestion (Lerma)
Cloudflare account-level Observability → Destinations push Workers Logs and Workers Traces to Lerma over OTLP/HTTP JSON. This is a push stream: it adds no per-tenant polling, does not replace HealthMonitor/billing, and never wakes a sleeping container. HealthMonitor continues to drive instances.status/clients.status; destination telemetry only supplies diagnostics and uptime statistics.
| Variable | Class | Required | Description |
|---|---|---|---|
OTEL_INGEST_TOKEN | Runtime Secret | No | Shared bearer secret configured as the destination’s Authorization: Bearer … header. The ingest endpoints compare it in constant time and fail closed (401) when unset. Generate with openssl rand -hex 32. |
OTEL_INGEST_MAX_EVENTS | Variable | No | Maximum sanitized rows accepted per ingest request (default 1000). The remainder is reported as skipped. |
OTEL_INGEST_RETENTION_DAYS | Variable | No | Retention for stored destination telemetry (default 90, matching PRIVACY.md). |
OTEL_INGEST_PRUNE_INTERVAL_MS | Variable | No | Retention sweep interval (default 86400000 / 24 h). A first sweep runs one minute after boot, then every interval. |
Destination names per environment:
| Environment | Logs destination | Traces destination | Lerma OTLP base |
|---|---|---|---|
| staging | lerma-worker-logs | lerma-worker-traces | https://lerma-dev.paso4.io/api/ingest/otel/v1 |
| production | lerma-worker-logs-prod | lerma-worker-traces-prod | https://lerma.paso4.io/api/ingest/otel/v1 |
apps/fouria/wrangler.jsonc references the staging names in env.staging and
the production names in env.production; Lerma’s CD passes the same names to
each environment’s tenant-worker metadata. The account-level destinations must
exist before a Worker that references them is deployed.
-
In Observability → Destinations, create the pair for the environment you are rolling out (staging first), pointing at that environment’s Lerma host. Add the custom header
Authorization: Bearer <OTEL_INGEST_TOKEN>to both. The API equivalent is:# staging values shown; for production use LERMA_HOST=lerma.paso4.io and # the `-prod` destination names. LERMA_HOST=lerma-dev.paso4.io LOGS_NAME=lerma-worker-logs TRACES_NAME=lerma-worker-traces for pair in "$LOGS_NAME opentelemetry-logs logs" "$TRACES_NAME opentelemetry-traces traces"; do set -- $pair curl -sS "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/workers/observability/destinations" \ -H "Authorization: Bearer $CF_OBSERVABILITY_TOKEN" -H 'Content-Type: application/json' \ -d "{\"name\":\"$1\",\"enabled\":true,\"configuration\":{\"type\":\"logpush\",\"logpushDataset\":\"$2\",\"url\":\"https://$LERMA_HOST/api/ingest/otel/v1/$3\",\"headers\":{\"Authorization\":\"Bearer $OTEL_INGEST_TOKEN\"}}}" doneUse a credential with Workers Observability Write (the standard
LERMA_API_TOKENmay need that permission added). Cloudflare’s destination preflight needs a live, decompressing receiver, so create the staging pair after staging Lerma is on the gzip-aware release, and the production pair after production Lerma is.The dashboard CD runs
mix lerma.observability.destinations --environment <env> --warn-onlybefore each deploy (staging and production). It lists the destinations, creates missing ones, and PATCHes existing ones so a rotatedOTEL_INGEST_TOKENpropagates. It prefersCF_OBSERVABILITY_TOKENand falls back toLERMA_API_TOKEN; either must carry Workers Observability Write. With--warn-only, a missing permission logs a warning instead of failing the deploy, so add that permission to the token (or set a dedicatedCF_OBSERVABILITY_TOKEN) to make the rollout fully self-healing. -
Set
OTEL_INGEST_TOKENon that Lerma deployment and setWORKER_OBSERVABILITY_LOGS_DESTINATIONS/WORKER_OBSERVABILITY_TRACES_DESTINATIONSto the environment’s two names. The CD does this automatically (staging: un-suffixed names; production:-prodnames). -
Redeploy the tenant workers so their upload metadata references the destinations (new tenants inherit them automatically). Existing tenants pick them up on their next Lerma upgrade/redeploy.
-
Confirm delivery on the destination status (Last: n minutes ago) and query
GET /api/analytics/v1/uptime/tenant-<slug>?window=24h.
The ingest endpoints accept OTLP/HTTP JSON only (Cloudflare gzips the body while keeping Content-Type: application/json, so LermaWeb.Plugs.OtlpBodyReader decompresses it before parsing) and answer 202 with {accepted, skipped} so Cloudflare does not retry-loop; binary protobuf is rejected with 415 and a hint. Ingestion is idempotent — the destinations deliver at least once, and a replayed batch adds no rows. Only allowlisted metadata is stored: log message text, stack traces, URLs/query strings, user agents, request/response bodies and DB query text are dropped before persistence.
Health Monitoring
Lerma’s HealthMonitor periodically probes each tenant’s public GET /api/health endpoint and keeps clients.status and instances.status accurate. /api/health is the unified health endpoint (returns startup/readiness/liveness probes plus the canonical status and processId); GET /api/status is a legacy alias over the same snapshot. Both are exposed by every User Worker without auth and are made publicly reachable via path-scoped Cloudflare Access applications (*.fouria.io/api/health, *.fouria.io/api/status, see apps/fouria/iac/access.tf). Probing also cold-starts asleep containers (ensureGateway), so a stopped VM recovers on the next check — pass ?passive=1 (used by ROOT diagnostics) to probe without warming.
Each instance is probed at its own environment hostname (acme.fouria.io for production, acme-dev.fouria.io for dev).
up means the gateway is running. A probe is up only when the snapshot reports status: "running". A 200 status: "sleeping" (an idle container whose gateway is not running) is not up: it records a down transition for the uptime widget and marks the instance sleeping (alive, but not running). A 503 (stopped) or a transport failure records a down transition but leaves the instance status untouched — health is observability-only, so a probe miss (no traffic, a slow cold start, or an Access challenge) never flips a tenant to error or takes it out of service. A sleeping/stopped probe also does not recover a client left in error; only a genuine running probe does.
An exact-host per-tenant Access app shadows the wildcard health app, so the provisioner (Lerma.Cloudflare.AccessPolicy) also creates a per-tenant health application ({slug}.fouria.io/api/health) that carries no role requirement (any authenticated user) and allows the health-monitor service token. The monitor authenticates with HEALTH_MONITOR_ACCESS_CLIENT_ID / HEALTH_MONITOR_ACCESS_CLIENT_SECRET.
The dashboard CD (.github/workflows/deploy-dashboard.yml) wires these credentials for both staging and production: it passes HEALTH_MONITOR_ACCESS_CLIENT_ID, HEALTH_MONITOR_ACCESS_CLIENT_SECRET, and CF_ACCESS_HEALTH_MONITOR_TOKEN_ID to Ansible as health_monitor_access_client_id / health_monitor_access_client_secret / cf_access_health_monitor_token_id. When they are omitted, the deployed Lerma probes go out unauthenticated and every Access-protected tenant answers 302 (an Access login redirect), which the probes surface as an Access challenge.
| Variable | Class | Required | Description |
|---|---|---|---|
HEALTH_MONITOR_ENABLED | Variable | No | Enable the periodic health monitor (default: true). Always disabled in test env. |
HEALTH_MONITOR_INTERVAL_MS | Variable | No | Poll interval in milliseconds (default: 120000). |
HEALTH_MONITOR_TIMEOUT_MS | Variable | No | Per-probe HTTP timeout in milliseconds (default: 15000). |
HEALTH_MONITOR_PLATFORM_DOMAIN | Variable | No | Domain the tenant workers are served under (default: fouria.io). |
HEALTH_MONITOR_SLUG_PREFIX | Variable | No | Hostname prefix for staging tenants (default: "", staging auto-set to staging-). |
HEALTH_MONITOR_ACCESS_CLIENT_ID | Secret | No | Cloudflare Access service-token client ID sent on each probe (cf-access-client-id). Required for probing Access-protected tenants. |
HEALTH_MONITOR_ACCESS_CLIENT_SECRET | Secret | No | Cloudflare Access service-token client secret (cf-access-client-secret). |
CF_ACCESS_HEALTH_MONITOR_TOKEN_ID | Variable | No | Pins the service-token id granted access to each per-tenant health app. When empty, AccessPolicy resolves the FourIA Health Monitor token by name. |
INTERNAL_DISPATCH_INVALIDATE_TOKEN | Secret | Yes | Bearer Lerma sends on the Dispatch Worker’s internal routes (/_control/*, /_invalidate/*), and the value the Worker compares (digest, constant time) for production. The Worker has no hardcoded fallback and fails closed (403) when unset or mismatched, so cache invalidation, freeze/unfreeze, and the pre-replacement checkpoint cannot run. Production only; staging uses STAGING_INTERNAL_DISPATCH_INVALIDATE_TOKEN. |
STAGING_INTERNAL_DISPATCH_INVALIDATE_TOKEN | Secret | Yes (staging) | Staging-only counterpart uploaded to fouria-dispatch-staging and passed to staging Lerma. Never reuse the production value. Roll out the Dispatch Worker before Lerma in each environment. |
Dashboard connectivity checks (StartupHealth)
Lerma’s StartupHealth runs external connectivity checks (Cloudflare API token validity/permissions and the container image registry) and caches the result in ETS. The result drives the dashboard connectivity banner (/dashboard) and the /health/liveness + /health/readiness probes. Checks run at startup and then on a recurring interval so a transient boot-time failure (e.g. a registry request timing out) clears itself once connectivity is restored: the banner updates live over PubSub and the readiness probe recovers, without restarting the app.
| Variable | Class | Required | Description |
|---|---|---|---|
STARTUP_HEALTH_INTERVAL_MS | Variable | No | Re-check interval in milliseconds (default: 300000). Set to 0 to run the connectivity checks only once at startup (startup-only behaviour). |
Provisioning health probe (HealthPoller)
Health verification is asynchronous and post-deployment: once a provisioning
run (or upgrade) is finalized, the provisioner schedules a background task that
runs the non-passive /api/health probe (HealthPoller). It is not a
deployment step, so a slow container cold start (the Worker’s boot budget is
STARTUP_TIMEOUT_MS, 300s) never blocks, times out, or rolls back a deployment.
A failed probe is observability-only: the downtime is recorded for the
instance uptime widget, but the instance status is left untouched (a probe miss
is not an outage). The poll succeeds (:ok) only when the snapshot reports
status: "running" — a 200 status: "sleeping" is not ready and keeps
retrying until the deadline. The background check is disabled with
skip_health_poll: true.
The probe targets the instance’s environment hostname — the client slug plus
the -dev environment suffix for a dev instance, then the HEALTH_MONITOR_SLUG_PREFIX
platform prefix ({prefix}{slug}[-dev].fouria.io). The platform suffix (-staging)
is deliberately not part of the hostname; hostname construction must never
reuse the routable_slug used for Cloudflare resource names (R2/AI Gateway),
or the platform axis is applied twice.
Upgrade rollback never deletes the existing worker script / container application, so a transient failure cannot destroy a running tenant. Fresh provisions still clean up the resources created for the new tenant.
| Variable | Class | Required | Description |
|---|---|---|---|
HEALTH_PROBE_TIMEOUT_MS | Variable | No | Provisioning health-probe deadline in milliseconds (default: 360000). Must exceed the Worker cold-start budget (300s). |
HEALTH_PROBE_BACKOFF_MS | Variable | No | Delay between provisioning health-probe attempts in milliseconds (default: 3000). |
How Secrets Are Deployed
The provisioner deploys all instance secrets as secret_text bindings directly on the User Worker. These are encrypted at rest by Cloudflare and available at runtime as plain string values (not Secrets Store objects).
The provisioner no longer binds a Cloudflare Secrets Store placeholder on tenant Workers. Dynamic secrets use the F4E1-encrypted R2 store; instance secrets are never placed in a Secrets Store.
# Per-tenant secrets (generated by lerma provisioner)
%{type: "secret_text", name: "MOLTBOT_GATEWAY_TOKEN", text: gateway_token}
%{type: "secret_text", name: "CDP_SECRET", text: derived_cdp_secret} # HMAC of the gateway token
%{type: "secret_text", name: "OTEL_SHARED_SECRET", text: otel_secret}
%{type: "secret_text", name: "CF_ACCESS_PLATFORM_AUD", text: platform_aud}
%{type: "secret_text", name: "CF_ACCESS_CLIENT_AUD", text: tenant_aud} # Optional
# Platform-level secrets (from Lerma ENV, optional)
%{type: "secret_text", name: "CLOUDFLARE_AI_GATEWAY_API_KEY", text: ai_gateway_key}
%{type: "secret_text", name: "CLOUDFLARE_API_TOKEN", text: ai_gateway_key} # Tenant REST alias
# ... etc.
See apps/lerma/lib/lerma/provisioning/tenant_provisioner.ex for the authoritative binding list.
Container-Backed Durable Objects
The tenant User Worker’s Sandbox DO class is a container-backed Durable Object (it extends
Container from @cloudflare/sandbox). lerma deploys the worker into the dispatch namespace
with a containers array in the upload metadata (declaring Sandbox), which enables the
container runtime for that class — without it, Cloudflare throws
Containers have not been enabled for this Durable Object class when the worker calls into the
container (e.g. /api/doctor). lerma also creates/updates a container application for the
tenant (image + instance type + max instances) linked to the Sandbox DO namespace.
Container app image comes from CONTAINER_IMAGE, and the container instance type is derived
from the instance type chosen at provisioning time. When none is chosen it defaults to basic
(4 GB disk — lite’s 2 GB disk is too small to unpack the sandbox image).
CONTAINER_MAX_INSTANCES configures the max instances (see table above).
The tenant container application is pinned to the WEUR deployment zone by default
(CONTAINER_REGIONS) and container log capture is enabled (CONTAINER_OBSERVABILITY_LOGS).
Observability for the User Worker itself (logs + traces + invocation logs) mirrors the
apps/fouria/wrangler.jsonc block and is enabled by default
(WORKER_OBSERVABILITY_* vars above).
Legacy Channel Secrets (E2E Testing Only)
Deprecated. Channel secrets are now managed through the integrations dashboard (F4E1-encrypted R2 vault). These env vars are kept only for E2E test fixtures. Do not use in production.
Secrets reference: See docs/schemas/secrets-manifest.schema.json for the canonical credential inventory.
Internal Variables (Auto-Set)
These are automatically derived by the Worker and should never be manually configured:
| Variable | Source |
|---|---|
OPENCLAW_GATEWAY_TOKEN | Mapped from MOLTBOT_GATEWAY_TOKEN |
OPENCLAW_DEV_MODE | Derived from the -dev hostname (WORKER_URL) or an explicit OPENCLAW_DEV_MODE override; DEV_MODE for local development |
FOURIA_DOCS_CHANNEL | Derived: latest on a production-platform production instance, staging on the staging platform or a -dev environment (src/docs-channel.ts) |
FOURIA_DOCS_BASE_URL | Auto: https://fouria.io/<channel> — the channel-matched docs site |
FOURIA_DOCS_FALLBACK_URL | Auto: https://fouria-docs.pages.dev/<channel> — public mirror of the same channel |
FOURIA_VERSION | Auto: the Worker bundle’s component CalVer (src/build-info.ts), so agents can compare against the docs channel’s versions.json |
OPENCLAW_E2E_TEST_MODE | Mapped from E2E_TEST_MODE |
CUSTOMER_SLUG | Set from tenant resolution |
OTEL_EXPORTER_OTLP_HEADERS | Auto: Authorization=Bearer {OTEL_SHARED_SECRET} |
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT | Auto: http://otel.internal/v1/metrics (sandbox bridge) |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT | Auto: http://otel.internal/v1/traces (sandbox bridge) |
OTEL_TRACES_ENABLED | Provisioned by Lerma (true for production/staging, false for dev); 'true' enables diagnostics.otel.traces |
MOLTLAZY_FEATURE_FLAGS | Set as per-customer binding (managed via FourIA API); the staging wrangler.jsonc env default enables {"softwareProvisioning":true} |
MOLTLAZY_ANALYTICS_DATASET | Auto-set to tenant Analytics Engine dataset |
CUSTOMER_ANALYTICS_DATASET | Auto-set to tenant Analytics Engine dataset |
STARTUP_LOG_VERBOSITY | Passed through from Worker env when set (detailed default in container) |
OPENCODE_API_KEY | Per-client key minted by Lerma (Lerma.Opencode) from the platform management credential; bound only to the client’s dev instances |
MOLTLAZY_AGENTS | Deployed by the provisioner when an instance has a selected agent roster; comma-separated premade agent ids |
MOLTLAZY_PASO4_AGENTS | Deployed by the provisioner ("true") only on Paso4-exclusive (scope: "paso4") instances |
GITHUB_TOKEN | Deployed by the provisioner to Paso4-exclusive instances only; authenticates the paso4-dev agent’s gh CLI |
PASO4_TELEGRAM_GROUP_ID | Deployed by the provisioner to dev instances (lerma env, default -5442477312); Telegram group for platform alerts |
OPENCLAW_CONFIG_PATH | Baked into image: /root/openclaw/.openclaw/openclaw.json — the REAL config path (see below) |
OPENCLAW_STATE_DIR | Baked into image: /root/openclaw/.openclaw — the REAL state dir (see below) |
Why
OPENCLAW_CONFIG_PATH/OPENCLAW_STATE_DIRpoint at/root/openclaw/.openclaw:HOME=/home/openclawand/home/openclaw/.openclawis a symlink to the real config dir at/root/openclaw/.openclaw(kept under the/home/openclawsnapshot root for R2 backups). OpenClaw 2026.8.2+ fs-safe refuses atomic file replaces when the immediate parent directory is a symlink (“Atomic replace parent must be a real directory”), so config/state writes through the/homesymlink fail. These two env vars point OpenClaw at the real path for its own writes; the/home/openclaw/.openclawsymlink is preserved so backups and the Worker’s/home-based config reads/writes keep working. Set in the Dockerfile (apps/fouria/Dockerfile), not per-tenant.
Setting Variables
Local Development (.dev.vars)
Copy from .dev.vars.example:
cp apps/fouria/.dev.vars.example apps/fouria/.dev.vars
# Edit .dev.vars with your values
# .dev.vars is gitignored — never commit it
In local dev, instance secrets are plain strings (simulating secret_text bindings). Dynamic secrets need BACKUP_ENCRYPTION_KEY configured with the F4E1 R2 store.
Production Deployment
Instance secrets are deployed automatically by lerma when a tenant is provisioned. No manual wrangler secret put commands are needed.
Platform-level secrets should be set as environment variables in the dashboard’s deployment:
CLOUDFLARE_AI_GATEWAY_API_KEY=...
OPENCODE_API_KEY=... # management key, used to mint per-client dev keys
# ... etc.OpenTofu (IaC) Mapping
Secrets reference: See docs/schemas/secrets-manifest.schema.json for the canonical credential inventory.