F FourIA staging 2026.10.29 GitHub ↗

Staging / Dev Environment Promotion Schema

On this page

Status: foundation + CI build-to-staging promotion + in-product dev→production agent push implemented — data model, naming, dispatch routing, automatic Fouria artifact promotion to the reserved staging client, and the HTTP Worker→Worker promotion of the promotable manifest (see below). Cloud provisioning of dual instances and lerma-side orchestration remain follow-ups.

Motivation

Fouria provisions a sandboxed OpenClaw instance per client. Historically, “staging” meant the clients provisioned by the lerma dev deployment — a separate lerma with its own Supabase, dispatch namespace (fouria-tenants-staging) and staging- hostname prefix. That made the whole dev platform the staging environment for every customer.

This document changes the model: each client owns its own environments. Every client gets a production instance (provisioned eagerly) and a dev instance (provisioned lazily) for developing agents, tools, and configuration without touching production. Lerma dev remains a separate deployment, but it is no longer “the staging environment” — it just follows the same schema for its own clients.

Promotion moves the immutable configuration (agent workspace, tools, and config files) from the client’s dev instance to its production instance using the Cloudflare Sandbox squashfs backup API with .gitignore-style exclusions — a git-repo-style deployment.

Two orthogonal axes

AxisValuesEncoded by
Platform (the lerma deployment)production (fouria.io), staging/dev (dev.fouria.io dashboard)staging- hostname prefix, -staging bucket/gateway suffix, dispatch namespace + Supabase
Client environment (the instance)production, dev-dev hostname / worker-name suffix

The staging- prefix continues to select the dev platform (unchanged). The -dev suffix is new and selects the client’s dev environment. Base slugs are reserved against both (staging- prefix and -dev suffix) so they never collide with the two axes.

Naming

For a client with base slug acme:

PlatformEnvHostnameWorker nameContainer appNamespaceR2 bucket
productionproductionacme.fouria.iotenant-acmetenant-acme-sandboxfouria-tenantsfouria-backup-acme
productiondevacme-dev.fouria.iotenant-acme-devtenant-acme-dev-sandboxfouria-tenantsfouria-backup-acme-dev
stagingproductionstaging-acme.fouria.iotenant-acmetenant-acme-sandbox-stagingfouria-tenants-stagingfouria-backup-acme-staging
stagingdevstaging-acme-dev.fouria.iotenant-acme-devtenant-acme-dev-sandbox-stagingfouria-tenants-stagingfouria-backup-acme-dev-staging

Naming is deterministic and centralized in Lerma.Instances.Naming (apps/lerma/lib/lerma/instances/naming.ex).

Why the container application carries the platform suffix. Worker scripts are isolated by their dispatch namespace and Durable Object namespaces are created inside that namespace, but container applications live at the Cloudflare account level. Without the platform axis a client with the same slug in both landscapes resolves to the same tenant-{slug}-sandbox; whichever platform creates it first wins, and because durable_objects.namespace_id is immutable at create time the other platform cannot rebind it — its container is cross-wired to the wrong DO namespace. Naming.container_application_name/1 adds {platform_suffix} (and ensure_container_application/5 recreates a same-named app that is bound to the wrong namespace) to close this gap.

Data model

  • instances.environment column ("production" | "dev", default "production") with a unique index on (client_id, environment) — one instance per client per environment.
  • Client creation provisions the production instance eagerly; the dev instance is provisioned on first dev request (provision_staging/1 follows the same pipeline with the -dev naming).

Develop build promotion to the staging client

.github/workflows/promote-staging.yml is triggered when the Test workflow completes on develop. It queries the source Actions run and proceeds only when the build-and-push job conclusion is success; a skipped/failed job cannot promote even if other jobs completed. The workflow uses the source run’s commit SHA as an immutable unstable-<sha> version. That one version already names both the image pushed by build-and-push and the matching R2 bundle object, so the promotion performs no mutable tag-copy step.

The workflow calls POST /api/provisioning/upgrade/client on staging Lerma with the Fouria application’s key (Authorization: Bearer … and X-FourIA-Application: fouria). Lerma scopes an application key to instances linked to that application; the optional PROVISIONING_API_KEY runtime value remains a platform-wide compatibility key. The workflow checks GET /api/provisioning/upgrade/client first, skips an already-completed promotion, and polls status until every running instance for Fouria reports the target version. The upgrade handler also skips instances already running or upgrading to the requested version, so reruns do not start duplicate work.

Convergence means a running tenant, not a declared image

A promotion is not converged when the target version is recorded: that value is derived from the container application’s declared image, which a rollout sets the moment it is created. On 2026-11-03 both staging tenants were promoted “successfully” in 40 s while every instance was down — the new image had no runtime artifact yet, the instances failed to pull (ImagePullError, 10-minute budget) and the app sat at zero healthy instances for over an hour.

The same field lies in the other direction: on 2026-11-18 a staging promotion failed, its rollouts never activated, and the health monitor then saw the still-healthy old container and flipped the rows back to running. A rerun would have passed both deployed_version and container_health while the tenants kept serving the previous image. The gate therefore also compares serving_version (the live container image tag from container_health.image), which is null when health cannot be read.

Three guards close that gap:

  • Prepare the image before rolling. Lerma.Cloudflare.ImagePreparation resolves the tag to its digest and waits for the platform’s runtime artifact (POST /containers/image-preparations) to report ready. Containers.deploy_script_and_container/1 runs it before the script upload, so a failure aborts without touching the tenant, and the restart / instance-type rollouts run it before recreating instances.
  • Gate on live health and the served image. GET /api/provisioning/upgrade/:slug reports container_health (healthy/starting/failed/errors/rollout_active) and serving_version per instance; promote-staging.sh requires every active instance to report the target version as both deployed_version and serving_version, with at least one healthy instance, and fails fast when the application recorded a provisioning error with no healthy instance (printing each offending instance’s health).
  • Never mark a failed rollout as a success. Instances.do_upgrade/3 only settles an upgrade once the application’s live image carries the target version; a finished rollout that still points at the previous image (reverted, or never activated) keeps waiting and ends in error.

The v1 surface is dev-only and artifact-gated

The staging platform also hosts the Sandbox SDK 1.0 track, but contract C1 has no production cutover and Lerma provisions the v1 surface (the SandboxV1 Durable Object binding and its durable_object container application) only for a client’s dev instance. The automatic develop → staging promotion publishes just fouria:unstable-<sha>; only the manual validation publisher pushes the matching fouria:v1-unstable-<sha>. A promotion therefore must not assume the v1 artifact exists:

  • a production-environment instance on staging is never given the v1 surface;
  • a dev instance whose image has no published v1-unstable-* tag deploys legacy-only (the develop promotion path);
  • an instance that already serves on v1 (born-v1, or converted — its row keeps sandbox_runtime: "legacy" and only the durable pointer says v1) fails closed instead of silently dropping the class, so a converted tenant is never re-deployed without its v1 surface.

ROOT operators generate/rotate/revoke application keys on Lerma’s Applications detail page. The plaintext is shown once; only its SHA-256 digest is stored in the application row. For staging, copy the generated Fouria key into the staging-only GitHub secret STAGING_PROVISIONING_API_KEY. It is never deployed to tenant Workers or reused for production.

The dashboard Access application protects the whole host, so /api/* also needs a path-scoped bypass application (mirroring the invitations bypass) for CI to reach the API without a service token. The app-level LermaWeb.Plugs.CfAccessAuth already allows /api, and each API route authenticates with its own bearer credential.

The staging deploy-dashboard.yml path applies pending Lerma migrations with CF_ENVIRONMENT=staging and then runs Lerma.StagingTenant.ensure_client/0. That idempotent bootstrap renames the old seeded fouria-staging record when appropriate and ensures the client tenant row. The migration creates a database environment marker and delete/rename/truncate guards; staging bootstrap sets the marker to staging, while production migrations default it to production and never insert or modify the production client row. Production and staging database credentials remain separate.

The application-scoped API contract and key lifecycle are covered by Lerma ExUnit tests. The cloud promotion workflow itself requires STAGING_PROVISIONING_API_KEY and the staging Lerma endpoint.

Manual validation lane (staging-validation.yml)

The P2/P5a/P5b Sandbox 1.0 validation work (epic #380) uses a separate, manual-only two-phase workflow, .github/workflows/staging-validation.yml, instead of the automatic promote-staging.yml above. It runs only via workflow_dispatch from the sandbox-v1-validation ref and promotes an approved staging Fouria client selected by the required target_slug input (default client-sandbox-v1); it never touches the reserved client slug or production. The staging host is a fixed constant and cannot be changed by any input. Effects are staging-only and there is no automatic rollback.

target_slug must be a lowercase DNS-label slug (^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$, max 63 chars): uppercase, ., /, %, spaces, underscores, empty strings and path traversal are rejected, and the reserved client slug is refused both in the workflow and again in the promotion script. A mutating POST additionally requires confirm_target to equal the validated target_slug exactly; a blank or mismatched confirm_target is read-only (no POST).

Phase 1 (local, manual) is .github/scripts/publish-validation.sh, run from a private, clean sandbox-v1-validation worktree with a private, non-symlinked node_modules. It builds the bundle + both runtime images, pushes the unique validation tags unstable-<sha> / v1-unstable-<sha>, resolves the true registry manifest digests with the P5b helpers (never docker inspect .Id), generates the compatible manifest and publishes the versioned bundle + manifest to the validation R2 key. It never deploys the Worker. --dry-run is preflight only; --publish --sha <sha> is required to mutate, and it aborts before any write unless branch/SHA/dirty/node_modules/disk/auth preflight passes (disk thresholds: 15 GiB before the build, 12 GiB after install, 5 GiB during build).

Phase 2 (this workflow) verifies the successful Test run for the exact SHA, then .github/scripts/verify-validation-artifacts.sh verifies the prepublished R2 bundle + manifest and both exact registry refs/digests read-only (no image build on the undersized ubuntu-latest runner). The promote job runs the read-only status preflight for a blank confirm_target and only POSTs when the operator explicitly passes confirm_target equal to the validated target_slug, then polls to exact Lerma version convergence only — it performs no automatic post-deploy health probe or log query; operators follow health/observability separately. Operational detail lives in .github/workflows/README-staging-validation.md.

Dispatch routing

workers/fouria-dispatch parses both axes from the hostname label:

  1. extractSlug — as before.
  2. Platform: leading staging- prefix → dev platform (staging namespace + staging Supabase). Unchanged semantics.
  3. Environment: trailing -dev suffix → client dev environment; otherwise production (parseEnvironmentSlug).
  4. Worker name is derived directly from the routable slug: tenant-{acme} or tenant-{acme-dev} (resolveTenant).
  5. The Supabase tenant query filters the instance by (base_slug, environment) so a frozen/stopped dev instance never blocks the production instance and vice-versa.
  6. Operational gate (tenantRouteGate): production routes only when clients.status is active/production; dev routes on the instance status instead (the client may still be draft while its dev instance is provisioned lazily), rejecting provisioning/frozen/stopped/missing instances.

The fouria worker’s extractSlugFromHostname (src/auth/routing.ts) strips the staging- prefix but preserves the -dev suffix so metrics and the R2 secrets namespace stay consistent per environment.

Promotion (squashfs + gitignore)

Moltlazy owns the immutable config generation, so it also owns the promotion manifest (packages/moltlazy/src/promotion/index.ts):

  • Promotable: openclaw.json, moltlazy-session.json, moltlazy-tools.json, moltlazy-logging.json, workspace/ (agents, tools, memory, skills).
  • Excluded (never promoted): state/, credentials/, agents/, sessions/, secrets/, keys, .env.

Promotion copies the two release artifacts (a createBackup({ useGitignore: true }) squashfs of the workspace plus the versioned config objects) from the client’s dev bucket to its production bucket and records a release-pointer.json, then triggers the production worker to restore (copy- on-write FUSE overlay) and re-run moltlazy patch. Each environment uses a distinct BACKUP_ENCRYPTION_KEY, so the copy decrypts with the dev key and re-encrypts with the production key.

Isolation guarantees

  • One instance per (client, environment): distinct worker script, Sandbox DO, container application, R2 bucket, AI gateway, and encryption key.
  • Dispatch resolves a -dev routable slug to exactly the dev instance and a plain slug to exactly the production instance — never cross-environment.
  • A dev instance’s frozen/stopped status never affects the production route.
  • Promotion only ever carries the immutable manifest; runtime state, credentials, and sessions never cross the environment boundary.

Implementation status

Done:

  • instances.environment migration + schema + unique index.
  • Lerma.Instances.Naming + tests.
  • Slug reservation (staging- prefix, -dev suffix) + tests.
  • Dispatch two-axis parsing (parseEnvironmentSlug, environment-aware resolveTenant, environment-filtered Supabase query) + tests.
  • Fouria extractSlugFromHostname normalization + tests.
  • Moltlazy PatchOptions.environment + promotion manifest module + tests.
  • Root tsconfig.json excludes **/dist/** and **/node_modules/** from tsc so stale build artifacts no longer break bun run typecheck.

Follow-ups:

  • Dual-instance provisioning in TenantProvisioner (prod eager / dev lazy).
  • Access Policy per environment (dev Access application for acme-dev.fouria.io).
  • Worker admin promote endpoint (POST /api/admin/promote) + /_control/:slug/promote. Implemented as a private R2 transport (no HTTP push): POST /api/admin/releases/promote (dev side) stages the promotable manifest in the dev instance’s own bucket (promotion/staged/<id>.tar.gz + promotion/pending.json); the production instance reads it through a read-only PROMOTION_SOURCE_BUCKET binding to that sibling bucket and applies it on boot (after the restore barrier, before the launch) or on POST /api/admin/releases/apply. GET /api/admin/releases/status reports the running release plus available. The apply holds the shared persistence lease and is skipped while parked / during a maintenance window or restore. There is no /api/admin/releases/import and no PROMOTION_PUSH_TOKEN; the lerma control-plane UI button and /_control/:slug/promote (dispatch) remain open (they would drive the same stage/apply surface).
  • Lerma control-plane promotion orchestration + UI button.
  • moltlazy release CLI assembling the release artifacts.
  • The shared-bucket release-pointer.json variant (squashfs + createBackup({ useGitignore: true })). The current implementation stores the bundle in the dev bucket and records the applied pointer in the production bucket (promotion/applied.json), plus the container release pointer; it does not yet use a createBackup({ useGitignore: true }) squashfs archive.