Staging / Dev Environment Promotion Schema
On this page
Status: foundation + CI build-to-staging promotion + in-product dev→production agent push implemented — data model, naming, dispatch routing, automatic Fouria artifact promotion to the reserved staging
client, and the HTTP Worker→Worker promotion of the promotable manifest (see below). Cloud provisioning of dual instances and lerma-side orchestration remain follow-ups.
Motivation
Fouria provisions a sandboxed OpenClaw instance per client. Historically,
“staging” meant the clients provisioned by the lerma dev deployment — a
separate lerma with its own Supabase, dispatch namespace (fouria-tenants-staging)
and staging- hostname prefix. That made the whole dev platform the staging
environment for every customer.
This document changes the model: each client owns its own environments.
Every client gets a production instance (provisioned eagerly) and a dev
instance (provisioned lazily) for developing agents, tools, and configuration
without touching production. Lerma dev remains a separate deployment, but it is
no longer “the staging environment” — it just follows the same schema for its
own clients.
Promotion moves the immutable configuration (agent workspace, tools, and
config files) from the client’s dev instance to its production instance
using the Cloudflare Sandbox squashfs backup API with .gitignore-style
exclusions — a git-repo-style deployment.
Two orthogonal axes
| Axis | Values | Encoded by |
|---|---|---|
| Platform (the lerma deployment) | production (fouria.io), staging/dev (dev.fouria.io dashboard) | staging- hostname prefix, -staging bucket/gateway suffix, dispatch namespace + Supabase |
| Client environment (the instance) | production, dev | -dev hostname / worker-name suffix |
The staging- prefix continues to select the dev platform (unchanged). The
-dev suffix is new and selects the client’s dev environment. Base slugs are
reserved against both (staging- prefix and -dev suffix) so they never
collide with the two axes.
Naming
For a client with base slug acme:
| Platform | Env | Hostname | Worker name | Container app | Namespace | R2 bucket |
|---|---|---|---|---|---|---|
| production | production | acme.fouria.io | tenant-acme | tenant-acme-sandbox | fouria-tenants | fouria-backup-acme |
| production | dev | acme-dev.fouria.io | tenant-acme-dev | tenant-acme-dev-sandbox | fouria-tenants | fouria-backup-acme-dev |
| staging | production | staging-acme.fouria.io | tenant-acme | tenant-acme-sandbox-staging | fouria-tenants-staging | fouria-backup-acme-staging |
| staging | dev | staging-acme-dev.fouria.io | tenant-acme-dev | tenant-acme-dev-sandbox-staging | fouria-tenants-staging | fouria-backup-acme-dev-staging |
Naming is deterministic and centralized in Lerma.Instances.Naming
(apps/lerma/lib/lerma/instances/naming.ex).
Why the container application carries the platform suffix. Worker scripts are isolated by their dispatch namespace and Durable Object namespaces are created inside that namespace, but container applications live at the Cloudflare account level. Without the platform axis a client with the same slug in both landscapes resolves to the same
tenant-{slug}-sandbox; whichever platform creates it first wins, and becausedurable_objects.namespace_idis immutable at create time the other platform cannot rebind it — its container is cross-wired to the wrong DO namespace.Naming.container_application_name/1adds{platform_suffix}(andensure_container_application/5recreates a same-named app that is bound to the wrong namespace) to close this gap.
Data model
instances.environmentcolumn ("production"|"dev", default"production") with a unique index on(client_id, environment)— one instance per client per environment.- Client creation provisions the
productioninstance eagerly; thedevinstance is provisioned on first dev request (provision_staging/1follows the same pipeline with the-devnaming).
Develop build promotion to the staging client
.github/workflows/promote-staging.yml is triggered when the Test workflow
completes on develop. It queries the source Actions run and proceeds only when
the build-and-push job conclusion is success; a skipped/failed job cannot
promote even if other jobs completed. The workflow uses the source run’s commit
SHA as an immutable unstable-<sha> version. That one version already names
both the image pushed by build-and-push and the matching R2 bundle object, so
the promotion performs no mutable tag-copy step.
The workflow calls POST /api/provisioning/upgrade/client on staging Lerma with
the Fouria application’s key (Authorization: Bearer … and
X-FourIA-Application: fouria). Lerma scopes an application key to instances
linked to that application; the optional PROVISIONING_API_KEY runtime value
remains a platform-wide compatibility key. The workflow checks
GET /api/provisioning/upgrade/client first, skips an already-completed
promotion, and polls status until every running instance for Fouria reports the
target version. The upgrade handler also skips instances already running or
upgrading to the requested version, so reruns do not start duplicate work.
ROOT operators generate/rotate/revoke application keys on Lerma’s Applications
detail page. The plaintext is shown once; only its SHA-256 digest is stored in
the application row. For staging, copy the generated Fouria key into the
staging-only GitHub secret STAGING_PROVISIONING_API_KEY. It is never deployed
to tenant Workers or reused for production.
The dashboard Access application protects the whole host, so /api/* also needs
a path-scoped bypass application (mirroring the invitations bypass) for CI to
reach the API without a service token. The app-level
LermaWeb.Plugs.CfAccessAuth already allows /api, and each API route
authenticates with its own bearer credential.
The staging deploy-dashboard.yml path applies pending Lerma migrations with
CF_ENVIRONMENT=staging and then runs Lerma.StagingTenant.ensure_client/0.
That idempotent bootstrap renames the old seeded fouria-staging record when
appropriate and ensures the client tenant row. The migration creates a
database environment marker and delete/rename/truncate guards; staging bootstrap
sets the marker to staging, while production migrations default it to
production and never insert or modify the production client row. Production
and staging database credentials remain separate.
The application-scoped API contract and key lifecycle are covered by Lerma
ExUnit tests. The cloud promotion workflow itself requires
STAGING_PROVISIONING_API_KEY and the staging Lerma endpoint.
Manual validation lane (staging-validation.yml)
The P2/P5a/P5b Sandbox 1.0 validation work (epic #380) uses a separate,
manual-only two-phase workflow, .github/workflows/staging-validation.yml,
instead of the automatic promote-staging.yml above. It runs only via
workflow_dispatch from the sandbox-v1-validation ref and promotes an
approved staging Fouria client selected by the required target_slug input
(default client-sandbox-v1); it never touches the reserved client slug or
production. The staging host is a fixed constant and cannot be changed by any
input. Effects are staging-only and there is no automatic rollback.
target_slug must be a lowercase DNS-label slug (^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$,
max 63 chars): uppercase, ., /, %, spaces, underscores, empty strings and
path traversal are rejected, and the reserved client slug is refused both in
the workflow and again in the promotion script. A mutating POST additionally
requires confirm_target to equal the validated target_slug exactly; a blank
or mismatched confirm_target is read-only (no POST).
Phase 1 (local, manual) is .github/scripts/publish-validation.sh, run from a
private, clean sandbox-v1-validation worktree with a private, non-symlinked
node_modules. It builds the bundle + both runtime images, pushes the unique
validation tags unstable-<sha> / v1-unstable-<sha>, resolves the true
registry manifest digests with the P5b helpers (never docker inspect .Id),
generates the compatible manifest and publishes the versioned bundle + manifest
to the validation R2 key. It never deploys the Worker. --dry-run is preflight
only; --publish --sha <sha> is required to mutate, and it aborts before any
write unless branch/SHA/dirty/node_modules/disk/auth preflight passes (disk
thresholds: 15 GiB before the build, 12 GiB after install, 5 GiB during build).
Phase 2 (this workflow) verifies the successful Test run for the exact SHA,
then .github/scripts/verify-validation-artifacts.sh verifies the prepublished
R2 bundle + manifest and both exact registry refs/digests read-only (no image
build on the undersized ubuntu-latest runner). The promote job runs the
read-only status preflight for a blank confirm_target and only POSTs when the
operator explicitly passes confirm_target equal to the validated target_slug,
then polls to exact Lerma version convergence only — it performs no automatic
post-deploy health probe or log query; operators follow health/observability
separately. Operational detail lives in
.github/workflows/README-staging-validation.md.
Dispatch routing
workers/fouria-dispatch parses both axes from the hostname label:
extractSlug— as before.- Platform: leading
staging-prefix → dev platform (staging namespace + staging Supabase). Unchanged semantics. - Environment: trailing
-devsuffix → clientdevenvironment; otherwiseproduction(parseEnvironmentSlug). - Worker name is derived directly from the routable slug:
tenant-{acme}ortenant-{acme-dev}(resolveTenant). - The Supabase tenant query filters the instance by
(base_slug, environment)so a frozen/stoppeddevinstance never blocks theproductioninstance and vice-versa. - Operational gate (
tenantRouteGate): production routes only whenclients.statusisactive/production;devroutes on the instance status instead (the client may still bedraftwhile its dev instance is provisioned lazily), rejectingprovisioning/frozen/stopped/missing instances.
The fouria worker’s extractSlugFromHostname (src/auth/routing.ts) strips
the staging- prefix but preserves the -dev suffix so metrics and the R2
secrets namespace stay consistent per environment.
Promotion (squashfs + gitignore)
Moltlazy owns the immutable config generation, so it also owns the promotion
manifest (packages/moltlazy/src/promotion/index.ts):
- Promotable:
openclaw.json,moltlazy-session.json,moltlazy-tools.json,moltlazy-logging.json,workspace/(agents, tools, memory, skills). - Excluded (never promoted):
state/,credentials/,agents/,sessions/,secrets/, keys,.env.
Promotion copies the two release artifacts (a createBackup({ useGitignore: true }) squashfs of the workspace plus the versioned config
objects) from the client’s dev bucket to its production bucket and records a
release-pointer.json, then triggers the production worker to restore (copy-
on-write FUSE overlay) and re-run moltlazy patch. Each environment uses a
distinct BACKUP_ENCRYPTION_KEY, so the copy decrypts with the dev key and
re-encrypts with the production key.
Isolation guarantees
- One instance per (client, environment): distinct worker script, Sandbox DO, container application, R2 bucket, AI gateway, and encryption key.
- Dispatch resolves a
-devroutable slug to exactly thedevinstance and a plain slug to exactly theproductioninstance — never cross-environment. - A dev instance’s frozen/stopped status never affects the production route.
- Promotion only ever carries the immutable manifest; runtime state, credentials, and sessions never cross the environment boundary.
Implementation status
Done:
instances.environmentmigration + schema + unique index.Lerma.Instances.Naming+ tests.- Slug reservation (
staging-prefix,-devsuffix) + tests. - Dispatch two-axis parsing (
parseEnvironmentSlug, environment-awareresolveTenant, environment-filtered Supabase query) + tests. - Fouria
extractSlugFromHostnamenormalization + tests. - Moltlazy
PatchOptions.environment+ promotion manifest module + tests. - Root
tsconfig.jsonexcludes**/dist/**and**/node_modules/**fromtscso stale build artifacts no longer breakbun run typecheck.
Follow-ups:
- Dual-instance provisioning in
TenantProvisioner(prod eager / dev lazy). - Access Policy per environment (dev Access application for
acme-dev.fouria.io). Worker admin promote endpoint (Implemented (HTTP Worker→Worker, deferred apply):POST /api/admin/promote) +/_control/:slug/promote.POST /api/admin/releases/promote(dev side) exports the promotable manifest and pushes it toPOST /api/admin/releases/import(production side); the import acknowledges202and applies + reloads the gateway off the response path.POST /api/admin/releases/applyandGET /api/admin/releases/statusround out the surface, andPROMOTION_PUSH_TOKENauthenticates the worker→worker hop. The lerma control-plane UI button and/_control/:slug/promote(dispatch) remain open.- Lerma control-plane promotion orchestration + UI button.
moltlazy releaseCLI assembling the release artifacts.- The shared-bucket
release-pointer.jsonvariant (squashfs +createBackup({ useGitignore: true })). The current implementation streams the bundle over HTTP and stores it only transiently in the target container; it does not yet write a durable per-tenantrelease-pointer.jsonto R2.