# Staging / Dev Environment Promotion Schema

> Status: **foundation + CI build-to-staging promotion + in-product dev→production
> agent push implemented** — data model, naming, dispatch routing, automatic
> Fouria artifact promotion to the reserved staging `client`, and the HTTP
> Worker→Worker promotion of the promotable manifest (see below). Cloud
> provisioning of dual instances and lerma-side orchestration remain follow-ups.

## Motivation

Fouria provisions a sandboxed OpenClaw instance per client. Historically,
"staging" meant the clients provisioned by the **lerma dev deployment** — a
separate lerma with its own Supabase, dispatch namespace (`fouria-tenants-staging`)
and `staging-` hostname prefix. That made the whole dev platform the staging
environment for every customer.

This document changes the model: **each client owns its own environments**.
Every client gets a `production` instance (provisioned eagerly) and a `dev`
instance (provisioned lazily) for developing agents, tools, and configuration
without touching production. Lerma dev remains a separate deployment, but it is
no longer "the staging environment" — it just follows the same schema for its
own clients.

Promotion moves the _immutable_ configuration (agent workspace, tools, and
config files) from the client's `dev` instance to its `production` instance
using the Cloudflare Sandbox squashfs backup API with `.gitignore`-style
exclusions — a git-repo-style deployment.

## Two orthogonal axes

| Axis                                  | Values                                                            | Encoded by                                                                                          |
| ------------------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| **Platform** (the lerma deployment)   | `production` (fouria.io), `staging`/dev (dev.fouria.io dashboard) | `staging-` hostname **prefix**, `-staging` bucket/gateway **suffix**, dispatch namespace + Supabase |
| **Client environment** (the instance) | `production`, `dev`                                               | `-dev` hostname / worker-name **suffix**                                                            |

The `staging-` prefix continues to select the dev platform (unchanged). The
`-dev` suffix is new and selects the client's `dev` environment. Base slugs are
reserved against both (`staging-` prefix and `-dev` suffix) so they never
collide with the two axes.

## Naming

For a client with base slug `acme`:

| Platform   | Env        | Hostname                     | Worker name       | Container app                     | Namespace                | R2 bucket                        |
| ---------- | ---------- | ---------------------------- | ----------------- | --------------------------------- | ------------------------ | -------------------------------- |
| production | production | `acme.fouria.io`             | `tenant-acme`     | `tenant-acme-sandbox`             | `fouria-tenants`         | `fouria-backup-acme`             |
| production | dev        | `acme-dev.fouria.io`         | `tenant-acme-dev` | `tenant-acme-dev-sandbox`         | `fouria-tenants`         | `fouria-backup-acme-dev`         |
| staging    | production | `staging-acme.fouria.io`     | `tenant-acme`     | `tenant-acme-sandbox-staging`     | `fouria-tenants-staging` | `fouria-backup-acme-staging`     |
| staging    | dev        | `staging-acme-dev.fouria.io` | `tenant-acme-dev` | `tenant-acme-dev-sandbox-staging` | `fouria-tenants-staging` | `fouria-backup-acme-dev-staging` |

Naming is deterministic and centralized in `Lerma.Instances.Naming`
(`apps/lerma/lib/lerma/instances/naming.ex`).

> **Why the container application carries the platform suffix.** Worker scripts
> are isolated by their dispatch namespace and Durable Object namespaces are
> created inside that namespace, but **container applications live at the
> Cloudflare account level**. Without the platform axis a client with the same
> slug in both landscapes resolves to the same `tenant-{slug}-sandbox`;
> whichever platform creates it first wins, and because
> `durable_objects.namespace_id` is immutable at create time the other platform
> cannot rebind it — its container is cross-wired to the wrong DO namespace.
> `Naming.container_application_name/1` adds `{platform_suffix}` (and
> `ensure_container_application/5` recreates a same-named app that is bound to
> the wrong namespace) to close this gap.

## Data model

- `instances.environment` column (`"production"` | `"dev"`, default
  `"production"`) with a **unique index on `(client_id, environment)`** — one
  instance per client per environment.
- Client creation provisions the `production` instance eagerly; the `dev`
  instance is provisioned on first dev request (`provision_staging/1` follows
  the same pipeline with the `-dev` naming).

## Develop build promotion to the staging client

`.github/workflows/promote-staging.yml` is triggered when the `Test` workflow
completes on `develop`. It queries the source Actions run and proceeds only when
the `build-and-push` job conclusion is `success`; a skipped/failed job cannot
promote even if other jobs completed. The workflow uses the source run's commit
SHA as an immutable `unstable-<sha>` version. That one version already names
both the image pushed by `build-and-push` and the matching R2 bundle object, so
the promotion performs no mutable tag-copy step.

The workflow calls `POST /api/provisioning/upgrade/client` on staging Lerma with
the Fouria application's key (`Authorization: Bearer …` and
`X-FourIA-Application: fouria`). Lerma scopes an application key to instances
linked to that application; the optional `PROVISIONING_API_KEY` runtime value
remains a platform-wide compatibility key. The workflow checks
`GET /api/provisioning/upgrade/client` first, skips an already-completed
promotion, and polls status until every running instance for Fouria reports the
target version. The upgrade handler also skips instances already running or
upgrading to the requested version, so reruns do not start duplicate work.

### Convergence means a running tenant, not a declared image

A promotion is not converged when the target version is recorded: that value is
derived from the container application's **declared** image, which a rollout
sets the moment it is created. On 2026-11-03 both staging tenants were promoted
"successfully" in 40 s while every instance was down — the new image had no
runtime artifact yet, the instances failed to pull (`ImagePullError`, 10-minute
budget) and the app sat at zero healthy instances for over an hour.

The same field lies in the other direction: on 2026-11-18 a staging promotion
failed, its rollouts never activated, and the health monitor then saw the
still-healthy _old_ container and flipped the rows back to `running`. A rerun
would have passed both `deployed_version` and `container_health` while the
tenants kept serving the previous image. The gate therefore also compares
`serving_version` (the **live** container image tag from
`container_health.image`), which is `null` when health cannot be read.

Three guards close that gap:

- **Prepare the image before rolling.** `Lerma.Cloudflare.ImagePreparation`
  resolves the tag to its digest and waits for the platform's runtime artifact
  (`POST /containers/image-preparations`) to report `ready`.
  `Containers.deploy_script_and_container/1` runs it before the script upload,
  so a failure aborts without touching the tenant, and the restart /
  instance-type rollouts run it before recreating instances.
- **Gate on live health and the served image.** `GET /api/provisioning/upgrade/:slug`
  reports `container_health` (`healthy`/`starting`/`failed`/`errors`/`rollout_active`)
  and `serving_version` per instance; `promote-staging.sh` requires every active
  instance to report the target version as both `deployed_version` and
  `serving_version`, with at least one healthy instance, and fails fast when the
  application recorded a provisioning error with no healthy instance (printing
  each offending instance's health).
- **Never mark a failed rollout as a success.** `Instances.do_upgrade/3` only
  settles an upgrade once the application's live image carries the target
  version; a finished rollout that still points at the previous image (reverted,
  or never activated) keeps waiting and ends in `error`.

### The v1 surface is dev-only and artifact-gated

The staging platform also hosts the Sandbox SDK 1.0 track, but contract C1 has no
production cutover and Lerma provisions the v1 surface (the `SandboxV1` Durable
Object binding and its `durable_object` container application) only for a
client's **dev** instance. The automatic develop → staging promotion publishes
just `fouria:unstable-<sha>`; only the manual validation publisher pushes the
matching `fouria:v1-unstable-<sha>`. A promotion therefore must not assume the v1
artifact exists:

- a production-environment instance on staging is never given the v1 surface;
- a dev instance whose image has no published `v1-unstable-*` tag deploys
  legacy-only (the develop promotion path);
- an instance that already serves on v1 (born-v1, or converted — its row keeps
  `sandbox_runtime: "legacy"` and only the durable pointer says v1) fails closed
  instead of silently dropping the class, so a converted tenant is never
  re-deployed without its v1 surface.

ROOT operators generate/rotate/revoke application keys on Lerma's Applications
detail page. The plaintext is shown once; only its SHA-256 digest is stored in
the application row. For staging, copy the generated Fouria key into the
staging-only GitHub secret `STAGING_PROVISIONING_API_KEY`. It is never deployed
to tenant Workers or reused for production.

The dashboard Access application protects the whole host, so `/api/*` also needs
a path-scoped **bypass** application (mirroring the invitations bypass) for CI to
reach the API without a service token. The app-level
`LermaWeb.Plugs.CfAccessAuth` already allows `/api`, and each API route
authenticates with its own bearer credential.

The staging `deploy-dashboard.yml` path applies pending Lerma migrations with
`CF_ENVIRONMENT=staging` and then runs `Lerma.StagingTenant.ensure_client/0`.
That idempotent bootstrap renames the old seeded `fouria-staging` record when
appropriate and ensures the `client` tenant row. The migration creates a
database environment marker and delete/rename/truncate guards; staging bootstrap
sets the marker to `staging`, while production migrations default it to
`production` and never insert or modify the production `client` row. Production
and staging database credentials remain separate.

The application-scoped API contract and key lifecycle are covered by Lerma
ExUnit tests. The cloud promotion workflow itself requires
`STAGING_PROVISIONING_API_KEY` and the staging Lerma endpoint.

### Manual validation lane (`staging-validation.yml`)

The P2/P5a/P5b Sandbox 1.0 validation work (epic #380) uses a **separate,
manual-only** two-phase workflow, `.github/workflows/staging-validation.yml`,
instead of the automatic `promote-staging.yml` above. It runs only via
`workflow_dispatch` from the `sandbox-v1-validation` ref and promotes an
**approved staging Fouria client** selected by the required `target_slug` input
(default `client-sandbox-v1`); it never touches the reserved `client` slug or
production. The staging host is a fixed constant and cannot be changed by any
input. Effects are **staging-only** and there is **no automatic rollback**.

`target_slug` must be a lowercase DNS-label slug (`^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$`,
max 63 chars): uppercase, `.`, `/`, `%`, spaces, underscores, empty strings and
path traversal are rejected, and the reserved `client` slug is refused both in
the workflow and again in the promotion script. A mutating POST additionally
requires `confirm_target` to equal the validated `target_slug` exactly; a blank
or mismatched `confirm_target` is read-only (no POST).

Phase 1 (local, manual) is `.github/scripts/publish-validation.sh`, run from a
private, clean `sandbox-v1-validation` worktree with a private, non-symlinked
`node_modules`. It builds the bundle + both runtime images, pushes the unique
validation tags `unstable-<sha>` / `v1-unstable-<sha>`, resolves the true
registry manifest digests with the P5b helpers (never `docker inspect .Id`),
generates the compatible manifest and publishes the versioned bundle + manifest
to the validation R2 key. It never deploys the Worker. `--dry-run` is preflight
only; `--publish --sha <sha>` is required to mutate, and it aborts before any
write unless branch/SHA/dirty/node_modules/disk/auth preflight passes (disk
thresholds: 15 GiB before the build, 12 GiB after install, 5 GiB during build).

Phase 2 (this workflow) verifies the successful `Test` run for the exact SHA,
then `.github/scripts/verify-validation-artifacts.sh` verifies the prepublished
R2 bundle + manifest and both exact registry refs/digests **read-only** (no image
build on the undersized `ubuntu-latest` runner). The `promote` job runs the
read-only status preflight for a blank `confirm_target` and only POSTs when the
operator explicitly passes `confirm_target` equal to the validated `target_slug`,
then polls to exact **Lerma version convergence** only — it performs no automatic
post-deploy health probe or log query; operators follow health/observability
separately. Operational detail lives in
[`.github/workflows/README-staging-validation.md`](https://github.com/0xCAB0/fouria/blob/develop/.github/workflows/README-staging-validation.md).

## Dispatch routing

`workers/fouria-dispatch` parses both axes from the hostname label:

1. `extractSlug` — as before.
2. **Platform:** leading `staging-` prefix → dev platform (staging namespace +
   staging Supabase). Unchanged semantics.
3. **Environment:** trailing `-dev` suffix → client `dev` environment;
   otherwise `production` (`parseEnvironmentSlug`).
4. Worker name is derived directly from the routable slug:
   `tenant-{acme}` or `tenant-{acme-dev}` (`resolveTenant`).
5. The Supabase tenant query filters the instance by
   `(base_slug, environment)` so a frozen/stopped `dev` instance never blocks
   the `production` instance and vice-versa.
6. **Operational gate** (`tenantRouteGate`): production routes only when
   `clients.status` is `active`/`production`; `dev` routes on the **instance**
   status instead (the client may still be `draft` while its dev instance is
   provisioned lazily), rejecting `provisioning`/`frozen`/`stopped`/missing
   instances.

The fouria worker's `extractSlugFromHostname` (`src/auth/routing.ts`) strips
the `staging-` prefix but preserves the `-dev` suffix so metrics and the R2
secrets namespace stay consistent per environment.

## Promotion (squashfs + gitignore)

Moltlazy owns the immutable config generation, so it also owns the promotion
manifest (`packages/moltlazy/src/promotion/index.ts`):

- **Promotable:** `openclaw.json`, `moltlazy-session.json`, `moltlazy-tools.json`, `moltlazy-logging.json`, `workspace/` (agents, tools,
  memory, skills).
- **Excluded (never promoted):** `state/`, `credentials/`, `agents/`,
  `sessions/`, `secrets/`, keys, `.env`.

Promotion copies the two release artifacts (a `createBackup({
useGitignore: true })` squashfs of the workspace plus the versioned config
objects) from the client's dev bucket to its production bucket and records a
`release-pointer.json`, then triggers the production worker to restore (copy-
on-write FUSE overlay) and re-run `moltlazy patch`. Each environment uses a
distinct `BACKUP_ENCRYPTION_KEY`, so the copy decrypts with the dev key and
re-encrypts with the production key.

## Isolation guarantees

- One instance per (client, environment): distinct worker script, Sandbox DO,
  container application, R2 bucket, AI gateway, and encryption key.
- Dispatch resolves a `-dev` routable slug to exactly the `dev` instance and a
  plain slug to exactly the `production` instance — never cross-environment.
- A dev instance's frozen/stopped status never affects the production route.
- Promotion only ever carries the immutable manifest; runtime state,
  credentials, and sessions never cross the environment boundary.

## Implementation status

Done:

- `instances.environment` migration + schema + unique index.
- `Lerma.Instances.Naming` + tests.
- Slug reservation (`staging-` prefix, `-dev` suffix) + tests.
- Dispatch two-axis parsing (`parseEnvironmentSlug`, environment-aware
  `resolveTenant`, environment-filtered Supabase query) + tests.
- Fouria `extractSlugFromHostname` normalization + tests.
- Moltlazy `PatchOptions.environment` + promotion manifest module + tests.
- Root `tsconfig.json` excludes `**/dist/**` and `**/node_modules/**` from
  `tsc` so stale build artifacts no longer break `bun run typecheck`.

Follow-ups:

- Dual-instance provisioning in `TenantProvisioner` (prod eager / dev lazy).
- Access Policy per environment (dev Access application for `acme-dev.fouria.io`).
- ~~Worker admin promote endpoint (`POST /api/admin/promote`) + `/_control/:slug/promote`.~~
  **Implemented as a private R2 transport (no HTTP push):**
  `POST /api/admin/releases/promote` (dev side) stages the promotable manifest in
  the **dev instance's own bucket** (`promotion/staged/<id>.tar.gz` +
  `promotion/pending.json`); the production instance reads it through a read-only
  `PROMOTION_SOURCE_BUCKET` binding to that sibling bucket and applies it **on
  boot** (after the restore barrier, before the launch) or on
  `POST /api/admin/releases/apply`. `GET /api/admin/releases/status` reports the
  running release plus `available`. The apply holds the shared persistence lease
  and is skipped while parked / during a maintenance window or restore.
  There is no `/api/admin/releases/import` and no `PROMOTION_PUSH_TOKEN`; the
  lerma control-plane UI button and `/_control/:slug/promote` (dispatch) remain
  open (they would drive the same stage/apply surface).
- Lerma control-plane promotion orchestration + UI button.
- `moltlazy release` CLI assembling the release artifacts.
- The shared-bucket `release-pointer.json` variant (squashfs + `createBackup({
  useGitignore: true })`). The current implementation stores the bundle in the
  dev bucket and records the applied pointer in the production bucket
  (`promotion/applied.json`), plus the container release pointer; it does not yet
  use a `createBackup({ useGitignore: true })` squashfs archive.
