# Staging / Dev Environment Promotion Schema

> Status: **foundation + CI build-to-staging promotion + in-product dev→production
> agent push implemented** — data model, naming, dispatch routing, automatic
> Fouria artifact promotion to the reserved staging `client`, and the HTTP
> Worker→Worker promotion of the promotable manifest (see below). Cloud
> provisioning of dual instances and lerma-side orchestration remain follow-ups.

## Motivation

Fouria provisions a sandboxed OpenClaw instance per client. Historically,
"staging" meant the clients provisioned by the **lerma dev deployment** — a
separate lerma with its own Supabase, dispatch namespace (`fouria-tenants-staging`)
and `staging-` hostname prefix. That made the whole dev platform the staging
environment for every customer.

This document changes the model: **each client owns its own environments**.
Every client gets a `production` instance (provisioned eagerly) and a `dev`
instance (provisioned lazily) for developing agents, tools, and configuration
without touching production. Lerma dev remains a separate deployment, but it is
no longer "the staging environment" — it just follows the same schema for its
own clients.

Promotion moves the _immutable_ configuration (agent workspace, tools, and
config files) from the client's `dev` instance to its `production` instance
using the Cloudflare Sandbox squashfs backup API with `.gitignore`-style
exclusions — a git-repo-style deployment.

## Two orthogonal axes

| Axis                                  | Values                                                            | Encoded by                                                                                          |
| ------------------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| **Platform** (the lerma deployment)   | `production` (fouria.io), `staging`/dev (dev.fouria.io dashboard) | `staging-` hostname **prefix**, `-staging` bucket/gateway **suffix**, dispatch namespace + Supabase |
| **Client environment** (the instance) | `production`, `dev`                                               | `-dev` hostname / worker-name **suffix**                                                            |

The `staging-` prefix continues to select the dev platform (unchanged). The
`-dev` suffix is new and selects the client's `dev` environment. Base slugs are
reserved against both (`staging-` prefix and `-dev` suffix) so they never
collide with the two axes.

## Naming

For a client with base slug `acme`:

| Platform   | Env        | Hostname                     | Worker name       | Container app                     | Namespace                | R2 bucket                        |
| ---------- | ---------- | ---------------------------- | ----------------- | --------------------------------- | ------------------------ | -------------------------------- |
| production | production | `acme.fouria.io`             | `tenant-acme`     | `tenant-acme-sandbox`             | `fouria-tenants`         | `fouria-backup-acme`             |
| production | dev        | `acme-dev.fouria.io`         | `tenant-acme-dev` | `tenant-acme-dev-sandbox`         | `fouria-tenants`         | `fouria-backup-acme-dev`         |
| staging    | production | `staging-acme.fouria.io`     | `tenant-acme`     | `tenant-acme-sandbox-staging`     | `fouria-tenants-staging` | `fouria-backup-acme-staging`     |
| staging    | dev        | `staging-acme-dev.fouria.io` | `tenant-acme-dev` | `tenant-acme-dev-sandbox-staging` | `fouria-tenants-staging` | `fouria-backup-acme-dev-staging` |

Naming is deterministic and centralized in `Lerma.Instances.Naming`
(`apps/lerma/lib/lerma/instances/naming.ex`).

> **Why the container application carries the platform suffix.** Worker scripts
> are isolated by their dispatch namespace and Durable Object namespaces are
> created inside that namespace, but **container applications live at the
> Cloudflare account level**. Without the platform axis a client with the same
> slug in both landscapes resolves to the same `tenant-{slug}-sandbox`;
> whichever platform creates it first wins, and because
> `durable_objects.namespace_id` is immutable at create time the other platform
> cannot rebind it — its container is cross-wired to the wrong DO namespace.
> `Naming.container_application_name/1` adds `{platform_suffix}` (and
> `ensure_container_application/5` recreates a same-named app that is bound to
> the wrong namespace) to close this gap.

## Data model

- `instances.environment` column (`"production"` | `"dev"`, default
  `"production"`) with a **unique index on `(client_id, environment)`** — one
  instance per client per environment.
- Client creation provisions the `production` instance eagerly; the `dev`
  instance is provisioned on first dev request (`provision_staging/1` follows
  the same pipeline with the `-dev` naming).

## Develop build promotion to the staging client

`.github/workflows/promote-staging.yml` is triggered when the `Test` workflow
completes on `develop`. It queries the source Actions run and proceeds only when
the `build-and-push` job conclusion is `success`; a skipped/failed job cannot
promote even if other jobs completed. The workflow uses the source run's commit
SHA as an immutable `unstable-<sha>` version. That one version already names
both the image pushed by `build-and-push` and the matching R2 bundle object, so
the promotion performs no mutable tag-copy step.

The workflow calls `POST /api/provisioning/upgrade/client` on staging Lerma with
the Fouria application's key (`Authorization: Bearer …` and
`X-FourIA-Application: fouria`). Lerma scopes an application key to instances
linked to that application; the optional `PROVISIONING_API_KEY` runtime value
remains a platform-wide compatibility key. The workflow checks
`GET /api/provisioning/upgrade/client` first, skips an already-completed
promotion, and polls status until every running instance for Fouria reports the
target version. The upgrade handler also skips instances already running or
upgrading to the requested version, so reruns do not start duplicate work.

ROOT operators generate/rotate/revoke application keys on Lerma's Applications
detail page. The plaintext is shown once; only its SHA-256 digest is stored in
the application row. For staging, copy the generated Fouria key into the
staging-only GitHub secret `STAGING_PROVISIONING_API_KEY`. It is never deployed
to tenant Workers or reused for production.

The dashboard Access application protects the whole host, so `/api/*` also needs
a path-scoped **bypass** application (mirroring the invitations bypass) for CI to
reach the API without a service token. The app-level
`LermaWeb.Plugs.CfAccessAuth` already allows `/api`, and each API route
authenticates with its own bearer credential.

The staging `deploy-dashboard.yml` path applies pending Lerma migrations with
`CF_ENVIRONMENT=staging` and then runs `Lerma.StagingTenant.ensure_client/0`.
That idempotent bootstrap renames the old seeded `fouria-staging` record when
appropriate and ensures the `client` tenant row. The migration creates a
database environment marker and delete/rename/truncate guards; staging bootstrap
sets the marker to `staging`, while production migrations default it to
`production` and never insert or modify the production `client` row. Production
and staging database credentials remain separate.

The application-scoped API contract and key lifecycle are covered by Lerma
ExUnit tests. The cloud promotion workflow itself requires
`STAGING_PROVISIONING_API_KEY` and the staging Lerma endpoint.

### Manual validation lane (`staging-validation.yml`)

The P2/P5a/P5b Sandbox 1.0 validation work (epic #380) uses a **separate,
manual-only** two-phase workflow, `.github/workflows/staging-validation.yml`,
instead of the automatic `promote-staging.yml` above. It runs only via
`workflow_dispatch` from the `sandbox-v1-validation` ref and promotes an
**approved staging Fouria client** selected by the required `target_slug` input
(default `client-sandbox-v1`); it never touches the reserved `client` slug or
production. The staging host is a fixed constant and cannot be changed by any
input. Effects are **staging-only** and there is **no automatic rollback**.

`target_slug` must be a lowercase DNS-label slug (`^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$`,
max 63 chars): uppercase, `.`, `/`, `%`, spaces, underscores, empty strings and
path traversal are rejected, and the reserved `client` slug is refused both in
the workflow and again in the promotion script. A mutating POST additionally
requires `confirm_target` to equal the validated `target_slug` exactly; a blank
or mismatched `confirm_target` is read-only (no POST).

Phase 1 (local, manual) is `.github/scripts/publish-validation.sh`, run from a
private, clean `sandbox-v1-validation` worktree with a private, non-symlinked
`node_modules`. It builds the bundle + both runtime images, pushes the unique
validation tags `unstable-<sha>` / `v1-unstable-<sha>`, resolves the true
registry manifest digests with the P5b helpers (never `docker inspect .Id`),
generates the compatible manifest and publishes the versioned bundle + manifest
to the validation R2 key. It never deploys the Worker. `--dry-run` is preflight
only; `--publish --sha <sha>` is required to mutate, and it aborts before any
write unless branch/SHA/dirty/node_modules/disk/auth preflight passes (disk
thresholds: 15 GiB before the build, 12 GiB after install, 5 GiB during build).

Phase 2 (this workflow) verifies the successful `Test` run for the exact SHA,
then `.github/scripts/verify-validation-artifacts.sh` verifies the prepublished
R2 bundle + manifest and both exact registry refs/digests **read-only** (no image
build on the undersized `ubuntu-latest` runner). The `promote` job runs the
read-only status preflight for a blank `confirm_target` and only POSTs when the
operator explicitly passes `confirm_target` equal to the validated `target_slug`,
then polls to exact **Lerma version convergence** only — it performs no automatic
post-deploy health probe or log query; operators follow health/observability
separately. Operational detail lives in
[`.github/workflows/README-staging-validation.md`](https://github.com/0xCAB0/fouria/blob/develop/.github/workflows/README-staging-validation.md).

## Dispatch routing

`workers/fouria-dispatch` parses both axes from the hostname label:

1. `extractSlug` — as before.
2. **Platform:** leading `staging-` prefix → dev platform (staging namespace +
   staging Supabase). Unchanged semantics.
3. **Environment:** trailing `-dev` suffix → client `dev` environment;
   otherwise `production` (`parseEnvironmentSlug`).
4. Worker name is derived directly from the routable slug:
   `tenant-{acme}` or `tenant-{acme-dev}` (`resolveTenant`).
5. The Supabase tenant query filters the instance by
   `(base_slug, environment)` so a frozen/stopped `dev` instance never blocks
   the `production` instance and vice-versa.
6. **Operational gate** (`tenantRouteGate`): production routes only when
   `clients.status` is `active`/`production`; `dev` routes on the **instance**
   status instead (the client may still be `draft` while its dev instance is
   provisioned lazily), rejecting `provisioning`/`frozen`/`stopped`/missing
   instances.

The fouria worker's `extractSlugFromHostname` (`src/auth/routing.ts`) strips
the `staging-` prefix but preserves the `-dev` suffix so metrics and the R2
secrets namespace stay consistent per environment.

## Promotion (squashfs + gitignore)

Moltlazy owns the immutable config generation, so it also owns the promotion
manifest (`packages/moltlazy/src/promotion/index.ts`):

- **Promotable:** `openclaw.json`, `moltlazy-session.json`, `moltlazy-tools.json`, `moltlazy-logging.json`, `workspace/` (agents, tools,
  memory, skills).
- **Excluded (never promoted):** `state/`, `credentials/`, `agents/`,
  `sessions/`, `secrets/`, keys, `.env`.

Promotion copies the two release artifacts (a `createBackup({
useGitignore: true })` squashfs of the workspace plus the versioned config
objects) from the client's dev bucket to its production bucket and records a
`release-pointer.json`, then triggers the production worker to restore (copy-
on-write FUSE overlay) and re-run `moltlazy patch`. Each environment uses a
distinct `BACKUP_ENCRYPTION_KEY`, so the copy decrypts with the dev key and
re-encrypts with the production key.

## Isolation guarantees

- One instance per (client, environment): distinct worker script, Sandbox DO,
  container application, R2 bucket, AI gateway, and encryption key.
- Dispatch resolves a `-dev` routable slug to exactly the `dev` instance and a
  plain slug to exactly the `production` instance — never cross-environment.
- A dev instance's frozen/stopped status never affects the production route.
- Promotion only ever carries the immutable manifest; runtime state,
  credentials, and sessions never cross the environment boundary.

## Implementation status

Done:

- `instances.environment` migration + schema + unique index.
- `Lerma.Instances.Naming` + tests.
- Slug reservation (`staging-` prefix, `-dev` suffix) + tests.
- Dispatch two-axis parsing (`parseEnvironmentSlug`, environment-aware
  `resolveTenant`, environment-filtered Supabase query) + tests.
- Fouria `extractSlugFromHostname` normalization + tests.
- Moltlazy `PatchOptions.environment` + promotion manifest module + tests.
- Root `tsconfig.json` excludes `**/dist/**` and `**/node_modules/**` from
  `tsc` so stale build artifacts no longer break `bun run typecheck`.

Follow-ups:

- Dual-instance provisioning in `TenantProvisioner` (prod eager / dev lazy).
- Access Policy per environment (dev Access application for `acme-dev.fouria.io`).
- ~~Worker admin promote endpoint (`POST /api/admin/promote`) + `/_control/:slug/promote`.~~
  **Implemented (HTTP Worker→Worker, deferred apply):**
  `POST /api/admin/releases/promote` (dev side) exports the promotable manifest
  and pushes it to `POST /api/admin/releases/import` (production side); the
  import acknowledges `202` and applies + reloads the gateway off the response
  path. `POST /api/admin/releases/apply` and `GET /api/admin/releases/status`
  round out the surface, and `PROMOTION_PUSH_TOKEN` authenticates the worker→worker
  hop. The lerma control-plane UI button and `/_control/:slug/promote` (dispatch)
  remain open.
- Lerma control-plane promotion orchestration + UI button.
- `moltlazy release` CLI assembling the release artifacts.
- The shared-bucket `release-pointer.json` variant (squashfs + `createBackup({
  useGitignore: true })`). The current implementation streams the bundle over
  HTTP and stores it only transiently in the target container; it does not yet
  write a durable per-tenant `release-pointer.json` to R2.
