# Fouria Dispatcher

Wildcard dispatch Worker for FourIA's multi-tenant OpenClaw gateway platform. Routes all `*.fouria.io` traffic to the correct tenant worker, enforces Cloudflare Access auth per-tenant via JWT audience matching, and applies plan-level CPU/subrequest limits.

## Goal

The dispatch Worker is the entry point for all tenant traffic. On every request it:

1. **Extracts the tenant slug** from the hostname (`tenant-a.fouria.io` → `tenant-a`)
2. **Resolves the tenant** — checks KV cache (TTL: 60s), falls back to a Supabase query
3. **Enforces auth** — verifies the Cloudflare Access JWT signature. In production, non-ROOT callers must present a JWT `aud` that resolves (via the Cloudflare API) to the requested hostname's slug; Paso4 ROOT operators are exempt (platform wildcard AUD `*.fouria.io` **or** membership of the `Paso4 Root Users` Access group). Staging is ROOT-only (platform AUD or ROOT group membership), **except the tenant health endpoints** (`/api/health`, `/api/status` and their sub-paths): those path-scoped Access apps carry no role requirement — they admit any authenticated user and lerma's `FourIA Health Monitor` service token — so Dispatch lets a request that already passed Access through on those paths (every other path stays ROOT-only).
4. **Applies the operational gate** — rejects `draft`/`suspended`/`archived` and answers 503 for `provisioning` and for `frozen`/`stopped` (ROOT-exempt). A health-derived `error` never blocks (the container is loaded lazily).
5. **Resolves the forwarded role** — `X-FourIA-Role` carries the lerma-assigned client role (`client_user_instances`). On a `dev` instance a client-level **ADMIN** (validated `client_users` row) is admitted even without an explicit dev-instance assignment, since admins are the ones developing there; BASE_USERs still need an explicit dev-instance assignment.
6. **Builds billing metadata** — queries `clients`, `credit_pools`, and `client_users` tables, encodes as base64 JSON in `X-FourIA-Metadata`
7. **Dispatches** — forwards the request via `DISPATCHER.get(workerName)` with plan-level limits and billing context

## Architecture

```
Request → *.fouria.io
  │
  ├─► extractSlug() — parse tenant from hostname
  │     OR apex (fouria.io) — resolve slug from Access JWT aud via Cloudflare API
  │
  ├─► resolveTenant()
  │   ├─► KV cache (slug:<tenant>) — 60s TTL
  │   └─► Supabase query (clients + credit_pools + client_users)
  │
  ├─► requireAccessAuth() — Cloudflare Access JWT (jose)
  ├─► resolveSlugByAud() — Cloudflare SDK (production hostname + apex)
  │
  ├─► tenantRouteGate() — lifecycle + frozen/stopped (ROOT-exempt); never health
  │
  ├─► buildBillingMetadata() → X-FourIA-Metadata base64 header
  │
  └─► DISPATCHER.get(workerName).fetch() — plan limits + billing context
```

### Data flow

The dispatch Worker resolves tenants by querying **Supabase** (shared Postgres backing lerma) using `SUPABASE_URL` + `SUPABASE_SERVICE_ROLE_KEY`. No HTTP dependency on the dashboard (lerma). The Supabase queries span three tables:

| Table          | Columns used                            |
| -------------- | --------------------------------------- |
| `clients`      | `id, slug, billing_plan, status`        |
| `credit_pools` | `lifetime_consumed, lifetime_purchased` |
| `client_users` | count by `client_id` (seats used)       |

### Apex tag routing

Requests to `fouria.io` (host === `PLATFORM_DOMAIN`) are resolved differently:

1. Verify the Cloudflare Access JWT signature (`jose`)
2. Extract the `aud` claim from the JWT payload
3. Call the Cloudflare API (`cloudflare` SDK) to list Access apps filtered by `aud`
4. Find the app whose `aud` matches; read its domain (`{slug}.fouria.io`) to derive the slug
5. Dispatch to `tenant-{slug}` in the production dispatch namespace

Staging tenants are **NOT reachable via apex** — they only use the hostname path (`staging-{slug}.fouria.io`).

### Auth via Cloudflare SDK

JWT signature/JWKS verification uses `jose` (as before). Tenant-by-aud resolution uses the official `cloudflare` npm package (`github.com/cloudflare/cloudflare-typescript`). Per-tenant AUD is **NOT persisted in the database anymore** — the dispatch worker resolves it dynamically from the Cloudflare Access app configuration.

### Staging Prefix

Staging tenants use the `staging-{slug}.fouria.io` pattern to avoid collisions with production tenants. The dispatch Worker handles both production (`{slug}.fouria.io`) and staging (`staging-{slug}.fouria.io`) subdomains on the same PLATFORM_DOMAIN. When `extractSlug()` encounters a `staging-` prefix, it strips the prefix and routes to the `fouria-tenants-staging` dispatch namespace with the un-prefixed slug.

| Subdomain pattern          | Namespace                | Example                                |
| -------------------------- | ------------------------ | -------------------------------------- |
| `{slug}.fouria.io`         | `fouria-tenants`         | `acme.fouria.io` → tenant-acme         |
| `staging-{slug}.fouria.io` | `fouria-tenants-staging` | `staging-acme.fouria.io` → tenant-acme |

> **Dashboard traffic** (`dash.fouria.io`, `dev-dash.fouria.io`) is served via bypass worker routes on the fouria.io zone — the Dispatch Worker does not intercept these hostnames.

### Client environments and the routing gate

Routing has **two independent axes** (see `Lerma.Instances.Naming`):

| Axis               | Encoded as                           | Example                                  |
| ------------------ | ------------------------------------ | ---------------------------------------- |
| Platform           | `staging-` hostname prefix           | `staging-acme.fouria.io` → `tenant-acme` |
| Client environment | `-dev` hostname / worker-name suffix | `acme-dev.fouria.io` → `tenant-acme-dev` |

`resolveTenant()` parses both axes: the `-dev` suffix selects the client's `dev`
environment, the `staging-` prefix selects the staging platform, and the base
slug is looked up in `clients` with the `instances.environment` filter applied.

The operational gate before dispatch depends on the environment:

- **production** — `clients.status` gates routing: `draft`/`suspended`/`archived`
  are rejected (404) and `provisioning` returns 503. A client left in `error` by
  the HealthMonitor is **routable**.
- **dev** — the **instance is the source of truth**. A `dev` instance is
  provisioned lazily while the client is still `draft` (a dev provision never
  drives the client status), so `clients.status = draft` must NOT block it. A
  `dev` request routes when the instance exists and is `running`/`upgrading`/
  `error`; it is rejected when the instance is missing (404) or `provisioning`
  (503).

`frozen`/`stopped` instances return 503 for non-ROOT callers (billing
protection), but a **ROOT operator is exempt** so they can always reach the
worker to revive it. Explicit operator revocations (`clients.status =
suspended`/`archived`) block both environments.

A health-derived `error` status is **never** a routing gate: tenant containers
are loaded lazily by the User Worker, so a tenant the HealthMonitor could not
reach is not an outage, and blocking it would prevent operators from reviving
it. The gate runs **after authentication** so the ROOT exemption can be applied.

### Role resolution on a dev instance

The forwarded role (`X-FourIA-Role`) resolves from the user's `client_users`
row linked to the requested instance. A `dev` instance has an extra fallback:
when no per-instance role is found, a validated client-level **ADMIN** is
granted access — client admins develop on the dev instance and are typically
only assigned to the production instance. BASE_USERs are not granted dev
access this way; ROOT operators are unaffected (they resolve before the
fallback).

Separately, the tenant Worker lets a resolved **ROOT** operator _view and
perform as_ the tenant's `ADMIN` on a dev instance via the dashboard's "View
as" switcher (a dev-only, strict-downgrade role preview — see
[`apps/fouria/docs/WORKER-DEVELOPMENT.md` §RBAC](/staging/llms/platform/apps/fouria/worker-development/index.md)).
That is a tenant-side concern; Dispatch still forwards the real ROOT role.

### Periodic backup fan-out

The Worker's cron trigger sweeps every tenant in `TENANT_INDEX` (`slug:` keys)
and forwards a `backup` control action to each (`POST /control`, header
`X-FourIA-Control: backup`). Deterministic per-tenant delays spread the fleet
across a jitter window so snapshots do not all start on the same wall-clock
minute.

#### Sweep and queue semantics

1. **Paginated sweep.** `collectTenantSlugs` follows KV `list_complete` /
   `cursor`, so every tenant is reached even past KV's 1,000-key page (the
   previous single-page loop silently starved tenant #1,001 onward). Keys are
   deduped in a `Set`.
2. **Queue fan-out.** When the `CONTROL_QUEUE` producer is bound, the sweep
   sends one message per tenant (`{ slug, requestedAt }`) in batches of at most
   100 — Cloudflare's `sendBatch` limit — instead of holding one subrequest per
   tenant in the cron invocation. The consumer (`queue()` in `src/index.ts`)
   resolves `slug:<slug>` against the **live** index at delivery time, so a
   tenant frozen or deleted after the sweep is skipped rather than woken.
   Missing/frozen/nameless entries are skipped; `delivered` and `skipped` are
   `ack`ed, `failed` is retried.
3. **Per-tenant jitter.** Each queued message carries `delaySeconds` derived
   deterministically from the slug (FNV-1a, `0..CONTROL_FANOUT_JITTER_SECONDS`,
   default 300 s). Determinism keeps a redelivered sweep on the same schedule;
   the offset is queue-only, so the direct fallback delivers immediately.
4. **Bounded fallback.** Without the queue binding (local dev or an environment
   that has not created it yet), the sweep delivers directly with a concurrency
   cap of 8, so one slow tenant cannot serialize the whole fleet. Delivery is
   idempotent: the tenant enqueues a durable snapshot job and the backup gate is
   interval/lock-based.

The tenant answers without ever starting the container: it reads
`sandbox.isRuntimeActive()` (a **non-waking** bridge call) and returns
`{ ok: true, skipped: 'not-running' }` for a sleeping instance. Only a live
instance proceeds to the interval/lock-gated `scheduleBackupIfNeeded`. This is
the backstop for tenants the lerma `HealthMonitor` does not poll, and because it
never wakes an idle container it does not defeat `sleepAfter` or restart billing.
Health probes themselves are observability-only and never write the backup
catalog.

Queue names: `fouria-dispatch-control` (local), `-staging`, `-prod`. The
`deploy-dispatch` workflow creates them idempotently (`wrangler queues info … ||
wrangler queues create …`) before deploying. Set
`CONTROL_FANOUT_JITTER_SECONDS=0` to disable jitter, or another positive value
(capped at 86400) to widen/narrow the window.

### Pre-replacement checkpoint

`POST /_control/:slug/checkpoint` forwards a **synchronous, verified** checkpoint
(the `checkpoint` control action) to the tenant worker with the same raised CPU
budget as `backup` (`BACKUP_CONTROL_CPU_MS`). The Lerma control plane calls it
immediately before an image rollout / container replacement so the state about
to be discarded is durably captured first. Unlike `backup` it is **not**
interval-gated and **fails closed**: the tenant worker only reports success once
the committed archive is restorable, and a failure propagates as a non-2xx so the
replacement is aborted. A sleeping/destroyed instance is reported as
`skipped: not-running` (never woken).

### Sandbox v1 migration

`POST /_control/:slug/migrate` forwards the `migrate` control action to the
tenant worker with the same raised CPU budget as `backup`/`checkpoint`
(`BACKUP_CONTROL_CPU_MS`). The tenant worker **only enqueues**: it records a
durable job on its `SandboxV1` Durable Object and answers immediately, so the
request never spans the archive round-trip (the conversion runs on the DO
alarm). The tenant worker is the authority for the migration (the staging-only
platform gate, the reserved `client` slug, and the conversion lease), so a
refusal arrives as `403`/`409` with its `code` preserved.

`POST /_control/:slug/migration-status` forwards the **read-only**
`migration-status` action (same route, no raised CPU budget, no tenant-cache
invalidation). It exists because `migrate` is an idempotent enqueue — it starts
work from `absent`/`failed` — so a poll that reused it would convert a tenant
just by asking. `migration-status` reads the durable state and can never enqueue.

Both echoes carry the durable conversion state, which the Lerma control plane
renders:

| Tenant answer                      | Forwarded status | Echo                                        |
| ---------------------------------- | ---------------- | ------------------------------------------- |
| `{state: "queued" \| "running"}`   | `202`            | `state`                                     |
| `{state: "committed", recordId}`   | `200`            | `state`, `recordId`                         |
| `{state: "absent"}` (status only)  | `200`            | `state`                                     |
| `{state: "failed", error}`         | `502` (migrate)  | `state`, `tenantError`                      |
| `{ok: false, code: "MIGRATION_…"}` | `403`/`409`      | `code`, `tenantError` (no invented `state`) |

A `failed` conversion is a durable state, not a transport error, so its `state`
is echoed on the failure path too — the control plane distinguishes "the
conversion broke" from "the request was refused" by the presence of `state`
versus `code`. Only `POST /_control/:slug/migrate` can start a conversion;
`migration-status` is safe to poll.

## Environment Variables

### Wrangler vars (plaintext)

| Variable                        | Staging                            | Production                         | Description                                                               |
| ------------------------------- | ---------------------------------- | ---------------------------------- | ------------------------------------------------------------------------- |
| `PLATFORM_DOMAIN`               | `fouria.io`                        | `fouria.io`                        | Domain suffix for tenant slug extraction                                  |
| _(staging prefix)_              | `staging-{slug}`                   | _(not used in production)_         | Staging tenants use `staging-{slug}.fouria.io` via dispatch               |
| `CF_ACCESS_TEAM_DOMAIN`         | `paso4.cloudflareaccess.com`       | `paso4.cloudflareaccess.com`       | Cloudflare Access team domain for JWT auth                                |
| `SUPABASE_URL`                  | _(set per env)_                    | _(set per env)_                    | Supabase project URL                                                      |
| `CF_ACCOUNT_ID`                 | `35afea16440634aa2350331d2a736eec` | `35afea16440634aa2350331d2a736eec` | Cloudflare account ID (used by Cloudflare SDK)                            |
| `CONTROL_FANOUT_JITTER_SECONDS` | _(optional)_                       | _(optional)_                       | Per-tenant fan-out delay window; default `300`, `0` disables, max `86400` |

### Wrangler secrets

| Secret                      | Description                                                                                                                |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `SUPABASE_SERVICE_ROLE_KEY` | Supabase service role key (bypasses RLS)                                                                                   |
| `CF_ACCESS_PLATFORM_AUD`    | Cloudflare Access audience for Paso4 platform / staging auth (ROOT is also granted by `Paso4 Root Users` group membership) |
| `CF_ACCESS_API_TOKEN`       | Cloudflare API token (Access:Read)                                                                                         |

### Bound resources

| Binding         | Resource                                                                        | Description                       |
| --------------- | ------------------------------------------------------------------------------- | --------------------------------- |
| `TENANT_INDEX`  | KV namespace `081f07b1e0e54fb2abd1253626b05880` (shared)                        | Tenant cache (slug → TenantEntry) |
| `DISPATCHER`    | Dispatch namespace `fouria-tenants` (prod) / `fouria-tenants-staging` (staging) | Tenant worker dispatch            |
| `CONTROL_QUEUE` | Queue `fouria-dispatch-control[-staging                                         | -prod]`                           | Scheduled control fan-out (producer + consumer) |

## Provisioning Environment Variables

All env vars are defined in `wrangler.jsonc` under `vars` (top-level for local) and `env.staging.vars` / `env.production.vars`. To add or update a variable:

### 1. Edit `wrangler.jsonc`

```jsonc
// workers/fouria-dispatch/wrangler.jsonc
"vars": {
  "YOUR_VAR": "value",
},
"env": {
  "staging": {
    "vars": {
      "YOUR_VAR": "staging-value",
    },
  },
  "production": {
    "vars": {
      "YOUR_VAR": "production-value",
    },
  },
}
```

> **Important:** vars are **not inherited** between environments. If a var is needed in production, it must be defined in `env.production.vars` explicitly.

### 2. Add the TypeScript type

Update `DispatchEnv` interface in `src/index.ts`:

```typescript
export interface DispatchEnv {
  // ...
  YOUR_VAR: string;
}
```

### 3. If the var is a secret (not plaintext)

Use `wrangler secret put` instead of `wrangler.jsonc`:

```bash
npx wrangler secret put YOUR_SECRET --env staging
npx wrangler secret put YOUR_SECRET --env production
```

Secrets are available as `env.YOUR_SECRET` at runtime, same as vars. They do not need to be declared in `wrangler.jsonc` but **do** need the TypeScript type in `DispatchEnv`.

### Internal control-plane authentication

`GET /_invalidate/:slug` and `POST /_control/:slug/:action` require the Worker
secret `INTERNAL_DISPATCH_INVALIDATE_TOKEN` as `Authorization: Bearer <token>`.
The Worker compares SHA-256 digests in constant time and answers `403` when the
secret is unset or the credential does not match — there is no hardcoded
fallback. `wrangler.jsonc` lists it under `secrets.required` in every
environment, and `deploy-dispatch.yml` uploads it from GitHub
(`INTERNAL_DISPATCH_INVALIDATE_TOKEN` for production,
`STAGING_INTERNAL_DISPATCH_INVALIDATE_TOKEN` for staging) before deploying.
Lerma must send the matching value; deploy the Worker first in each environment.

### 4. Deploy

```bash
# Staging
bun run deploy:staging

# Production
bun run deploy
```

## Cloudflare API Token Permissions

This project requires a Cloudflare API token (`CF_ACCESS_API_TOKEN`) with these minimum permissions:

| Permission                    | Reason                                                                  |
| ----------------------------- | ----------------------------------------------------------------------- |
| `Access:Apps & Policies:Read` | Resolve tenant slugs from Access JWT audience claims                    |
| `Access:Groups:Read`          | Resolve `Paso4 Root Users` group membership for ROOT operator detection |

The token is deployed as a worker secret and used at runtime for every request that goes through apex routing. All credentials are documented in [docs/schemas/secrets-manifest.schema.json](../../../docs/schemas/secrets-manifest.schema.json).

## Supabase

The shared Postgres database (backing lerma). Key tables and columns used by dispatch:

### clients

| Column         | Type   | Purpose                                                                                                                                                                                |
| -------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id`           | `uuid` | Primary key (used to join credit_pools/client_users)                                                                                                                                   |
| `slug`         | `TEXT` | Unique tenant identifier (from hostname)                                                                                                                                               |
| `billing_plan` | `TEXT` | Plan name (`free`, `basic`, `pro`, `enterprise`)                                                                                                                                       |
| `status`       | `TEXT` | Client status. Production routes for `active`/`production`/`error` (a health `error` never blocks); a `dev` instance routes on its own instance status even when the client is `draft` |

### credit_pools

| Column               | Type   | Purpose                           |
| -------------------- | ------ | --------------------------------- |
| `client_id`          | `uuid` | FK to clients.id                  |
| `lifetime_consumed`  | `int8` | Credits consumed by this tenant   |
| `lifetime_purchased` | `int8` | Credits purchased for this tenant |

### client_users

| Column      | Type   | Purpose                              |
| ----------- | ------ | ------------------------------------ |
| `client_id` | `uuid` | FK to clients.id (counted for seats) |

## Local Development

```bash
bun install
bun run test    # Vitest with @cloudflare/vitest-plugin
bun run typecheck
```

Tests use miniflare with ephemeral in-memory KV and mocked Supabase.
