F FourIA staging 2026.10.29 GitHub ↗

Lerma

Customer dashboard for provisioning, billing, cost analytics, and tenant management.

Cloudflare-native platform for managing isolated application provisioning, cost analytics, and billing reporting. Lerma provisions and monitors tenant applications — each running in a fully isolated environment — through a Phoenix LiveView dashboard.

Goal

Lerma is a self-serve platform that allows administrators to:

  1. Provision — deploy applications using Workers for Platforms dispatch namespaces, with each application instance running in an isolated environment
  2. Monitor — track health, usage metrics, and cost analytics across all deployed application instances
  3. Bill — manage credit-based billing, invoicing, and subscription plans for tenants

Lerma serves on its own subdomain (lerma.paso4.io) on the paso4.io zone.

Documentation map

NeedWhere
Get starteddocs/GETTING-STARTED.md (hosted: /docs/getting-started.html)
First steps (local setup)docs/FIRST-STEPS.md (hosted: /docs/first-steps.html)
API documentation (HexDocs)https://lerma.paso4.io/docs (see API documentation)
Deployment runbookDEPLOY.md
Product vision & pricingdocs/VISION.md (symlink to the monorepo vision)
UI/design tokensdocs/STYLE.md
Infrastructure as CodeInfrastructure as Code below
Environment variablesConfiguration below + docs/ENV-VARIABLES.md
AI agent conventionsAGENTS.md

Architecture

Lerma is a standalone Phoenix application (Elixir 1.20 / OTP 29) running on Hetzner VMs behind Cloudflare:

  • Phoenix LiveView — real-time dashboard UI, controllers, and JSON API under /api.
  • Ecto / PostgreSQL — the multi-tenant data layer. Production uses a dedicated Supabase PostgreSQL database per environment; local development runs PostgreSQL in Docker Compose.
  • Cloudflare Access — every browser request is gated by an Access JWT verified in LermaWeb.Plugs.CfAccessAuth (see Local Auth Setup). Invitation links (/invitations/*) plus the static assets the page needs (/assets/*, /images/*, /favicon.ico) are the deliberate exception: they are served outside Access so invitees who are not FourIA operators can review and accept an invitation.
  • Cloudflare API — Lerma provisions per-tenant Workers, AI Gateways, Access applications, R2 buckets, and service tokens at onboarding time using LERMA_API_TOKEN.
Browser
   │  HTTPS (proxied)
   ▼
Cloudflare DNS (paso4.io zone)
   │  ├── lerma.paso4.io      ──► Production Hetzner VM
   │  └── lerma-dev.paso4.io  ──► Staging Hetzner VM
   │                                  │
   │                                  ▼
   │                            Docker Compose ──► Phoenix release ──► Supabase (per-env)
   │
   └── lerma-host.paso4.io (DNS-only) ──► Production VM :22   (maintainer SSH)
       lerma-staging-host.paso4.io (DNS-only) ──► Staging VM :22

Tenant traffic is routed independently by fouria-dispatch on *.fouria.io; Lerma only hosts the dashboard on the paso4.io zone. The dashboard JSON API is served at /api on the dashboard domain — there is no separate API subdomain.

Environments

EnvironmentDomainServerImage tagDatabase
Productionlerma.paso4.ioHetzner CX23:latestSupabase (prod)
Staginglerma-dev.paso4.ioHetzner CX23:latest-preSupabase (staging)
Locallocalhost:4000Dockerdev buildpostgres:16-alpine

Each environment also publishes a DNS-only host record for maintainer SSH access (lerma-host.paso4.io, lerma-staging-host.paso4.io) — see DEPLOY.md.

Instance types and sizing

The platform default is standard-1 (4 GiB memory, 0.5 vCPU, 8 GB disk). basic is selectable but cannot host a tenant: its 1 GiB is below the OpenClaw gateway’s ~1.1 GiB steady-state RSS (~2.6 GiB peak), so it OOM-loops on first use. lite cannot even unpack the image (2 GB disk). The default used to be basic; any instance provisioned without an explicit type before that fix keep their stored type.

TypeMemoryvCPUDiskGuidance
lite0.25 GiB0.06252 GBNot usable (image does not fit).
basic1 GiB0.254 GBNot usable (below gateway RSS).
standard-14 GiB0.58 GBDefault; practical floor for an agent-running tenant.
standard-26 GiB112 GBCheap headroom (1.5× memory).
standard-38 GiB216 GBDeliberate headroom (2× memory, 4× core).
standard-412 GiB420 GBHeavy Builder workloads.

Changing an instance type is a Cloudflare rollout, not a PATCH: Lerma.Cloudflare.Containers.change_instance_type/2 creates a full_auto rollout with target_configuration, which replaces the container — plan one cold start plus an R2 restore per tenant. The Worker’s dev-gated memory watchdog reads the container’s cgroup limit and uses 75% of it (3 GiB on standard-1, 6 GiB on standard-3), so the threshold follows a resize without a Worker deploy. See docs/plans/sandbox-memory-usage.md for the full analysis.

Paso4-exclusive instances

Besides client tenants, Lerma can provision Paso4-exclusive instances: internal platform instances owned by Paso4 itself. They:

  • have no associated client and no billing plan (the billing gate is bypassed entirely),
  • keep the same per-instance dev/production environment axis (routable at paso4.fouria.io / paso4-dev.fouria.io), and
  • are reachable only by @paso4.io identities — the provisioner creates a domain-scoped Cloudflare Access group (Paso4 Exclusive Users - paso4) plus the shared root group, and the Dispatch Worker re-checks the email domain as defense-in-depth.

Provision them from Dashboard → Instances → Provision Paso4 instance. They also appear in the unified instances list (marked with a Paso4 badge) and are managed from the same instance detail page as client instances. The operator selects the environment, instance type, and an optional agent roster, and the provisioning pipeline is rendered inline on the instances page. The roster is deployed to the container as MOLTLAZY_AGENTS; Paso4 instances additionally set MOLTLAZY_PASO4_AGENTS=true, which makes the opt-in paso4-dev agent eligible. paso4-dev picks up open issues from the Paso4 GitHub organizations and the FourIA repository, using the bundled github skill (gh CLI) authenticated by the platform GITHUB_TOKEN env var.

On the data model, Paso4 instances are instances.scope = "paso4" with a nullable client_id; runs use provisioning_runs.scope and a dedicated partial unique index (one_active_run_per_paso4). See Lerma.Instances.Owner for the owner abstraction the provisioner works against.

Lerma can also authorize an operator’s ssh-ed25519 public key on dev Container applications via the optional DEV_INSTANCE_SSH_PUBLIC_KEY secret. SSH uses Wrangler and Cloudflare account access; it does not expose a public port or pass the key into a tenant Worker/container environment. See the deployment guide for setup and connection commands.

Container-backed tenant deploys

Every tenant script is uploaded into a dispatch namespace and bound to a per-tenant container application whose durable_objects.namespace_id is that script’s Sandbox Durable Object namespace. The upload metadata’s containers entry must carry the full container configuration on every deploy, not only the class name:

The provisioner also deploys two non-secret bindings used by the Sandbox v1 migration gate: WORKER_PLATFORM (from Lerma’s cf_environment, so only the staging platform may migrate — contract C1) and SANDBOX_V1_CUTOVER (default false, so unselected tenants stay on the legacy runtime byte-identically).

%{
  class_name: "Sandbox",
  scheduling_policy: "default",
  image: image,
  instance_type: instance_type,
  max_instances: max_instances
}

A class-name-only entry is enough for the first upload (it creates and container-enables the class). On an update, Cloudflare drops the class’s container enablement when the configuration is missing, so the next Durable Object instantiation throws Containers have not been enabled for this Durable Object class and the tenant can no longer start its sandbox. Because Durable Objects keep running the code they were instantiated with, this stays latent until the object is evicted — which is what happened to tenant-tarjetero-dev after the v2026.10.29 upgrade.

Lerma.Cloudflare.DispatchNamespace accepts either class names or full configuration maps, and Lerma.Cloudflare.Containers.deploy_script_and_container/1 always sends the full entry. Regression tests: test/lerma/cloudflare/dispatch_namespace_test.exs and test/lerma/cloudflare/containers_test.exs.

Image preparation and the rollout health gate

A rollout only asks Cloudflare to replace the instances; it does not mean the new image can start. Cloudflare Containers build a runtime artifact (a rootfs snapshot) per image digest, and until it exists an instance sits in starting and is marked failed with an ImagePullError after a 10-minute pull budget. The artifact is not retained indefinitely, and every CI build produces a new digest, so a freshly promoted tag usually has no artifact yet.

Two guards prevent a promotion from pointing a tenant at an image it cannot run (the 2026-11-03 staging outage: both tenants dropped to zero healthy instances and the promotion still reported success in 40 s):

  • Lerma.Cloudflare.ImagePreparation resolves the tag to its digest (short-lived pull-only registry credential) and polls POST /containers/image-preparations until the platform reports ready. Containers.deploy_script_and_container/1 runs it before the script upload (so a failure cannot trigger the atomic-deploy rollback that deletes a live tenant’s script), and the restart / instance-type rollouts run it before recreating instances. An image outside registry.cloudflare.com is skipped (the API supports the managed registry only), as is an unconfigured environment.
  • Live health, not the declared image. Containers.fetch_deployed_config/1 reports the image a rollout was asked to apply, which is set the moment the rollout is created. Containers.fetch_application_health/1 reads health.instances (plus the error count and active_rollout_id) instead. Instances.do_upgrade/3 waits for a finished rollout with a healthy instance before marking the instance running, GET /api/provisioning/upgrade/:slug exposes it as container_health, and .github/scripts/promote-staging.sh gates convergence on it and fails fast on a recorded provisioning error.

An instance already at the requested version but with no healthy instance is re-deployed rather than skipped, so re-running a promotion repairs a tenant that failed to start.

A promotion also retries error instances: POST /api/provisioning/upgrade/:slug drives every running/sleeping instance plus any instance left in error by an earlier rollout, as long as it does not already serve the requested bundle. Instances.at_requested_version?/2 verifies the live container image and the deployed script content (the requested main module must be present) before skipping, so a failed script upload — image at target, script still on the previous bundle — is repaired by re-promoting. A genuinely broken container that already serves the target bundle is still not looped. Correspondingly, the validation gate (.github/scripts/promote-validation.sh) refuses to report convergence while any instance is in error, whatever version it declares — previously an error instance was filtered out of the gate, so a promotion could report success while one tenant was broken at an old version (2026-11-06).

Choosing the sandbox runtime when deploying an instance

A new instance is deployed on one of two runtimes, chosen once, at provisioning time:

sandbox_runtimeRuntimeAvailability
legacy (default)the v0 Sandbox class (.sqsh archives) — what every existing tenant runsevery environment/platform
v1the Sandbox SDK 1.0 SandboxV1 class (DirectoryBackup records)dev environment on the staging platform only

v1 is refused for a production-environment instance (schema invariant) and off the staging platform (cf_environment == "staging"), for the same reason the conversion track is staging-only: the v1 track has no production cutover. The column is immutable — update_instance/2 and the changeset reject a change, so a runtime change is a migration, never an edit.

Surfaces:

# API — per instance
POST /api/clients/:client_id/instances
  { "instance": { "environment": "dev", "sandbox_runtime": "v1" } }
  → 202 { run_id, sandbox_runtime: "v1", ... }
  → 422 { errors: { detail: "v1 sandbox runtime is available for dev instances only ..." } }
  → 422 { errors: { detail: "v1 sandbox runtime is available on the staging platform only" } }
# Elixir — one runtime per environment (the UI sends the same shape)
Instances.create_instances(client.id, %{
  "environments" => ["dev"],
  "sandbox_runtimes" => %{"dev" => "v1"}   # a single "sandbox_runtime" still works
})

The client page’s provision form renders a runtime select per environment row (sandbox_runtimes[env]); the v1 option is disabled for production with the hint “v1 is dev-only — no production cutover”, and the instance list shows the runtime as a badge. The unified Instances dashboard (/dashboard/instances) resolves the effective runtime with Instances.runtime/1: a tenant converted from legacy to v1 keeps sandbox_runtime: "legacy" on its row, so the stable pointer decides the badge (v1), and an unreadable pointer is shown as unknown instead of claiming either runtime. Pointers are only read on the staging platform (the v1 track has no production cutover) and when the Cloudflare client is configured, so a production or local dashboard never reaches the network.

What a fresh v1 instance does

There is no conversion. The provisioner writes a durable runtime marker before the script deploy, and the Worker resolves a fresh v1 tenant from it:

s3://fouria-backup-<hostname-label><platform-suffix>/sandbox-migration/<canonical-slug>/runtime.json
{ "version": 1, "tenant": "<canonical-slug>", "runtime": "v1", "mode": "fresh",
  "provisionedAt": "<ISO8601>" }
  • Canonical slug is what the tenant Worker derives from its own env — the worker name with the dispatch prefix and -dev suffix stripped (tenant-acme-dev → acme), from Lerma.Instances.Naming.canonical_tenant_slug/1.
  • Create-only, and written as the :sandbox_runtime_marker pipeline step, i.e. immediately before :dispatch_deploy. A tenant whose marker cannot be written is never deployed (the run rolls back and the step fails).
  • The Worker also gets SANDBOX_RUNTIME ("legacy" / "v1", re-asserted on every re-deploy) and SANDBOX_V1_CUTOVER=true for v1. A v1 binding without a marker fails closed on the Worker (pending), which is why the marker is a hard step rather than a best-effort write.
  • On boot the tenant onboards inside the v1 container (no restore, no legacy state), and the idle snapshot cadence takes the first DirectoryBackup + stable.json pointer — that record becomes the recovery point for park/restore/cold boot.

The first boot is also self-healing. A born-v1 first provision can leave the SandboxV1 container application with zero instances: the DO reports running, every containerFetch times out and the gateway never boots, which used to require deprovision + re-provision. The background health check for a fresh v1 provision with no committed recovery point now recreates that application (Containers.recreate_v1_application/1) — scoped to the app, so the deployed script and its Durable Object state are untouched — and retries the probe once. An upgrade never opts in, and a tenant with a recovery point is never reconciled.

Refusals in the marker step:

OutcomeMeaning
{:ok, :existing}a matching marker is already there — idempotent retry, and a rollback leaves it alone
{:error, :already_converted}sandbox-migration/<slug>/intent.json exists: the tenant is being/was converted from legacy
{:error, {:runtime_mismatch, r}}the marker names another runtime; it is never overwritten

A rollback deletes the marker only if that run created it.

Inspecting a tenant’s runtime

# What Lerma thinks the instance runs on
psql -c "select cf_worker_name, environment, sandbox_runtime from instances"

# The authoritative marker the Worker reads (canonical slug, no `tenant-`/`-dev`)
curl -s -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT/r2/buckets/fouria-backup-acme-dev-staging/objects/sandbox-migration%2Facme%2Fruntime.json"

The column is the runtime chosen at provisioning, not necessarily what a tenant serves: a converted tenant keeps legacy there while its stable pointer makes the Worker serve v1. Resolve the effective runtime in Elixir when the answer must include conversions:

Instances.runtime(instance)   # :legacy | :v1 | :unknown
Instances.runtimes(instances) # %{instance_id => :legacy | :v1 | :unknown}

A legacy instance has no marker file — absence is the normal legacy state, and the Worker keeps serving it on the legacy class. Never create or edit a marker by hand; it selects the runtime of a live tenant.

A v1 instance is never a conversion candidate

POST /api/provisioning/migrate/<slug> (and its dry run) treats an instance with sandbox_runtime == "v1" as already v1: the dry run reports eligible: false, reason sandbox_runtime_v1, already_v1: true, and a real request answers 200 {state: "committed", already: true, runtime: "v1"} with no control call at all. Converting a tenant that is already v1 would have nothing to restore, so it is never attempted. The already_converted pointer check stays for tenants that were converted from legacy.

Migrating a legacy (v0) tenant to Sandbox v1

The v1-sandbox track can convert a tenant from the legacy Sandbox runtime (v0, .sqsh archives) to SandboxV1 (SDK 1.0, DirectoryBackup records). The conversion runs as a durable job on the tenant Worker’s SandboxV1 Durable Object (POST /control with X-FourIA-Control: migrate): the tenant’s committed legacy archive is streamed into a fresh v1 container, uploaded as a v1 record, catalogued, and promoted to the stable pointer.

Lerma only decides whether an operator may ask for it and reports why not. It never waits for a conversion: the control call enqueues and returns the job’s current state, and a separate poll reads it.

Eligibility

A run is refused, before any control request is spent, when:

Refusal (reasons)Meaning
platform_not_stagingThis Lerma is not the staging platform (cf_environment != "staging"), mirroring the Worker’s WORKER_PLATFORM === "staging" authority (contract C1).
reserved_clientThe reserved client slug never migrates (contract C2).
production_environmentOnly a client’s non-production (dev) environment instance may be the first conversion (contract S7: no existing customer is the first conversion).
no_active_instanceThe tenant has no running/sleeping instance.
instance_not_foundAn explicit --instance / instance: names a worker the client does not own.
no_legacy_recovery_pointThe tenant’s backup bucket has no stable-backup.json: there is nothing to convert.
already_convertedThe tenant already has a v1 stable pointer; the run would be a no-op (reported as committed, see below).
sandbox_runtime_v1The instance was provisioned on v1 (sandbox_runtime: "v1"); it is already v1, so there is nothing to convert (short-circuits to committed, no control call).

Warnings never block: legacy_recovery_point_unknown (the R2 read could not be verified — missing credential or transport error) and checkpoint_recommended_before_conversion (the instance is running, and the conversion restores the last committed recovery point, so checkpoint first — POST /control with X-FourIA-Control: checkpoint — to avoid losing writes made since the last snapshot).

One instance is converted per run: the dev instance. A client’s production and dev tenants are separate workers with separate backup buckets (they do not share a data plane), and the control plane is addressed per tenant slug, so the conversion always targets exactly one of them.

Trigger and poll

The conversion is asynchronous, in one direction only:

POST /api/provisioning/migrate/<slug>   → 202 {state: "queued"}   enqueue (never waits)
                                        → 202 {state: "running"}  already in flight
GET  /api/provisioning/migrate/<slug>   → 200 {state: "absent"}   no job recorded (read-only)
                                        → 202 {state: "queued"|"running"}
                                        → 200 {state: "committed", record_id}
                                        → 502 {state: "failed", error}

The POST returns as soon as the durable job records the request, so the old edge-timeout caveat no longer applies to the trigger: a 524/gateway timeout can no longer be ambiguous about whether a conversion started, and an operator does not hold a request open for the whole archive round-trip.

Only the POST starts a conversion. The GET is a read-only status read (migration-status control action): it never enqueues, never starts work and never wakes the container, so it is always safe to poll — including on a tenant that is still legacy (it answers state: "absent") or whose job vanished. A conversion is started by POST /api/provisioning/migrate/:slug, or by mix lerma.sandbox_v1.migrate, which enqueues first unless --dry-run is given.

Mix task

# Preflight only: prints the table (instance, status, environment, eligible,
# recovery point, already-v1, reasons) and stops. No control call, no container wake.
mix lerma.sandbox_v1.migrate --client acme --dry-run

# Enqueue, then poll until terminal (default: 30 minutes, every 10 seconds)
mix lerma.sandbox_v1.migrate --client acme --yes

# Enqueue only; do not wait
mix lerma.sandbox_v1.migrate --client acme --yes --wait 0

# Explore the poll cadence / budget
mix lerma.sandbox_v1.migrate --client acme --yes --wait 900 --poll-interval 5

# Target an explicit instance
mix lerma.sandbox_v1.migrate --client acme --instance tenant-acme-dev --yes

The task enqueues and then polls the read-only migration-status action, so a poll can never start a conversion. It prints each transition with the elapsed time (state=queued (elapsed 0s) → state=running (elapsed 10s) → state=committed … record_id=… duration_ms=…) and ends with a state / record id / duration summary. --wait bounds the poll loop only (configurable with :migrate_wait_timeout_ms, 30 minutes by default); the conversion keeps running in the tenant if the loop gives up.

Exit codes: 0 dry-run eligible or conversion committed, 1 refused/invalid invocation/declined confirmation/enqueue-only (--wait 0)/poll budget exhausted, 2 conversion failed, the job vanished (state=absent) or the control plane was unreachable.

API

The same preflight/trigger/poll logic is available to the provisioning API, using the same authentication as /api/provisioning/upgrade/:slug (the global PROVISIONING_API_KEY or an application key plus X-FourIA-Application):

# Dry run (no control call): 200 with the eligibility report
curl -X POST "https://lerma-dev.paso4.io/api/provisioning/migrate/acme?dry_run=1" \
  -H "Authorization: Bearer $PROVISIONING_API_KEY"

# Enqueue (202 when queued/running, 200 when already committed)
curl -X POST "https://lerma-dev.paso4.io/api/provisioning/migrate/acme" \
  -H "Authorization: Bearer $PROVISIONING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"instance": "tenant-acme-dev"}'

# Poll until terminal (read-only: this never starts a conversion)
curl "https://lerma-dev.paso4.io/api/provisioning/migrate/acme?instance=tenant-acme-dev" \
  -H "Authorization: Bearer $PROVISIONING_API_KEY"

The dry run always answers 200:

{
  "ok": true,
  "dry_run": true,
  "eligible": true,
  "reasons": [],
  "warnings": ["checkpoint_recommended_before_conversion"],
  "already_converted": false,
  "v1_record_id": null,
  "target": {"cf_worker_name": "tenant-acme-dev", "environment": "dev"},
  "instances": [
    {
      "cf_worker_name": "tenant-acme-dev",
      "status": "running",
      "environment": "dev",
      "active": true,
      "eligible": true,
      "reasons": [],
      "legacy_recovery_point": true,
      "already_converted": false,
      "v1_record_id": null
    }
  ]
}

A real run answers:

StatusBodyMeaning
202{ok: true, state: "queued"|"running", record_id: null, duration_ms}enqueued / already in flight — poll the GET
200{ok: true, state: "committed", record_id, duration_ms}the durable job committed (or the no-op short-circuit)
200{ok: true, state: "committed", already: true, record_id, duration_ms: 0}already converted; no control call was made
200{ok: true, state: "absent", record_id: null}GET only: no conversion job is recorded (read-only)
502{ok: false, state: "failed", error} (or state: "unknown" for a control-plane failure)the job failed — retrying restarts it
503{ok: false, state: "unavailable", error}no SandboxV1 binding on the tenant Worker — retry later
403{ok: false, code: "MIGRATION_NOT_ELIGIBLE", reasons: ["platform_not_staging"]}not the staging platform
409{ok: false, code: "MIGRATION_NOT_ELIGIBLE", reasons: [...]}any other preflight refusal

The GET accepts the same ?instance= selector and answers the same shapes plus 200 {state: "absent"} and 503 (the 202/200/502 rows). Both include the eligibility report (target, instances, reasons, warnings) so a poll explains the tenant as well as its state. The control bearer and the tenant token are never returned or logged.

State machine and idempotency

StateMeaningHTTP
absentNo conversion job is recorded for the tenant; the GET reports it as the truthful state.200
queuedThe durable job is recorded on the DO and will convert on its next turn.202
runningThe DO is converting (streaming, extracting, uploading the v1 record).202
committedThe v1 record is catalogued and the stable pointer names it; the gateway boots from it.200
failedThe conversion failed (error carries the bounded reason); a retry restarts it.502

The trigger (migrate) is idempotent: a committed tenant answers committed without doing work, an in-flight one answers its queued/running state, and a failed one may be retried. Because the job lives in the tenant’s Durable Object, a dropped control call is safe to repeat: every step re-reads its own recorded state, and the caller never has to keep a request open. The status read (migration-status) is a separate, read-only action: it never enqueues, so polling a tenant — even one with no job, or a job that vanished, which answers absent — can never start a conversion.

An already-converted target short-circuits before any control call: the v1 stable pointer (sandbox-migration/<tenant>/stable.json) is the decisive signal (a conversion leaves the legacy archive in R2, so legacy_recovery_point alone would report it as eligible), and both the dry run (already_converted, v1_record_id) and a real run (state: "committed", already: true) report it. When the pointer is unreadable (nil) the idempotent control call is still made: the Worker is the authority.

One-way conversion

The conversion is one-way. Once the intent is written (before the first conversion step), the tenant Worker refuses to route or serve it from legacy again (SANDBOX_V1_RUNTIME_PENDING while pending, then v1); contract C5 makes this a point of no return with no automatic rollback. Returning a converted tenant to the legacy archive is an explicit operator action outside this surface (and outside this repository’s automation), and the legacy archive is left in place precisely so that the recovery point remains portable.

Restore and rollback boundaries on a converted tenant:

  • routine snapshots, manual backup/checkpoint, freeze and resume operate on v1 records (stable.json + records/<id>.json), not on the legacy catalog; the legacy backups/<id>/data.sqsh path is never used again;
  • a v1 restore applies a catalogued record and (by default) advances the stable pointer, so a later cold boot restores it;
  • the legacy recovery point stays readable (it is what the conversion restored from), but nothing in Lerma promotes or rewrites it after conversion.

What the operator verifies afterwards

  1. GET /api/health on the tenant reports readiness (the v1 boot restores the committed record and launches the gateway; softwareLog/jobLog show progress).
  2. The tenant Worker’s runtime gate reports the converted runtime (a subsequent poll answers committed), and any other control action is refused with SANDBOX_V1_CONTROL_UNSUPPORTED until its v1 equivalent exists.
  3. The conversion’s record_id is the recovery point a cold boot restores (stable.json → records/<id>.json). The admin Backups page still reads the legacy .sqsh catalog, so it does not yet list v1 records — that read model is separate work; use the conversion state/record id (and the Worker’s /api/admin/storage/* v1 routes) to inspect v1 persistence.

Reading the recovery point (C8, S6)

Every migrate response (dry run, trigger and poll) carries a recovery_point object per instance, reported as distinct fields instead of one boolean — the two C8 conditions must be observed separately:

{
  "runtime": "v1",                     // v1 | legacy | unknown
  "recovery_point": "complete",        // complete | pointer_absent | record_absent |
                                       // record_invalid | pointer_unknown | record_unknown
  "pointer": { "record_id": "…", "committed_at": "…" },  // or null / { "unreadable": true }
  "record":  { "state": "present", "source": "catalog", "dir": "/home/openclaw",
               "format": "tar+zstd/1", "size": 7948131, "issues": [] },
  "readiness": { "state": "ready", "source": "health_monitor",
                 "observed_at": "…" }  // ready | down | unknown
}
  • pointer is the id named by sandbox-migration/<tenant>/stable.json.
  • record validates the catalogued records/<id>.json (or the pre-catalog conversion.json fallback) against the Worker’s own shape check.
  • readiness is the latest real gateway observation: HealthMonitor records up only when /api/health reports status: "running", so a sleeping tenant is never ready. An instance the monitor does not cover (a dev tenant) reports unknown — never a claim.
  • recovery_point: "complete" requires both a valid pointer/record and is reported independently of readiness; pointer-without-readiness and readiness-without-pointer are both incomplete.

The same fields render on the instance page under Sandbox v1 recovery point (only for a v1 tenant), so an operator can tell “the pointer moved” from “the gateway is actually serving”.

Upgrading a v1 tenant

A v1 tenant’s container is the …-sandbox-v1-<platform> application (a durable_object instance application), not the legacy …-sandbox-<platform> one. The legacy application is unused by the v1 runtime and its rollout state can wedge independently, so Instances.container_health/1 reads the v1 application for a v1 instance: the legacy application’s rollout_active can no longer time out a promotion that the tenant is actually serving. The promotion gate’s serving_version falls back to the instance’s recorded deployed_version for a v1 tenant (the v1 application declares no image tag — the DO reads a digest-pinned named image from the script); the authoritative “the deployed script is the requested bundle” check remains Instances.at_requested_version?/2.

Quick Start

# Phoenix dependencies + local database
docker compose up -d db
mix setup

# Run the Phoenix app
mix phx.server

Visit http://localhost:4000 for the dashboard.

API documentation

The HexDocs-format API documentation is generated with ExDoc from module docs and the README/DEPLOY/STYLE extras, and served by the Phoenix endpoint at:

/docs is served from priv/static/docs (see docs/0 in mix.exs) and stays behind Cloudflare Access like the rest of the dashboard. Generate it locally with:

mix docs          # writes priv/static/docs, open priv/static/docs/index.html

mix docs is also run during the Docker image build, so the hosted docs always match the deployed release. Add new guides by listing them under extras in mix.exs.

Local Auth Setup (Cloudflare Access)

The dashboard authenticates via Cloudflare Access JWT. In production, CF Access sits in front of the app. For local development, use a cloudflared tunnel to get real JWTs.

One-Time Setup

Create a Terraform workspace and apply the local dev configuration:

cd apps/lerma/iac/tf

# Ensure .env has TF_VAR_deploy_environment=local
# Then create and select the "local" workspace
tofu workspace new local
tofu workspace select local

# Source the .env (which has CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID, etc.)
# and apply — this only creates CF Access resources, no Hetzner/Supabase
(set +a && source ../../.env && set -a && tofu apply)

CLOUDFLARE_API_TOKEN remains Lerma’s administrative provisioning credential. Set CLOUDFLARE_AI_GATEWAY_API_KEY separately to a real AI-scoped Cloudflare credential; the tenant provisioner copies that credential to User Workers and derives their CLOUDFLARE_API_TOKEN REST binding without exposing Lerma’s token.

Then copy the output values into your .env:

tofu output -raw local_access_aud            # → CF_ACCESS_AUD
tofu output -raw local_service_token_id      # → CF_ACCESS_SERVICE_TOKEN_ID
tofu output -raw local_service_token_secret  # → CF_ACCESS_SERVICE_TOKEN_SECRET
tofu output -raw local_tunnel_token          # → CLOUDFLARED_TUNNEL_TOKEN

Daily Development

Start the tunnel in one terminal:

CLOUDFLARED_TUNNEL_TOKEN=<token> bin/local-tunnel.sh

Start the app in another:

mix phx.server

Access via https://local.lerma.paso4.io — Cloudflare Access will prompt for OAuth login.

Escape Hatch

To run without the tunnel (auth bypassed):

SKIP_AUTH=true mix phx.server

Infrastructure as Code

All Lerma infrastructure lives under apps/lerma/iac/ and has two layers:

apps/lerma/iac/
├── tf/                       # OpenTofu — infrastructure provisioning
│   ├── providers.tf          # hcloud + cloudflare + supabase providers
│   ├── main.tf               # Hetzner server, firewall, SSH keys
│   ├── supabase.tf           # Supabase PostgreSQL project (prevent_destroy)
│   ├── access.tf             # Cloudflare Access app + Paso4/Partner root groups
│   ├── variables.tf          # Input variables (see terraform.tfvars.example)
│   ├── outputs.tf            # Hand-off values for Ansible + GitHub secrets
│   ├── tests/*.tftest.hcl    # `tofu test` suites (mocked providers)
│   ├── justfile              # Task aliases (plan/apply/output/destroy per env)
│   └── .aliases              # Shell-function fallback for the justfile
└── ansible/                  # Ansible — server configuration + deployment
    ├── inventory.yml         # Production inventory (SERVER_IP)
    ├── inventory.staging.yml # Staging inventory (STAGING_SERVER_IP)
    ├── playbooks/site.yml    # Master playbook
    ├── roles/                # common, firewall, docker, app-config, app
    ├── group_vars/all.yml    # Shared playbook variables
    ├── templates/            # docker-compose.yml.j2
    └── justfile              # `just deploy-staging` / `just deploy-production`

OpenTofu (iac/tf)

OpenTofu provisions everything outside the application:

FileManages
providers.tfhcloud, cloudflare, and supabase provider configuration
main.tfHetzner server + firewall + uploaded SSH keys (env-suffixed per workspace)
supabase.tfOne Supabase PostgreSQL project per environment (prevent_destroy)
access.tfCloudflare Access applications for the dashboard + the Paso4/Partner Root groups, and the /invitations/* and static-asset (/assets/*, /images/*, /favicon.ico) bypass apps that keep invitation links public
variables.tfhcloud_token, lerma_api_token, cloudflare_account_id/zone_id, SSH keys, deploy_environment, server sizing, Supabase org/region, root email lists
outputs.tfserver_ipv4_address, database_url, host_fqdn, ansible_inventory, dashboard_access_aud, root group IDs, local tunnel/token values

Workspaces isolate state per environment (they are not git-branched):

WorkspaceEnvironmentEnv fileNotes
defaultProduction../../.envThe default workspace is production
stagingStaging../../.env.stagingSet TF_VAR_deploy_environment=staging implicitly via workspace
localLocal dev../../.envTF_VAR_deploy_environment=local; only CF Access + tunnel resources, no Hetzner/Supabase

Use the justfile (recommended) or source .aliases:

cd apps/lerma/iac/tf

just init                  # tofu init
just test                  # tofu test (all mocked suites)
just plan-production       # tofu plan   (default workspace, sources ../../.env)
just apply-staging         # tofu apply  (staging workspace, sources ../../.env.staging)
just output-production     # inspect outputs
just refresh-production    # tofu apply -refresh-only (reconcile after state drift)
just fmt-production        # tofu fmt
just help                  # list all targets

tofu test runs the suites in tests/ with mock_provider, so it needs no real credentials — run it before every IaC change.

Ansible (iac/ansible)

Ansible configures each provisioned VM and deploys the app container. Roles run in order:

RolePurpose
commonUpdate apt, install base packages (curl, gnupg, ufw)
firewallOpen ports 22, 80, 443; enable UFW with deny-incoming default
dockerInstall Docker CE + Compose plugin, start the service
app-configCopy the compose file, write .env with secrets
appLog in to GHCR, pull the image, restart the container, health-check

Inventories are environment-specific and read the server IP from an env var. Compose files: docker-compose.prod.yml (:latest, container lerma), docker-compose.staging.yml / docker-compose.pre.yml (:latest-pre, container lerma-staging).

cd apps/lerma/iac/ansible
cp .env.example .env         # fill in secrets
just deploy-staging          # deploy to lerma-dev.paso4.io
just deploy-production       # deploy to lerma.paso4.io

See DEPLOY.md for the complete OpenTofu + Ansible runbook, required GitHub secrets, and the full manual ansible-playbook invocations.

Relationship between the two layers

OpenTofu (iac/tf)  ──provisions──►  Hetzner VM + firewall + Supabase + Access
        │
        └── outputs (server IP, database_url, AUDs) ──► GitHub secrets
                                                              │
Ansible (iac/ansible) ──consumes secrets + outputs──► configures VM, deploys container

Deployment

# Staging (push to develop triggers CD after Lerma CI passes)
git push origin develop

# Production (publish a GitHub release; the pipeline promotes :latest-pre → :latest)
gh release create <tag>

See DEPLOY.md for the full deployment guide, including migrations, container lifecycle, DNS, and monitoring.

Configuration

Lerma reads its runtime configuration from environment variables in config/runtime.exs and its deployment credentials from apps/lerma/.env (IaC) and apps/lerma/iac/ansible/.env (server deploy). Never commit these files.

VariableRequiredPurpose
DATABASE_URLyesPostgreSQL (Supabase in prod; postgres:16-alpine locally)
SECRET_KEY_BASEyesPhoenix cookie/session signing key (mix phx.gen.secret)
PHX_SERVERprodStarts the Phoenix HTTP server
PHX_HOSTprodHostname used for URL generation (lerma.paso4.io)
PHX_CHECK_ORIGINprodAllowed LiveView WebSocket origins
DNS_CLUSTER_QUERYoptionalCluster discovery query
SKIP_AUTHdev onlyBypasses Cloudflare Access JWT verification

IaC and deploy credentials (see iac/ansible/.env.example and the secrets manifest) include CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_ZONE_ID, CLOUDFLARE_API_TOKEN / LERMA_API_TOKEN, CLOUDFLARE_AI_GATEWAY_API_KEY, CF_ACCESS_TEAM_DOMAIN, CF_ACCESS_AUD, CF_ACCESS_PLATFORM_AUD, GHCR_OWNER/GHCR_TOKEN, HCLOUD_TOKEN, and SUPABASE_*.

The complete cross-project env var reference is in docs/ENV-VARIABLES.md, and the machine-readable secrets manifest is docs/schemas/secrets-manifest.schema.json.

Health monitor service token

The health monitor authenticates its tenant /api/health probes with the FourIA Health Monitor Cloudflare Access service token created by the fouria IaC (apps/fouria/iac/access.tf):

VariableSource
HEALTH_MONITOR_ACCESS_CLIENT_IDtofu -chdir=apps/fouria/iac output -raw health_monitor_service_token_client_id
HEALTH_MONITOR_ACCESS_CLIENT_SECRETtofu -chdir=apps/fouria/iac output -raw health_monitor_service_token_client_secret
CF_ACCESS_HEALTH_MONITOR_TOKEN_IDtofu -chdir=apps/fouria/iac output -raw health_monitor_service_token_id (optional pin)

When the client id/secret are empty (e.g. local dev) the probes are sent unauthenticated. That is only safe when the health endpoints are not behind Access. The dashboard CD (.github/workflows/deploy-dashboard.yml) passes all three values to Ansible (as health_monitor_access_client_id / health_monitor_access_client_secret / cf_access_health_monitor_token_id), which writes them into the root-only Compose env file — so a deploy never ships a dashboard whose tenant probes are unauthenticated.

An exact-host per-tenant Access app shadows the wildcard *.fouria.io/api/health app, so the provisioner (Lerma.Cloudflare.AccessPolicy) also creates a per-tenant health application at {slug}.fouria.io/api/health that carries no role requirement — it allows any authenticated user and this service token. CF_ACCESS_HEALTH_MONITOR_TOKEN_ID pins the token id; when unset the token is resolved by name at provisioning time.

Dispatch control bearer

Lerma drives the Dispatch Worker’s control plane (POST /_control/:slug/{freeze,unfreeze,backup,checkpoint,migrate}, GET /_invalidate/:slug) with a shared bearer. The Worker has no hardcoded fallback: it compares digests in constant time against its own INTERNAL_DISPATCH_INVALIDATE_TOKEN secret and fails closed (403).

EnvironmentGitHub secretTarget
StagingSTAGING_INTERNAL_DISPATCH_INVALIDATE_TOKENfouria-dispatch-staging + staging Lerma
ProductionINTERNAL_DISPATCH_INVALIDATE_TOKENfouria-dispatch + production Lerma

Generate each with openssl rand -hex 32; never reuse a value across environments. Deploy the Dispatch Worker first, then Lerma — a mismatch answers 403 and only disables control-plane calls (cache invalidation, freeze/unfreeze, and the pre-replacement checkpoint) until both sides hold the same value.

up means the gateway is running. A probe is up only when the snapshot reports status: "running". A 200 status: "sleeping" (an idle container whose gateway is not running) records a down transition and marks the instance sleeping — alive, but not running — and does not recover a client left in error. Health is otherwise observability-only: a failed probe (a 503 stopped, a transport error, or an Access challenge) records a down transition for the uptime widget but never changes the instance or client status — tenant containers are loaded lazily, so a probe miss is not an outage, and marking the tenant error would take it out of service. Only a genuine running probe marks the instance running and recovers a client left in error by a provisioning failure.

Billing and task queues

Every provisioned tenant receives OTEL_TRACES_ENABLED (true for a production/staging instance, false for a client’s non-billable dev instance), the shared BILLING_QUEUE producer (fouria-billing / fouria-billing-staging), and a per-tenant TASKS queue (fouria-tasks-<slug>[-dev]) with the tenant script registered as its consumer. The billing queue is created/adopted at provisioning; the task queue is per-tenant because Cloudflare allows one consumer per queue. The E2E suite uses wrangler.jsonc and therefore does not exercise these Lerma-provisioned bindings.

Cost read-back (no tenant token)

Billable rows are written by the tenant Worker through its ANALYTICS binding — emission needs no credential and no token is deployed to tenants. Lerma reconciles credits from Analytics Engine with its own Cloudflare credentials, so billing is complete without any per-tenant secret.

ANALYTICS_ENGINE_API_TOKEN is therefore not provisioned by the provisioner. It remains a supported per-instance secret (instance page) for the optional dashboard “Usage & Billing” read-back: with it the worker queries the AE SQL API for its own rows; without it getCostSummary() returns an all-zero summary with available: false and the widget says usage is aggregated by the platform. There is no CLOUDFLARE_API_TOKEN fallback — that alias lacks Account Analytics:Read.

Per-client OpenCode keys

OPENCODE_API_KEY is the platform’s OpenCode management credential (a service-account key with Models: Request plus This account's API keys: Write). At provisioning, Lerma.Opencode uses it to mint a restricted (Models: Request) key per client, stores the key on the clients row, and binds it as OPENCODE_API_KEY on that client’s dev instances only. A ROOT operator can generate/rotate, reveal, revoke, or paste a key manually on the client page (/dashboard/clients/:id). When the management credential is unset, provisioning skips generation and the dev agents fall back to the primary model.

The key-management endpoint is configurable (OPENCODE_KEYS_URL, default https://opencode.ai/api/v1/keys) because OpenCode documents key permissions but not the management HTTP surface in its public Console API reference.

Private transports (OTLP and promotion)

Neither feature exposes an internet route on the tenant.

OTLP is ingested through the sandbox bridge: the container exports to http://otel.internal/v1/{traces,metrics}, the Sandbox SDK resolves that virtual host inside the Worker (TenantSandbox.outboundByHost), and the Worker still verifies OTEL_SHARED_SECRET. There is no /api/otel route and no path-scoped Access app.

Promotion uses R2, with both buckets kept separate:

  • the dev instance writes the bundle + promotion/pending.json to its own bucket;
  • the production instance receives a read-only PROMOTION_SOURCE_BUCKET binding to its sibling dev bucket (promotion_source_bindings/2 here and in TenantProvisioner), provisioned only for environment: "production";
  • production applies the pending release on boot (after the restore barrier, before the launch) or via POST /api/admin/releases/apply. The apply holds the shared persistence lease, is skipped while parked or while a maintenance window/restore is in flight, and is idempotent.

There is no PROMOTION_PUSH_TOKEN, no /api/admin/releases/import route and no machine Access bypass: the R2 binding is the authority, and a tenant can only read its own sibling’s prefix. Lerma ensures the sibling dev bucket exists before binding it, so a production-first provisioning order still deploys.

Cloudflare API Token Permissions

This project requires a Cloudflare API token (LERMA_API_TOKEN) with these minimum permissions:

PermissionReason
Workers Scripts:EditDeploy and manage tenant worker scripts
DNS:EditManage DNS records for tenant domains
Access:Apps & Policies:EditCreate and manage Cloudflare Access applications
Access:Service Tokens:EditCreate service tokens for CI/CD auth
R2:EditCreate/manage per-tenant backup buckets and read client documentation objects
AI Gateway:EditConfigure AI Gateway for unified billing
Email Sending:EditSend invitation emails via Cloudflare Email Service
Analytics:ReadQuery Analytics Engine for cost metrics
Account API Tokens:EditMint account-owned R2 S3 credentials (see below)

AI Gateway catalog and authentication

Cloudflare is the source of truth for which AI Gateways exist. The dashboard (/dashboard/ai-gateways) and provisioning selection read the account’s gateways directly from the Cloudflare API. Lerma’s ai_gateways table is only an optional policy override store — max_instances, enabled, environment, and display metadata — merged on top of the Cloudflare catalog. A gateway that exists only in Cloudflare is still listed and selectable, and is always bound by its gateway_id (never the local primary key).

Before a worker is deployed, provisioned, or upgraded, lerma ensures the bound gateway exists (create-only) so a redeploy never points at a deleted gateway. An upgrade likewise ensures the tenant’s R2 backup bucket exists (create-only, idempotent): Cloudflare rejects the worker upload (error 10085) when the BACKUP_BUCKET binding points at a missing bucket. The bucket is never deleted on an upgrade rollback — it holds tenant backups.

Zero-loss upgrades (parked checkpoint + resume)

A container replacement starts from a fresh disk, so the replacement restores the tenant from the checkpoint taken just before the rollout. @upgrade_pipeline makes that lossless:

  • :checkpoint calls Cloudflare.Containers.checkpoint(instance, park: true), which uses the Worker’s checkpoint-park control action: the tenant is drained, the stopped filesystem is snapshotted, and the tenant is left parked — mutating admin/debug requests and all container exec are refused until resume. The step is fail-closed: a checkpoint that cannot be taken aborts the upgrade before the container is replaced, because the replacement would otherwise discard writes made after the last archive. Only a sleeping instance (skipped: not-running) is exempt — it has nothing live to lose.
  • :dispatch_deploy uploads the script and rolls the image.
  • :resume (immediately after) calls Cloudflare.Containers.resume/1, which closes the park and lets the replacement container restore the checkpoint and boot the gateway. Both error paths also call resume/1 best-effort, so a failed upgrade never leaves a tenant parked; the 15-minute window TTL is the crash backstop.

The direct (bundle-only) upgrade path Instances.upgrade_instance/3 — used by the staging validation promotion API — parks with the same contract: the checkpoint is a hard gate before the worker bundle is re-uploaded, a non-parked (parked: false) or failed response aborts the upgrade, and the park is released after the rollout settles and on every error path. A sleeping instance (skipped: not-running) is never woken, so no resume is sent.

Cloudflare.Containers.checkpoint/1 remains available without park: true for callers that only need a snapshot and do not replace the container.

The park also drains live agent sessions before it stops the gateway: it switches them to permissionMode: 'read-only' and waits (bounded) for their running turn to finish, so a turn is not interrupted mid-write. The previous modes are stored outside the container and restored when the replacement container’s gateway is up (resume waits for it; the post-restore path applies them otherwise). New sessions are unaffected — the switch is per existing session, never a default. Telegram users who message during the park get a direct notice from the ingress (no AI, no gateway) and the update is acked instead of queued.

AI Gateways are created authenticated by default (authentication: true) so Cloudflare unified billing can attribute traffic to the gateway. An operator may opt into an unauthenticated BYOK gateway (tenants supply their own upstream key) only by explicitly setting authentication: false on the policy entry; the Cloudflare gateway is then created/updated to match that intent.

To repair gateways that drifted to unauthenticated (e.g. ones created before authenticated gateways were enforced), run reconciliation from an IEx/mix run session:

Lerma.AiGateways.reconcile_all_authentication()

Reconciliation only enables authentication on policy entries recorded as authentication: true; explicit BYOK entries are left untouched.

Infrastructure health checks

The instance page (/dashboard/instances/:id) exposes an Infrastructure Health panel that probes the tenant’s Cloudflare resources (AI Gateway, User Worker, R2 backup bucket, Access app/group). The AI Gateway row checks the gateway the instance is actually bound to (instances.ai_gateway_id, e.g. a shared fouria-shared-N gateway) and only falls back to the derived dedicated id (fouria-<slug><env>) for unbound/legacy instances.

The panel probes run when the page mounts. The Re-check button in the panel re-runs them on demand after an operator repairs a resource (for example reassigning a broken AI Gateway), and asks HealthMonitor to run a probe cycle immediately (instead of waiting up to HEALTH_MONITOR_INTERVAL_MS) so a recovered tenant’s instance status reconciles in the same action.

Staging Lerma healthcheck tunnel

The staging VM runs a remotely managed Cloudflare Tunnel sidecar for https://lerma-health-dev.paso4.io/health. Its ingress matches only ^/health$ and returns 404 for every other path; /health returns only ok. The Lerma dashboard and its API remain on lerma-dev.paso4.io behind their existing Access policies. The connector token is the GitHub Actions secret LERMA_STAGING_HEALTHCHECK_TUNNEL_TOKEN, written to the root-only Docker Compose environment file by Ansible; it is not passed into the Phoenix application.

Creating/updating the tunnel in the staging OpenTofu workspace requires a separate TF_VAR_healthcheck_tunnel_api_token with only Account Cloudflare Tunnel:Edit and paso4.io Zone DNS:Edit. Do not broaden the shared LERMA_API_TOKEN. After applying, retrieve the tunnel run token through the Cloudflare API and store it as the GitHub secret above; do not put the run token in Terraform outputs or source control.

This tunnel adds a health-only route and does not replace the staging dashboard’s existing proxied A record or change the VM firewall. Production has no healthcheck tunnel. The Cloudflare provider cannot delete a remote Tunnel config; rollback by removing the health DNS record and stopping the sidecar, then reset the tunnel config to the 404 catch-all or delete the tunnel through Cloudflare.

Observability and Tools sections

The instance page groups its signals under a dedicated Observability section and its actions under a dedicated Tools section.

Lerma.Instances.HealthEvents backs two widgets: LermaWeb.UptimeWidget (production only) surfaces the current up/down state and uptime over the last 24 h / 7 d instead of only logging the probe result, and LermaWeb.HealthEventsWidget renders the last 24 h as a compact time graph (up / down / unknown segments) for any instance. Hovering a segment animates it and reveals that window’s log (probe OK, or the HTTP status and error message) — there is no separate event list, and the header Inspect action opens the Infrastructure health panel. LermaWeb.WorkerTelemetryWidget sits next to them with uptime/failure ratios derived from the pushed Cloudflare traces. The connected widgets refresh every 15–30 seconds, so background transitions appear without reloading the instance page.

The two uptime numbers answer different questions: Uptime is time-weighted (seconds in up ÷ seconds in a known state; time before the first recorded check is unknown and excluded, and the widget surfaces the observed window), while Worker telemetry is request-weighted (successful root spans ÷ observed invocations, head-sampled). A sleeping tenant with no traffic, for example, lowers probe availability but produces no invocations — never “0 % uptime”.

Job activity closes the operational side of the same section. Lerma.Instances.JobEvents persists one row per instance job run in instance_job_events — backup_create (the snapshot taken before a freeze or a verified checkpoint/park), restore (the durable boot’s restore-from-recovery-point step) and gateway_start (the boot’s launch step, until /api/health answers ready). LermaWeb.JobActivityWidget renders the last 24 h as one lane per job; each run is a milestone icon placed at its start time, and hovering it reveals the status, duration, trigger, timestamp and (on failure) the error. WorkerTelemetryWidget gains a Job performance tab over the same 24 h / 7 d / 30 d windows: run count, success rate, average/p95/max duration, the last run and a bar graph of the selected job’s duration evolution.

Writes are instance-scoped and best-effort: a history row never fails the job it describes. backup_create is recorded from the control-plane response of the freeze/checkpoint; restore / gateway_start are folded from the boot and restore-progress fields of the non-passive health poll lerma already runs during provisioning (and a background poll after a manual start or restart, which also warms the gateway immediately). A cold start triggered by tenant traffic is not observed — lerma would have to poll (and wake) the instance to see it — and scheduled snapshots taken by the Dispatch cron are not observed either, so neither appears in this history. The reads never probe a tenant, so a sleeping instance is never woken to draw the timeline or the graph.

The Billable actions section embeds the client’s LermaWeb.UsageObservationWidget — credit spend per action (stacked daily chart), action log and stats. It is rendered for production instances (billable by definition) and for any instance whose client has an active billing plan (Lerma.BillingPlans.client_has_plan?/1); Paso4 instances have no client and are never billable, so they omit it.

Billable usage is production-only by contract: dev/staging instances are never billable and emit no billable-action rows (resolveInstanceDimensions → billable: false), so credits and the stacked chart cannot contain a dev series. Their infrastructure + AI cost is tracked separately as development expense (USD, Lerma.DevelopmentExpense). The Usage widget therefore carries a Production only badge and a two-tone environment legend (production = billable credits, dev = non-billable development expense) so the chart is never misread as covering both environments.

The Tools live in a sticky Swiss-knife toolbar at the top of the page, so every operational action stays one click away while the page scrolls. Blades: deploy, start/stop, environment, deployments, AI Gateway, gateway token, debug routes and deprovision. Deploy and start/stop keep a visible label; the remaining blades are icons with a tooltip and a state dot (token stored, debug routes enabled, instance frozen). Unfold opens the full tool cards — descriptions, badges and the primary buttons for each action — in a scrollable drawer, and folding hides a revealed gateway token again. The bar condenses to icons on mobile (the strip stays inside the viewport) and the drawer is capped at 55vh so it never covers the page. Infrastructure health lives on the Health events card, next to the graph it explains.

History lives in instance_health_events — one row per state change (up → down / down → up), never one per probe, so the table stays small. Both the async provisioning health check (source: "provision") and the periodic HealthMonitor (source: "monitor") write transitions through Lerma.Instances.HealthEvents; non-production instances are ignored. Uptime over a window is reconstructed from the transition stream (each state holds until the next event).

Client documentation widget

The client profile (ClientLive.Show) embeds LermaWeb.ClientDocsWidget: a source tree on the left and a live preview on the right.

Sources come from Lerma.ClientDocs and are read directly from R2 through the Cloudflare REST API (the same LERMA_API_TOKEN, which needs R2 object read) — the dashboard never wakes a tenant container and never handles R2 S3 credentials:

SourceBucket / prefix
Proposalglobal leads bucket (default paso4-leads), <client_slug>/
Client docsthe instance’s BACKUP_BUCKET, customers/<client_slug>/docs/
Client docs devthe dev instance’s BACKUP_BUCKET, customers/<client_slug>/docs/

The widget is lazy: opening the client profile renders a Load documentation placeholder and performs no R2 request. Clicking it lists the sources and loads the default document; live polling starts only after that first load, so the client page never waits for documentation.

The tree shows folders, file-type icons and thumbnails for image files. By default it lists only the client documents operators asked for — Word (.doc/.docx), PDF, XLSX/XLS, CSV and PowerPoint (.ppt/.pptx) — and hides other artifacts (Markdown, images, SVG, JSON, diagrams, …). The tree footer exposes Show all files (with the hidden count) for the full folder.

Markdown is rendered by Lerma.ClientDocs.Markdown (a small, XSS-safe, dependency-free renderer: headings, lists, tables, code, links, images; source HTML is always escaped). PDFs use the browser’s native viewer, CSV/text render in a scrollable code panel, images render inline, and Word/PowerPoint/ spreadsheet binaries offer an open/download link.

The documentation widget has a fixed height (42rem); the tree and preview fill the remaining space and scroll independently (overscroll-contain), so long proposals never stretch the client page. On narrow screens the two panels split the same fixed area vertically. Live re-lists the active source and re-fetches the selected document every 20 s; Paused stops the polling. The preview’s Open action and inline assets use /dashboard/clients/:id/docs/raw/:source?path=…, which is behind the dashboard Access session. Active content (HTML/SVG) is forced to a download with a sandbox CSP, paths are validated against traversal, and raw responses are no-store so uploads appear immediately.

R2 Backup Credentials

The fouria User Worker signs R2 presigned URLs for Sandbox SDK backups using R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY (plus CLOUDFLARE_ACCOUNT_ID and BACKUP_BUCKET_NAME). Cloudflare derives these S3 credentials from a regular API token: Access Key ID = token id, Secret Access Key = SHA-256 of the token value. Lerma can mint one via API:

CLOUDFLARE_API_TOKEN=<parent> CLOUDFLARE_ACCOUNT_ID=<acct> \
  mix lerma.r2_token --all-buckets --write ../../apps/fouria/.dev.vars

--all-buckets grants R2 object read/write across every bucket (covers the dynamic per-tenant fouria-backup-* buckets). Without it, pass the exact bucket names instead. The task also supports --name and --write <dotenv-path>.

The parent token must carry the Account → Account API Tokens → Edit permission (the account-scoped LERMA_API_TOKEN qualifies). The task creates an account-owned token — the right model for a platform R2 backup credential, since it survives individual user changes. Rotate R2 credentials on demand via this task.

The credential is mandatory

Since the backup credential is required by the Sandbox SDK on every tenant worker, lerma now treats it as mandatory: a provisioning run that cannot resolve one fails and rolls back (it no longer logs a warning and deploys a tenant that silently cannot back up).

The credential is selected by strategy (config :lerma, :r2_token_strategy, env R2_TOKEN_STRATEGY, default :auto):

StrategyBehaviour
:auto (default)Uses the known account token limit (:r2_token_max_per_account) to choose: <= 250 → :shared, > 250 or unknown → :per_instance.
:per_instanceMints a dedicated R2 S3 token scoped to the tenant’s backup bucket at instance creation (Lerma.Provisioning.R2Credentials). Persisted on the instance record.
:sharedReuses the lerma-level R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY pair from the environment.
  • Cloudflare exposes no API for the account’s maximum token count (GET /accounts/{account_id}/tokens lists existing tokens only), so the limit defaults to unknown, which selects :per_instance. Pin a known limit with R2_TOKEN_MAX_PER_ACCOUNT (or config :lerma, :r2_token_max_per_account).
  • Generated per-instance credentials are stored on the instance (r2_access_key_id, r2_secret_access_key, r2_credential_source) so redeploys/upgrades reuse the same credential; a token generated for a failed brand-new provisioning is deleted during rollback.
  • Existing instances provisioned before this change have no stored credential; the next upgrade/redeploy backfills one.
  • When the effective strategy is :shared, both R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY must be set in the lerma environment — otherwise provisioning fails with :missing_shared_r2_credentials.

Worker deployments and Durable Object migrations

A tenant Worker’s Durable Object classes (Sandbox, ActivityScheduler, TelegramIngress, plus BackupWatch while the change-watch flag is on) are declared through a Cloudflare migration, which Cloudflare accepts only once per class — error 10074 (Cannot apply new-sqlite-class migration to class '…' that is already depended on by existing Durable Objects) means the class already has migration history.

The class set is not derived from the script’s bindings, because the two can diverge:

  • an upgrade redeploy that binds fewer classes leaves the rest migrated but unbound (the upgrade path now re-declares the full DO binding set, carrying the existing namespace_id per class);
  • a rollback/cleanup that deletes the script leaves its classes (and Durable Objects) behind. Provisioning cleanup and a failed atomic deploy therefore tear the classes down first (delete_durable_object_namespaces/2), except on an upgrade rollback, where the Durable Objects hold the live tenant’s state.

Both upgrade entry points — the direct Instances.upgrade_instance/3 and the dashboard/API TenantProvisioner.upgrade_instance/3 pipeline — pass preserve_do_namespaces: true, so a failed container binding never deletes the script that routes a live tenant. A fresh provision passes false and still rolls the just-uploaded script (and its DO classes) back.

As a backstop (Lerma.Cloudflare.DispatchNamespace), an upload rejected with 10074 names the offending class and is retried without it, so a tenant that already regressed into this state recovers without manual Cloudflare intervention. The retry logs DO classes already migrated for <ns>/<script>.

Testing

# Phoenix tests
mix test

# ExDoc generation
mix docs

# Full repository pre-commit (run from the repository root)
cd ../..
mix precommit

Project Structure

lerma/
├── lib/
│   ├── lerma/                  # Business logic (Ecto/PostgreSQL)
│   └── lerma_web/              # Web layer (LiveView, controllers, plugs)
├── assets/                     # Phoenix frontend (JS/CSS)
├── config/                     # Phoenix config (runtime.exs, prod.exs)
├── priv/
│   ├── repo/migrations/        # Ecto migrations
│   └── static/docs/            # ExDoc output served at /docs (generated)
├── test/                       # Phoenix tests
├── iac/                        # Infrastructure as Code
│   ├── tf/                     # OpenTofu (Hetzner, DNS, Supabase, Access)
│   └── ansible/                # Ansible (server config, deployment)
├── bin/local-tunnel.sh         # cloudflared tunnel for local CF Access auth
├── Dockerfile                  # Elixir release image (also builds /docs)
├── docker-compose.yml          # Local dev (PostgreSQL only)
├── docker-compose.prod.yml     # Production compose
├── docker-compose.pre.yml      # Staging compose
└── mix.exs                     # Elixir project + ExDoc definition