Lerma
Customer dashboard for provisioning, billing, cost analytics, and tenant management.
Cloudflare-native platform for managing isolated application provisioning, cost analytics, and billing reporting. Lerma provisions and monitors tenant applications — each running in a fully isolated environment — through a Phoenix LiveView dashboard.
Goal
Lerma is a self-serve platform that allows administrators to:
- Provision — deploy applications using Workers for Platforms dispatch namespaces, with each application instance running in an isolated environment
- Monitor — track health, usage metrics, and cost analytics across all deployed application instances
- Bill — manage credit-based billing, invoicing, and subscription plans for tenants
Lerma serves on its own subdomain (lerma.paso4.io) on the paso4.io zone.
Documentation map
| Need | Where |
|---|---|
| Get started | docs/GETTING-STARTED.md (hosted: /docs/getting-started.html) |
| First steps (local setup) | docs/FIRST-STEPS.md (hosted: /docs/first-steps.html) |
| API documentation (HexDocs) | https://lerma.paso4.io/docs (see API documentation) |
| Deployment runbook | DEPLOY.md |
| Product vision & pricing | docs/VISION.md (symlink to the monorepo vision) |
| UI/design tokens | docs/STYLE.md |
| Infrastructure as Code | Infrastructure as Code below |
| Environment variables | Configuration below + docs/ENV-VARIABLES.md |
| AI agent conventions | AGENTS.md |
Architecture
Lerma is a standalone Phoenix application (Elixir 1.20 / OTP 29) running on Hetzner VMs behind Cloudflare:
- Phoenix LiveView — real-time dashboard UI, controllers, and JSON API under
/api. - Ecto / PostgreSQL — the multi-tenant data layer. Production uses a dedicated Supabase PostgreSQL database per environment; local development runs PostgreSQL in Docker Compose.
- Cloudflare Access — every browser request is gated by an Access JWT verified in
LermaWeb.Plugs.CfAccessAuth(see Local Auth Setup). Invitation links (/invitations/*) plus the static assets the page needs (/assets/*,/images/*,/favicon.ico) are the deliberate exception: they are served outside Access so invitees who are not FourIA operators can review and accept an invitation. - Cloudflare API — Lerma provisions per-tenant Workers, AI Gateways, Access applications, R2 buckets, and service tokens at onboarding time using
LERMA_API_TOKEN.
Browser
│ HTTPS (proxied)
▼
Cloudflare DNS (paso4.io zone)
│ ├── lerma.paso4.io ──► Production Hetzner VM
│ └── lerma-dev.paso4.io ──► Staging Hetzner VM
│ │
│ ▼
│ Docker Compose ──► Phoenix release ──► Supabase (per-env)
│
└── lerma-host.paso4.io (DNS-only) ──► Production VM :22 (maintainer SSH)
lerma-staging-host.paso4.io (DNS-only) ──► Staging VM :22
Tenant traffic is routed independently by fouria-dispatch on *.fouria.io; Lerma only hosts the dashboard on the paso4.io zone. The dashboard JSON API is served at /api on the dashboard domain — there is no separate API subdomain.
Environments
| Environment | Domain | Server | Image tag | Database |
|---|---|---|---|---|
| Production | lerma.paso4.io | Hetzner CX23 | :latest | Supabase (prod) |
| Staging | lerma-dev.paso4.io | Hetzner CX23 | :latest-pre | Supabase (staging) |
| Local | localhost:4000 | Docker | dev build | postgres:16-alpine |
Each environment also publishes a DNS-only host record for maintainer SSH access (lerma-host.paso4.io, lerma-staging-host.paso4.io) — see DEPLOY.md.
Paso4-exclusive instances
Besides client tenants, Lerma can provision Paso4-exclusive instances: internal platform instances owned by Paso4 itself. They:
- have no associated client and no billing plan (the billing gate is bypassed entirely),
- keep the same per-instance
dev/productionenvironment axis (routable atpaso4.fouria.io/paso4-dev.fouria.io), and - are reachable only by
@paso4.ioidentities — the provisioner creates a domain-scoped Cloudflare Access group (Paso4 Exclusive Users - paso4) plus the shared root group, and the Dispatch Worker re-checks the email domain as defense-in-depth.
Provision them from Dashboard → Instances → Provision Paso4 instance. They
also appear in the unified instances list (marked with a Paso4 badge) and are
managed from the same instance detail page as client instances. The operator
selects the environment, instance type, and an optional agent roster, and the
provisioning pipeline is rendered inline on the instances page. The roster is
deployed to the
container as MOLTLAZY_AGENTS; Paso4 instances additionally set
MOLTLAZY_PASO4_AGENTS=true, which makes the opt-in paso4-dev agent
eligible. paso4-dev picks up open issues from the Paso4 GitHub organizations
and the FourIA repository, using the bundled github skill (gh CLI)
authenticated by the platform GITHUB_TOKEN env var.
On the data model, Paso4 instances are instances.scope = "paso4" with a
nullable client_id; runs use provisioning_runs.scope and a dedicated
partial unique index (one_active_run_per_paso4). See
Lerma.Instances.Owner for the owner abstraction the provisioner works against.
Lerma can also authorize an operator’s ssh-ed25519 public key on dev
Container applications via the optional DEV_INSTANCE_SSH_PUBLIC_KEY secret.
SSH uses Wrangler and Cloudflare account access; it does not expose a public
port or pass the key into a tenant Worker/container environment. See the
deployment guide for setup and connection
commands.
Quick Start
# Phoenix dependencies + local database
docker compose up -d db
mix setup
# Run the Phoenix app
mix phx.server
Visit http://localhost:4000 for the dashboard.
API documentation
The HexDocs-format API documentation is generated with ExDoc from module docs and the README/DEPLOY/STYLE extras, and served by the Phoenix endpoint at:
- Production: https://lerma.paso4.io/docs
- Staging: https://lerma-dev.paso4.io/docs
/docs is served from priv/static/docs (see docs/0 in mix.exs) and stays behind Cloudflare Access like the rest of the dashboard. Generate it locally with:
mix docs # writes priv/static/docs, open priv/static/docs/index.html
mix docs is also run during the Docker image build, so the hosted docs always match the deployed release. Add new guides by listing them under extras in mix.exs.
Local Auth Setup (Cloudflare Access)
The dashboard authenticates via Cloudflare Access JWT. In production, CF Access sits in front of the app. For local development, use a cloudflared tunnel to get real JWTs.
One-Time Setup
Create a Terraform workspace and apply the local dev configuration:
cd apps/lerma/iac/tf
# Ensure .env has TF_VAR_deploy_environment=local
# Then create and select the "local" workspace
tofu workspace new local
tofu workspace select local
# Source the .env (which has CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID, etc.)
# and apply — this only creates CF Access resources, no Hetzner/Supabase
(set +a && source ../../.env && set -a && tofu apply)
CLOUDFLARE_API_TOKEN remains Lerma’s administrative provisioning credential. Set CLOUDFLARE_AI_GATEWAY_API_KEY separately to a real AI-scoped Cloudflare credential; the tenant provisioner copies that credential to User Workers and derives their CLOUDFLARE_API_TOKEN REST binding without exposing Lerma’s token.
Then copy the output values into your .env:
tofu output -raw local_access_aud # → CF_ACCESS_AUD
tofu output -raw local_service_token_id # → CF_ACCESS_SERVICE_TOKEN_ID
tofu output -raw local_service_token_secret # → CF_ACCESS_SERVICE_TOKEN_SECRET
tofu output -raw local_tunnel_token # → CLOUDFLARED_TUNNEL_TOKENDaily Development
Start the tunnel in one terminal:
CLOUDFLARED_TUNNEL_TOKEN=<token> bin/local-tunnel.sh
Start the app in another:
mix phx.server
Access via https://local.lerma.paso4.io — Cloudflare Access will prompt for OAuth login.
Escape Hatch
To run without the tunnel (auth bypassed):
SKIP_AUTH=true mix phx.serverInfrastructure as Code
All Lerma infrastructure lives under apps/lerma/iac/ and has two layers:
apps/lerma/iac/
├── tf/ # OpenTofu — infrastructure provisioning
│ ├── providers.tf # hcloud + cloudflare + supabase providers
│ ├── main.tf # Hetzner server, firewall, SSH keys
│ ├── supabase.tf # Supabase PostgreSQL project (prevent_destroy)
│ ├── access.tf # Cloudflare Access app + Paso4/Partner root groups
│ ├── variables.tf # Input variables (see terraform.tfvars.example)
│ ├── outputs.tf # Hand-off values for Ansible + GitHub secrets
│ ├── tests/*.tftest.hcl # `tofu test` suites (mocked providers)
│ ├── justfile # Task aliases (plan/apply/output/destroy per env)
│ └── .aliases # Shell-function fallback for the justfile
└── ansible/ # Ansible — server configuration + deployment
├── inventory.yml # Production inventory (SERVER_IP)
├── inventory.staging.yml # Staging inventory (STAGING_SERVER_IP)
├── playbooks/site.yml # Master playbook
├── roles/ # common, firewall, docker, app-config, app
├── group_vars/all.yml # Shared playbook variables
├── templates/ # docker-compose.yml.j2
└── justfile # `just deploy-staging` / `just deploy-production`OpenTofu (iac/tf)
OpenTofu provisions everything outside the application:
| File | Manages |
|---|---|
providers.tf | hcloud, cloudflare, and supabase provider configuration |
main.tf | Hetzner server + firewall + uploaded SSH keys (env-suffixed per workspace) |
supabase.tf | One Supabase PostgreSQL project per environment (prevent_destroy) |
access.tf | Cloudflare Access applications for the dashboard + the Paso4/Partner Root groups, and the /invitations/* and static-asset (/assets/*, /images/*, /favicon.ico) bypass apps that keep invitation links public |
variables.tf | hcloud_token, lerma_api_token, cloudflare_account_id/zone_id, SSH keys, deploy_environment, server sizing, Supabase org/region, root email lists |
outputs.tf | server_ipv4_address, database_url, host_fqdn, ansible_inventory, dashboard_access_aud, root group IDs, local tunnel/token values |
Workspaces isolate state per environment (they are not git-branched):
| Workspace | Environment | Env file | Notes |
|---|---|---|---|
default | Production | ../../.env | The default workspace is production |
staging | Staging | ../../.env.staging | Set TF_VAR_deploy_environment=staging implicitly via workspace |
local | Local dev | ../../.env | TF_VAR_deploy_environment=local; only CF Access + tunnel resources, no Hetzner/Supabase |
Use the justfile (recommended) or source .aliases:
cd apps/lerma/iac/tf
just init # tofu init
just test # tofu test (all mocked suites)
just plan-production # tofu plan (default workspace, sources ../../.env)
just apply-staging # tofu apply (staging workspace, sources ../../.env.staging)
just output-production # inspect outputs
just refresh-production # tofu apply -refresh-only (reconcile after state drift)
just fmt-production # tofu fmt
just help # list all targets
tofu test runs the suites in tests/ with mock_provider, so it needs no real credentials — run it before every IaC change.
Ansible (iac/ansible)
Ansible configures each provisioned VM and deploys the app container. Roles run in order:
| Role | Purpose |
|---|---|
common | Update apt, install base packages (curl, gnupg, ufw) |
firewall | Open ports 22, 80, 443; enable UFW with deny-incoming default |
docker | Install Docker CE + Compose plugin, start the service |
app-config | Copy the compose file, write .env with secrets |
app | Log in to GHCR, pull the image, restart the container, health-check |
Inventories are environment-specific and read the server IP from an env var. Compose files: docker-compose.prod.yml (:latest, container lerma), docker-compose.staging.yml / docker-compose.pre.yml (:latest-pre, container lerma-staging).
cd apps/lerma/iac/ansible
cp .env.example .env # fill in secrets
just deploy-staging # deploy to lerma-dev.paso4.io
just deploy-production # deploy to lerma.paso4.io
See DEPLOY.md for the complete OpenTofu + Ansible runbook, required GitHub secrets, and the full manual ansible-playbook invocations.
Relationship between the two layers
OpenTofu (iac/tf) ──provisions──► Hetzner VM + firewall + Supabase + Access
│
└── outputs (server IP, database_url, AUDs) ──► GitHub secrets
│
Ansible (iac/ansible) ──consumes secrets + outputs──► configures VM, deploys containerDeployment
# Staging (push to develop triggers CD after Lerma CI passes)
git push origin develop
# Production (publish a GitHub release; the pipeline promotes :latest-pre → :latest)
gh release create <tag>
See DEPLOY.md for the full deployment guide, including migrations, container lifecycle, DNS, and monitoring.
Configuration
Lerma reads its runtime configuration from environment variables in config/runtime.exs and its deployment credentials from apps/lerma/.env (IaC) and apps/lerma/iac/ansible/.env (server deploy). Never commit these files.
| Variable | Required | Purpose |
|---|---|---|
DATABASE_URL | yes | PostgreSQL (Supabase in prod; postgres:16-alpine locally) |
SECRET_KEY_BASE | yes | Phoenix cookie/session signing key (mix phx.gen.secret) |
PHX_SERVER | prod | Starts the Phoenix HTTP server |
PHX_HOST | prod | Hostname used for URL generation (lerma.paso4.io) |
PHX_CHECK_ORIGIN | prod | Allowed LiveView WebSocket origins |
DNS_CLUSTER_QUERY | optional | Cluster discovery query |
SKIP_AUTH | dev only | Bypasses Cloudflare Access JWT verification |
IaC and deploy credentials (see iac/ansible/.env.example and the secrets manifest) include CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_ZONE_ID, CLOUDFLARE_API_TOKEN / LERMA_API_TOKEN, CLOUDFLARE_AI_GATEWAY_API_KEY, ANALYTICS_ENGINE_API_TOKEN, CF_ACCESS_TEAM_DOMAIN, CF_ACCESS_AUD, CF_ACCESS_PLATFORM_AUD, GHCR_OWNER/GHCR_TOKEN, HCLOUD_TOKEN, and SUPABASE_*.
The complete cross-project env var reference is in docs/ENV-VARIABLES.md, and the machine-readable secrets manifest is docs/schemas/secrets-manifest.schema.json.
Health monitor service token
The health monitor authenticates its tenant /api/health probes with the
FourIA Health Monitor Cloudflare Access service token created by the fouria
IaC (apps/fouria/iac/access.tf):
| Variable | Source |
|---|---|
HEALTH_MONITOR_ACCESS_CLIENT_ID | tofu -chdir=apps/fouria/iac output -raw health_monitor_service_token_client_id |
HEALTH_MONITOR_ACCESS_CLIENT_SECRET | tofu -chdir=apps/fouria/iac output -raw health_monitor_service_token_client_secret |
CF_ACCESS_HEALTH_MONITOR_TOKEN_ID | tofu -chdir=apps/fouria/iac output -raw health_monitor_service_token_id (optional pin) |
When the client id/secret are empty (e.g. local dev) the probes are sent
unauthenticated. That is only safe when the health endpoints are not behind
Access. The dashboard CD (.github/workflows/deploy-dashboard.yml) passes all
three values to Ansible (as health_monitor_access_client_id /
health_monitor_access_client_secret / cf_access_health_monitor_token_id),
which writes them into the root-only Compose env file — so a deploy never ships
a dashboard whose tenant probes are unauthenticated.
An exact-host per-tenant Access app shadows the wildcard
*.fouria.io/api/health app, so the provisioner
(Lerma.Cloudflare.AccessPolicy) also creates a per-tenant health application at
{slug}.fouria.io/api/health that carries no role requirement — it allows any
authenticated user and this service token. CF_ACCESS_HEALTH_MONITOR_TOKEN_ID
pins the token id; when unset the token is resolved by name at provisioning time.
up means the gateway is running. A probe is up only when the snapshot
reports status: "running". A 200 status: "sleeping" (an idle container whose
gateway is not running) records a down transition and marks the instance
sleeping — alive, but not running — and does not recover a client left in
error. Health is otherwise observability-only: a failed probe (a 503
stopped, a transport error, or an Access challenge) records a down transition
for the uptime widget but never changes the instance or client status —
tenant containers are loaded lazily, so a probe miss is not an outage, and
marking the tenant error would take it out of service. Only a genuine running
probe marks the instance running and recovers a client left in error by a
provisioning failure.
Tenant cost-summary token
The dashboard Usage & Billing widget reads GET /api/admin/costs/v2/summary,
which aggregates the tenant’s emitted billable-action rows through the Analytics
Engine SQL API. Lerma reads ANALYTICS_ENGINE_API_TOKEN from its environment and
deploys it as a per-tenant secret_text binding (see tenant_provisioner.ex
@platform_secrets) so each tenant Worker can read its own rows.
The token must be scoped to Account Analytics:Read and must be distinct
from LERMA_API_TOKEN — the provisioner distributes this credential to every
tenant, so Lerma’s administrative token must never be used. It is optional: when
unset the worker falls back to CLOUDFLARE_API_TOKEN (the AI Gateway alias,
which lacks Analytics read), the AE SQL query returns 403, and the widget shows
zeros without erroring.
Cloudflare API Token Permissions
This project requires a Cloudflare API token (LERMA_API_TOKEN) with these minimum permissions:
| Permission | Reason |
|---|---|
Workers Scripts:Edit | Deploy and manage tenant worker scripts |
DNS:Edit | Manage DNS records for tenant domains |
Access:Apps & Policies:Edit | Create and manage Cloudflare Access applications |
Access:Service Tokens:Edit | Create service tokens for CI/CD auth |
R2:Edit | Create/manage per-tenant backup buckets and read client documentation objects |
AI Gateway:Edit | Configure AI Gateway for unified billing |
Email Sending:Edit | Send invitation emails via Cloudflare Email Service |
Analytics:Read | Query Analytics Engine for cost metrics |
Account API Tokens:Edit | Mint account-owned R2 S3 credentials (see below) |
AI Gateway catalog and authentication
Cloudflare is the source of truth for which AI Gateways exist. The dashboard
(/dashboard/ai-gateways) and provisioning selection read the account’s
gateways directly from the Cloudflare API. Lerma’s ai_gateways table is only an
optional policy override store — max_instances, enabled, environment,
and display metadata — merged on top of the Cloudflare catalog. A gateway that
exists only in Cloudflare is still listed and selectable, and is always bound by
its gateway_id (never the local primary key).
Before a worker is deployed, provisioned, or upgraded, lerma ensures the bound
gateway exists (create-only) so a redeploy never points at a deleted gateway.
An upgrade likewise ensures the tenant’s R2 backup bucket exists (create-only,
idempotent): Cloudflare rejects the worker upload (error 10085) when the
BACKUP_BUCKET binding points at a missing bucket. The bucket is never deleted
on an upgrade rollback — it holds tenant backups.
AI Gateways are created authenticated by default (authentication: true) so
Cloudflare unified billing can attribute traffic to the gateway. An operator may
opt into an unauthenticated BYOK gateway (tenants supply their own upstream
key) only by explicitly setting authentication: false on the policy entry; the
Cloudflare gateway is then created/updated to match that intent.
To repair gateways that drifted to unauthenticated (e.g. ones created before
authenticated gateways were enforced), run reconciliation from an IEx/mix run
session:
Lerma.AiGateways.reconcile_all_authentication()
Reconciliation only enables authentication on policy entries recorded as
authentication: true; explicit BYOK entries are left untouched.
Infrastructure health checks
The instance page (/dashboard/instances/:id) exposes an Infrastructure
Health panel that probes the tenant’s Cloudflare resources (AI Gateway, User
Worker, R2 backup bucket, Access app/group). The AI Gateway row checks the
gateway the instance is actually bound to (instances.ai_gateway_id, e.g. a
shared fouria-shared-N gateway) and only falls back to the derived dedicated id
(fouria-<slug><env>) for unbound/legacy instances.
The panel probes run when the page mounts. The Re-check button in the panel
re-runs them on demand after an operator repairs a resource (for example
reassigning a broken AI Gateway), and asks HealthMonitor to run a probe cycle
immediately (instead of waiting up to HEALTH_MONITOR_INTERVAL_MS) so a
recovered tenant’s instance status reconciles in the same action.
Staging Lerma healthcheck tunnel
The staging VM runs a remotely managed Cloudflare Tunnel sidecar for
https://lerma-health-dev.paso4.io/health. Its ingress matches only ^/health$
and returns 404 for every other path; /health returns only ok. The Lerma
dashboard and its API remain on lerma-dev.paso4.io behind their existing Access
policies. The connector token is the GitHub Actions secret
LERMA_STAGING_HEALTHCHECK_TUNNEL_TOKEN, written to the root-only Docker Compose
environment file by Ansible; it is not passed into the Phoenix application.
Creating/updating the tunnel in the staging OpenTofu workspace requires a
separate TF_VAR_healthcheck_tunnel_api_token with only Account Cloudflare
Tunnel:Edit and paso4.io Zone DNS:Edit. Do not broaden the shared
LERMA_API_TOKEN. After applying, retrieve the tunnel run token through the
Cloudflare API and store it as the GitHub secret above; do not put the run token
in Terraform outputs or source control.
This tunnel adds a health-only route and does not replace the staging dashboard’s existing proxied A record or change the VM firewall. Production has no healthcheck tunnel. The Cloudflare provider cannot delete a remote Tunnel config; rollback by removing the health DNS record and stopping the sidecar, then reset the tunnel config to the 404 catch-all or delete the tunnel through Cloudflare.
Observability and Tools sections
The instance page groups its signals under a dedicated Observability section and its actions under a dedicated Tools section.
Lerma.Instances.HealthEvents backs two widgets: LermaWeb.UptimeWidget
(production only) surfaces the current up/down state and uptime over the last
24 h / 7 d instead of only logging the probe result, and
LermaWeb.HealthEventsWidget renders the last 24 h as a compact time graph
(up / down / unknown segments) for any instance. Hovering a segment animates it
and reveals that window’s log (probe OK, or the HTTP status and error message) —
there is no separate event list, and the header Inspect action opens the
Infrastructure health panel. LermaWeb.WorkerTelemetryWidget sits next to them
with uptime/failure ratios derived from the pushed Cloudflare traces. The
connected widgets refresh every 15–30 seconds, so background transitions appear
without reloading the instance page.
The two uptime numbers answer different questions: Uptime is time-weighted
(seconds in up ÷ seconds in a known state; time before the first recorded
check is unknown and excluded, and the widget surfaces the observed window), while
Worker telemetry is request-weighted (successful root spans ÷ observed
invocations, head-sampled). A sleeping tenant with no traffic, for example,
lowers probe availability but produces no invocations — never “0 % uptime”.
The Billable actions section embeds the client’s
LermaWeb.UsageObservationWidget — credit spend per action (stacked daily
chart), action log and stats. It is rendered for production instances
(billable by definition) and for any instance whose client has an active
billing plan (Lerma.BillingPlans.client_has_plan?/1); Paso4 instances have no
client and are never billable, so they omit it.
Billable usage is production-only by contract: dev/staging instances are
never billable and emit no billable-action rows (resolveInstanceDimensions →
billable: false), so credits and the stacked chart cannot contain a dev
series. Their infrastructure + AI cost is tracked separately as development
expense (USD, Lerma.DevelopmentExpense). The Usage widget therefore carries
a Production only badge and a two-tone environment legend (production =
billable credits, dev = non-billable development expense) so the chart is never
misread as covering both environments.
The Tools live in a sticky Swiss-knife toolbar at the top of the page, so every operational action stays one click away while the page scrolls. Blades: deploy, start/stop, environment, deployments, AI Gateway, gateway token, debug routes and deprovision. Deploy and start/stop keep a visible label; the remaining blades are icons with a tooltip and a state dot (token stored, debug routes enabled, instance frozen). Unfold opens the full tool cards — descriptions, badges and the primary buttons for each action — in a scrollable drawer, and folding hides a revealed gateway token again. The bar condenses to icons on mobile (the strip stays inside the viewport) and the drawer is capped at 55vh so it never covers the page. Infrastructure health lives on the Health events card, next to the graph it explains.
History lives in instance_health_events — one row per state change
(up → down / down → up), never one per probe, so the table stays small. Both the
async provisioning health check (source: "provision") and the periodic
HealthMonitor (source: "monitor") write transitions through
Lerma.Instances.HealthEvents; non-production instances are ignored. Uptime
over a window is reconstructed from the transition stream (each state holds
until the next event).
Client documentation widget
The client profile (ClientLive.Show) embeds LermaWeb.ClientDocsWidget: a
source tree on the left and a live preview on the right.
Sources come from Lerma.ClientDocs and are read directly from R2 through the
Cloudflare REST API (the same LERMA_API_TOKEN, which needs R2 object read)
— the dashboard never wakes a tenant container and never handles R2 S3
credentials:
| Source | Bucket / prefix |
|---|---|
| Proposal | global leads bucket (default paso4-leads), <client_slug>/ |
| Client docs | the instance’s BACKUP_BUCKET, customers/<client_slug>/docs/ |
| Client docs dev | the dev instance’s BACKUP_BUCKET, customers/<client_slug>/docs/ |
The widget is lazy: opening the client profile renders a Load documentation placeholder and performs no R2 request. Clicking it lists the sources and loads the default document; live polling starts only after that first load, so the client page never waits for documentation.
The tree shows folders, file-type icons and thumbnails for image files.
By default it lists only the client documents operators asked for — Word
(.doc/.docx), PDF, XLSX/XLS, CSV and PowerPoint (.ppt/.pptx) — and
hides other artifacts (Markdown, images, SVG, JSON, diagrams, …). The tree
footer exposes Show all files (with the hidden count) for the full folder.
Markdown is rendered by Lerma.ClientDocs.Markdown (a small, XSS-safe,
dependency-free renderer: headings, lists, tables, code, links, images;
source HTML is always escaped). PDFs use the browser’s native viewer, CSV/text
render in a scrollable code panel, images render inline, and Word/PowerPoint/
spreadsheet binaries offer an open/download link.
The documentation widget has a fixed height (42rem); the tree and preview fill
the remaining space and scroll independently (overscroll-contain), so long
proposals never stretch the client page. On narrow screens the two panels split
the same fixed area vertically. Live re-lists the active source and
re-fetches the selected document every 20 s; Paused stops the polling. The preview’s Open action and inline assets use
/dashboard/clients/:id/docs/raw/:source?path=…, which is behind the dashboard
Access session. Active content (HTML/SVG) is forced to a download with a sandbox
CSP, paths are validated against traversal, and raw responses are no-store so
uploads appear immediately.
R2 Backup Credentials
The fouria User Worker signs R2 presigned URLs for Sandbox SDK backups using
R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY (plus CLOUDFLARE_ACCOUNT_ID and
BACKUP_BUCKET_NAME). Cloudflare derives these S3 credentials from a regular API
token: Access Key ID = token id, Secret Access Key = SHA-256 of the token
value. Lerma can mint one via API:
CLOUDFLARE_API_TOKEN=<parent> CLOUDFLARE_ACCOUNT_ID=<acct> \
mix lerma.r2_token --all-buckets --write ../../apps/fouria/.dev.vars
--all-buckets grants R2 object read/write across every bucket (covers the
dynamic per-tenant fouria-backup-* buckets). Without it, pass the exact bucket
names instead. The task also supports --name and --write <dotenv-path>.
The parent token must carry the Account → Account API Tokens → Edit permission (the account-scoped
LERMA_API_TOKENqualifies). The task creates an account-owned token — the right model for a platform R2 backup credential, since it survives individual user changes. Rotate R2 credentials on demand via this task.
The credential is mandatory
Since the backup credential is required by the Sandbox SDK on every tenant worker, lerma now treats it as mandatory: a provisioning run that cannot resolve one fails and rolls back (it no longer logs a warning and deploys a tenant that silently cannot back up).
The credential is selected by strategy (config :lerma, :r2_token_strategy,
env R2_TOKEN_STRATEGY, default :auto):
| Strategy | Behaviour |
|---|---|
:auto (default) | Uses the known account token limit (:r2_token_max_per_account) to choose: <= 250 → :shared, > 250 or unknown → :per_instance. |
:per_instance | Mints a dedicated R2 S3 token scoped to the tenant’s backup bucket at instance creation (Lerma.Provisioning.R2Credentials). Persisted on the instance record. |
:shared | Reuses the lerma-level R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY pair from the environment. |
- Cloudflare exposes no API for the account’s maximum token count
(
GET /accounts/{account_id}/tokenslists existing tokens only), so the limit defaults to unknown, which selects:per_instance. Pin a known limit withR2_TOKEN_MAX_PER_ACCOUNT(orconfig :lerma, :r2_token_max_per_account). - Generated per-instance credentials are stored on the instance
(
r2_access_key_id,r2_secret_access_key,r2_credential_source) so redeploys/upgrades reuse the same credential; a token generated for a failed brand-new provisioning is deleted during rollback. - Existing instances provisioned before this change have no stored credential; the next upgrade/redeploy backfills one.
- When the effective strategy is
:shared, bothR2_ACCESS_KEY_IDandR2_SECRET_ACCESS_KEYmust be set in the lerma environment — otherwise provisioning fails with:missing_shared_r2_credentials.
Worker deployments and Durable Object migrations
A tenant Worker’s Durable Object classes (Sandbox, ActivityScheduler,
TelegramIngress, plus BackupWatch while the change-watch flag is on) are
declared through a Cloudflare migration, which Cloudflare accepts only once
per class — error 10074 (Cannot apply new-sqlite-class migration to class '…' that is already depended on by existing Durable Objects) means the class already
has migration history.
The class set is not derived from the script’s bindings, because the two can diverge:
- an upgrade redeploy that binds fewer classes leaves the rest migrated but
unbound (the upgrade path now re-declares the full DO binding set, carrying the
existing
namespace_idper class); - a rollback/cleanup that deletes the script leaves its classes (and Durable
Objects) behind. Provisioning cleanup and a failed atomic deploy therefore tear
the classes down first (
delete_durable_object_namespaces/2), except on an upgrade rollback, where the Durable Objects hold the live tenant’s state.
As a backstop (Lerma.Cloudflare.DispatchNamespace), an upload rejected with
10074 names the offending class and is retried without it, so a tenant that
already regressed into this state recovers without manual Cloudflare
intervention. The retry logs DO classes already migrated for <ns>/<script>.
Testing
# Phoenix tests
mix test
# ExDoc generation
mix docs
# Full repository pre-commit (run from the repository root)
cd ../..
mix precommitProject Structure
lerma/
├── lib/
│ ├── lerma/ # Business logic (Ecto/PostgreSQL)
│ └── lerma_web/ # Web layer (LiveView, controllers, plugs)
├── assets/ # Phoenix frontend (JS/CSS)
├── config/ # Phoenix config (runtime.exs, prod.exs)
├── priv/
│ ├── repo/migrations/ # Ecto migrations
│ └── static/docs/ # ExDoc output served at /docs (generated)
├── test/ # Phoenix tests
├── iac/ # Infrastructure as Code
│ ├── tf/ # OpenTofu (Hetzner, DNS, Supabase, Access)
│ └── ansible/ # Ansible (server config, deployment)
├── bin/local-tunnel.sh # cloudflared tunnel for local CF Access auth
├── Dockerfile # Elixir release image (also builds /docs)
├── docker-compose.yml # Local dev (PostgreSQL only)
├── docker-compose.prod.yml # Production compose
├── docker-compose.pre.yml # Staging compose
└── mix.exs # Elixir project + ExDoc definition