Lerma

Customer dashboard for provisioning, billing, cost analytics, and tenant management.

Cloudflare-native platform for managing isolated application provisioning, cost analytics, and billing reporting. Lerma provisions and monitors tenant applications — each running in a fully isolated environment — through a Phoenix LiveView dashboard.

Goal

Lerma is a self-serve platform that allows administrators to:

Provision — deploy applications using Workers for Platforms dispatch namespaces, with each application instance running in an isolated environment
Monitor — track health, usage metrics, and cost analytics across all deployed application instances
Bill — manage credit-based billing, invoicing, and subscription plans for tenants

Lerma serves on its own subdomain (lerma.paso4.io) on the paso4.io zone.

Documentation map

Need	Where
Get started	docs/GETTING-STARTED.md (hosted: `/docs/getting-started.html`)
First steps (local setup)	docs/FIRST-STEPS.md (hosted: `/docs/first-steps.html`)
API documentation (HexDocs)	https://lerma.paso4.io/docs (see API documentation)
Deployment runbook	DEPLOY.md
Product vision & pricing	docs/VISION.md (symlink to the monorepo vision)
UI/design tokens	docs/STYLE.md
Infrastructure as Code	Infrastructure as Code below
Environment variables	Configuration below + `docs/ENV-VARIABLES.md`
AI agent conventions	AGENTS.md

Architecture

Lerma is a standalone Phoenix application (Elixir 1.20 / OTP 29) running on Hetzner VMs behind Cloudflare:

Phoenix LiveView — real-time dashboard UI, controllers, and JSON API under /api.
Ecto / PostgreSQL — the multi-tenant data layer. Production uses a dedicated Supabase PostgreSQL database per environment; local development runs PostgreSQL in Docker Compose.
Cloudflare Access — every browser request is gated by an Access JWT verified in LermaWeb.Plugs.CfAccessAuth (see Local Auth Setup).
Cloudflare API — Lerma provisions per-tenant Workers, AI Gateways, Access applications, R2 buckets, and service tokens at onboarding time using LERMA_API_TOKEN.

Browser
   │  HTTPS (proxied)
   ▼
Cloudflare DNS (paso4.io zone)
   │  ├── lerma.paso4.io      ──► Production Hetzner VM
   │  └── lerma-dev.paso4.io  ──► Staging Hetzner VM
   │                                  │
   │                                  ▼
   │                            Docker Compose ──► Phoenix release ──► Supabase (per-env)
   │
   └── lerma-host.paso4.io (DNS-only) ──► Production VM :22   (maintainer SSH)
       lerma-staging-host.paso4.io (DNS-only) ──► Staging VM :22

Tenant traffic is routed independently by fouria-dispatch on *.fouria.io; Lerma only hosts the dashboard on the paso4.io zone. The dashboard JSON API is served at /api on the dashboard domain — there is no separate API subdomain.

Environments

Environment	Domain	Server	Image tag	Database
Production	`lerma.paso4.io`	Hetzner CX23	`:latest`	Supabase (prod)
Staging	`lerma-dev.paso4.io`	Hetzner CX23	`:latest-pre`	Supabase (staging)
Local	`localhost:4000`	Docker	dev build	`postgres:16-alpine`

Each environment also publishes a DNS-only host record for maintainer SSH access (lerma-host.paso4.io, lerma-staging-host.paso4.io) — see DEPLOY.md.

Quick Start

# Phoenix dependencies + local database
docker compose up -d db
mix setup

# Run the Phoenix app
mix phx.server

Visit http://localhost:4000 for the dashboard.

API documentation

The HexDocs-format API documentation is generated with ExDoc from module docs and the README/DEPLOY/STYLE extras, and served by the Phoenix endpoint at:

Production: https://lerma.paso4.io/docs
Staging: https://lerma-dev.paso4.io/docs

/docs is served from priv/static/docs (see docs/0 in mix.exs) and stays behind Cloudflare Access like the rest of the dashboard. Generate it locally with:

mix docs          # writes priv/static/docs, open priv/static/docs/index.html

mix docs is also run during the Docker image build, so the hosted docs always match the deployed release. Add new guides by listing them under extras in mix.exs.

Local Auth Setup (Cloudflare Access)

The dashboard authenticates via Cloudflare Access JWT. In production, CF Access sits in front of the app. For local development, use a cloudflared tunnel to get real JWTs.

One-Time Setup

Create a Terraform workspace and apply the local dev configuration:

cd apps/lerma/iac/tf

# Ensure .env has TF_VAR_deploy_environment=local
# Then create and select the "local" workspace
tofu workspace new local
tofu workspace select local

# Source the .env (which has CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID, etc.)
# and apply — this only creates CF Access resources, no Hetzner/Supabase
(set +a && source ../../.env && set -a && tofu apply)

CLOUDFLARE_API_TOKEN remains Lerma’s administrative provisioning credential. Set CLOUDFLARE_AI_GATEWAY_API_KEY separately to a real AI-scoped Cloudflare credential; the tenant provisioner copies that credential to User Workers and derives their CLOUDFLARE_API_TOKEN REST binding without exposing Lerma’s token.

Then copy the output values into your .env:

tofu output -raw local_access_aud            # → CF_ACCESS_AUD
tofu output -raw local_service_token_id      # → CF_ACCESS_SERVICE_TOKEN_ID
tofu output -raw local_service_token_secret  # → CF_ACCESS_SERVICE_TOKEN_SECRET
tofu output -raw local_tunnel_token          # → CLOUDFLARED_TUNNEL_TOKEN

Daily Development

Start the tunnel in one terminal:

CLOUDFLARED_TUNNEL_TOKEN=<token> bin/local-tunnel.sh

Start the app in another:

mix phx.server

Access via https://local.lerma.paso4.io — Cloudflare Access will prompt for OAuth login.

Escape Hatch

To run without the tunnel (auth bypassed):

SKIP_AUTH=true mix phx.server

Infrastructure as Code

All Lerma infrastructure lives under apps/lerma/iac/ and has two layers:

apps/lerma/iac/
├── tf/                       # OpenTofu — infrastructure provisioning
│   ├── providers.tf          # hcloud + cloudflare + supabase providers
│   ├── main.tf               # Hetzner server, firewall, SSH keys
│   ├── supabase.tf           # Supabase PostgreSQL project (prevent_destroy)
│   ├── access.tf             # Cloudflare Access app + Paso4/Partner root groups
│   ├── variables.tf          # Input variables (see terraform.tfvars.example)
│   ├── outputs.tf            # Hand-off values for Ansible + GitHub secrets
│   ├── tests/*.tftest.hcl    # `tofu test` suites (mocked providers)
│   ├── justfile              # Task aliases (plan/apply/output/destroy per env)
│   └── .aliases              # Shell-function fallback for the justfile
└── ansible/                  # Ansible — server configuration + deployment
    ├── inventory.yml         # Production inventory (SERVER_IP)
    ├── inventory.staging.yml # Staging inventory (STAGING_SERVER_IP)
    ├── playbooks/site.yml    # Master playbook
    ├── roles/                # common, firewall, docker, app-config, app
    ├── group_vars/all.yml    # Shared playbook variables
    ├── templates/            # docker-compose.yml.j2
    └── justfile              # `just deploy-staging` / `just deploy-production`

OpenTofu (`iac/tf`)

OpenTofu provisions everything outside the application:

File	Manages
`providers.tf`	`hcloud`, `cloudflare`, and `supabase` provider configuration
`main.tf`	Hetzner server + firewall + uploaded SSH keys (env-suffixed per workspace)
`supabase.tf`	One Supabase PostgreSQL project per environment (`prevent_destroy`)
`access.tf`	Cloudflare Access application for the dashboard + the Paso4/Partner Root groups
`variables.tf`	`hcloud_token`, `lerma_api_token`, `cloudflare_account_id/zone_id`, SSH keys, `deploy_environment`, server sizing, Supabase org/region, root email lists
`outputs.tf`	`server_ipv4_address`, `database_url`, `host_fqdn`, `ansible_inventory`, `dashboard_access_aud`, root group IDs, local tunnel/token values

Workspaces isolate state per environment (they are not git-branched):

Workspace	Environment	Env file	Notes
`default`	Production	`../../.env`	The default workspace is production
`staging`	Staging	`../../.env.staging`	Set `TF_VAR_deploy_environment=staging` implicitly via workspace
`local`	Local dev	`../../.env`	`TF_VAR_deploy_environment=local`; only CF Access + tunnel resources, no Hetzner/Supabase

Use the justfile (recommended) or source .aliases:

cd apps/lerma/iac/tf

just init                  # tofu init
just test                  # tofu test (all mocked suites)
just plan-production       # tofu plan   (default workspace, sources ../../.env)
just apply-staging         # tofu apply  (staging workspace, sources ../../.env.staging)
just output-production     # inspect outputs
just refresh-production    # tofu apply -refresh-only (reconcile after state drift)
just fmt-production        # tofu fmt
just help                  # list all targets

tofu test runs the suites in tests/ with mock_provider, so it needs no real credentials — run it before every IaC change.

Ansible (`iac/ansible`)

Ansible configures each provisioned VM and deploys the app container. Roles run in order:

Role	Purpose
`common`	Update apt, install base packages (curl, gnupg, ufw)
`firewall`	Open ports 22, 80, 443; enable UFW with deny-incoming default
`docker`	Install Docker CE + Compose plugin, start the service
`app-config`	Copy the compose file, write `.env` with secrets
`app`	Log in to GHCR, pull the image, restart the container, health-check

Inventories are environment-specific and read the server IP from an env var. Compose files: docker-compose.prod.yml (:latest, container lerma), docker-compose.staging.yml / docker-compose.pre.yml (:latest-pre, container lerma-staging).

cd apps/lerma/iac/ansible
cp .env.example .env         # fill in secrets
just deploy-staging          # deploy to lerma-dev.paso4.io
just deploy-production       # deploy to lerma.paso4.io

See DEPLOY.md for the complete OpenTofu + Ansible runbook, required GitHub secrets, and the full manual ansible-playbook invocations.

Relationship between the two layers

OpenTofu (iac/tf)  ──provisions──►  Hetzner VM + firewall + Supabase + Access
        │
        └── outputs (server IP, database_url, AUDs) ──► GitHub secrets
                                                              │
Ansible (iac/ansible) ──consumes secrets + outputs──► configures VM, deploys container

Deployment

# Staging (push to develop triggers CD after Lerma CI passes)
git push origin develop

# Production (publish a GitHub release; the pipeline promotes :latest-pre → :latest)
gh release create <tag>

See DEPLOY.md for the full deployment guide, including migrations, container lifecycle, DNS, and monitoring.

Configuration

Lerma reads its runtime configuration from environment variables in config/runtime.exs and its deployment credentials from apps/lerma/.env (IaC) and apps/lerma/iac/ansible/.env (server deploy). Never commit these files.

Variable	Required	Purpose
`DATABASE_URL`	yes	PostgreSQL (Supabase in prod; `postgres:16-alpine` locally)
`SECRET_KEY_BASE`	yes	Phoenix cookie/session signing key (`mix phx.gen.secret`)
`PHX_SERVER`	prod	Starts the Phoenix HTTP server
`PHX_HOST`	prod	Hostname used for URL generation (`lerma.paso4.io`)
`PHX_CHECK_ORIGIN`	prod	Allowed LiveView WebSocket origins
`DNS_CLUSTER_QUERY`	optional	Cluster discovery query
`SKIP_AUTH`	dev only	Bypasses Cloudflare Access JWT verification

IaC and deploy credentials (see iac/ansible/.env.example and the secrets manifest) include CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_ZONE_ID, CLOUDFLARE_API_TOKEN / LERMA_API_TOKEN, CLOUDFLARE_AI_GATEWAY_API_KEY, CF_ACCESS_TEAM_DOMAIN, CF_ACCESS_AUD, CF_ACCESS_PLATFORM_AUD, GHCR_OWNER/GHCR_TOKEN, HCLOUD_TOKEN, and SUPABASE_*.

The complete cross-project env var reference is in docs/ENV-VARIABLES.md, and the machine-readable secrets manifest is docs/schemas/secrets-manifest.schema.json.

Health monitor service token

The health monitor authenticates its tenant /api/health probes with a Cloudflare Access service token:

Variable	Source
`HEALTH_MONITOR_ACCESS_CLIENT_ID`	`tofu output -raw local_service_token_id` (local) / GitHub secret (prod)
`HEALTH_MONITOR_ACCESS_CLIENT_SECRET`	`tofu output -raw local_service_token_secret`

When both are empty (e.g. local dev) the probes are sent unauthenticated.

Cloudflare API Token Permissions

This project requires a Cloudflare API token (LERMA_API_TOKEN) with these minimum permissions:

Permission	Reason
`Workers Scripts:Edit`	Deploy and manage tenant worker scripts
`DNS:Edit`	Manage DNS records for tenant domains
`Access:Apps & Policies:Edit`	Create and manage Cloudflare Access applications
`Access:Service Tokens:Edit`	Create service tokens for CI/CD auth
`R2:Edit`	Create and manage per-tenant backup buckets
`AI Gateway:Edit`	Configure AI Gateway for unified billing
`Email Sending:Edit`	Send invitation emails via Cloudflare Email Service
`Analytics:Read`	Query Analytics Engine for cost metrics
`Account API Tokens:Edit`	Mint account-owned R2 S3 credentials (see below)

AI Gateway catalog and authentication

Cloudflare is the source of truth for which AI Gateways exist. The dashboard (/dashboard/ai-gateways) and provisioning selection read the account’s gateways directly from the Cloudflare API. Lerma’s ai_gateways table is only an optional policy override store — max_instances, enabled, environment, and display metadata — merged on top of the Cloudflare catalog. A gateway that exists only in Cloudflare is still listed and selectable, and is always bound by its gateway_id (never the local primary key).

Before a worker is deployed, provisioned, or upgraded, lerma ensures the bound gateway exists (create-only) so a redeploy never points at a deleted gateway.

AI Gateways are created authenticated by default (authentication: true) so Cloudflare unified billing can attribute traffic to the gateway. An operator may opt into an unauthenticated BYOK gateway (tenants supply their own upstream key) only by explicitly setting authentication: false on the policy entry; the Cloudflare gateway is then created/updated to match that intent.

To repair gateways that drifted to unauthenticated (e.g. ones created before authenticated gateways were enforced), run reconciliation from an IEx/mix run session:

Lerma.AiGateways.reconcile_all_authentication()

Reconciliation only enables authentication on policy entries recorded as authentication: true; explicit BYOK entries are left untouched.

R2 Backup Credentials

The fouria User Worker signs R2 presigned URLs for Sandbox SDK backups using R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY (plus CLOUDFLARE_ACCOUNT_ID and BACKUP_BUCKET_NAME). Cloudflare derives these S3 credentials from a regular API token: Access Key ID = token id, Secret Access Key = SHA-256 of the token value. Lerma can mint one via API:

CLOUDFLARE_API_TOKEN=<parent> CLOUDFLARE_ACCOUNT_ID=<acct> \
  mix lerma.r2_token --all-buckets --write ../../apps/fouria/.dev.vars

--all-buckets grants R2 object read/write across every bucket (covers the dynamic per-tenant fouria-backup-* buckets). Without it, pass the exact bucket names instead. The task also supports --name and --write <dotenv-path>.

The parent token must carry the Account → Account API Tokens → Edit permission (the account-scoped LERMA_API_TOKEN qualifies). The task creates an account-owned token — the right model for a platform R2 backup credential, since it survives individual user changes. Rotate R2 credentials on demand via this task.

The credential is mandatory

Since the backup credential is required by the Sandbox SDK on every tenant worker, lerma now treats it as mandatory: a provisioning run that cannot resolve one fails and rolls back (it no longer logs a warning and deploys a tenant that silently cannot back up).

The credential is selected by strategy (config :lerma, :r2_token_strategy, env R2_TOKEN_STRATEGY, default :auto):

Strategy	Behaviour
`:auto` (default)	Uses the known account token limit (`:r2_token_max_per_account`) to choose: `<= 250` → `:shared`, `> 250` or unknown → `:per_instance`.
`:per_instance`	Mints a dedicated R2 S3 token scoped to the tenant’s backup bucket at instance creation (`Lerma.Provisioning.R2Credentials`). Persisted on the instance record.
`:shared`	Reuses the lerma-level `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` pair from the environment.

Cloudflare exposes no API for the account’s maximum token count (GET /accounts/{account_id}/tokens lists existing tokens only), so the limit defaults to unknown, which selects :per_instance. Pin a known limit with R2_TOKEN_MAX_PER_ACCOUNT (or config :lerma, :r2_token_max_per_account).
Generated per-instance credentials are stored on the instance (r2_access_key_id, r2_secret_access_key, r2_credential_source) so redeploys/upgrades reuse the same credential; a token generated for a failed brand-new provisioning is deleted during rollback.
Existing instances provisioned before this change have no stored credential; the next upgrade/redeploy backfills one.
When the effective strategy is :shared, both R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY must be set in the lerma environment — otherwise provisioning fails with :missing_shared_r2_credentials.

Testing

# Phoenix tests
mix test

# ExDoc generation
mix docs

# Full repository pre-commit (run from the repository root)
cd ../..
mix precommit

Project Structure

lerma/
├── lib/
│   ├── lerma/                  # Business logic (Ecto/PostgreSQL)
│   └── lerma_web/              # Web layer (LiveView, controllers, plugs)
├── assets/                     # Phoenix frontend (JS/CSS)
├── config/                     # Phoenix config (runtime.exs, prod.exs)
├── priv/
│   ├── repo/migrations/        # Ecto migrations
│   └── static/docs/            # ExDoc output served at /docs (generated)
├── test/                       # Phoenix tests
├── iac/                        # Infrastructure as Code
│   ├── tf/                     # OpenTofu (Hetzner, DNS, Supabase, Access)
│   └── ansible/                # Ansible (server config, deployment)
├── bin/local-tunnel.sh         # cloudflared tunnel for local CF Access auth
├── Dockerfile                  # Elixir release image (also builds /docs)
├── docker-compose.yml          # Local dev (PostgreSQL only)
├── docker-compose.prod.yml     # Production compose
├── docker-compose.pre.yml      # Staging compose
└── mix.exs                     # Elixir project + ExDoc definition