# Lerma Cloudflare-native platform for managing isolated application provisioning, cost analytics, and billing reporting. Lerma provisions and monitors tenant applications — each running in a fully isolated environment — through a Phoenix LiveView dashboard. ## Goal Lerma is a self-serve platform that allows administrators to: 1. **Provision** — deploy applications using Workers for Platforms dispatch namespaces, with each application instance running in an isolated environment 2. **Monitor** — track health, usage metrics, and cost analytics across all deployed application instances 3. **Bill** — manage credit-based billing, invoicing, and subscription plans for tenants Lerma serves on its own subdomain (`lerma.paso4.io`) on the `paso4.io` zone. ## Documentation map | Need | Where | | -------------------------------- | --------------------------------------------------------------------- | | Get started | [docs/GETTING-STARTED.md](/llms/platform/apps/lerma/getting-started/index.md) (hosted: `/docs/getting-started.html`) | | First steps (local setup) | [docs/FIRST-STEPS.md](/llms/platform/apps/lerma/first-steps/index.md) (hosted: `/docs/first-steps.html`) | | API documentation (HexDocs) | (see [API documentation](#api-documentation)) | | Deployment runbook | [DEPLOY.md](/llms/platform/apps/lerma/deploy/index.md) | | Product vision & pricing | [docs/VISION.md](/llms/platform/reference/vision/index.md) (symlink to the monorepo vision) | | UI/design tokens | [docs/STYLE.md](/llms/platform/apps/lerma/style/index.md) | | Infrastructure as Code | [Infrastructure as Code](#infrastructure-as-code) below | | Environment variables | [Configuration](#configuration) below + [`docs/ENV-VARIABLES.md`](/llms/platform/reference/env-variables/index.md) | | AI agent conventions | [AGENTS.md](https://github.com/0xCAB0/fouria/blob/develop/apps/lerma/AGENTS.md) | ## Architecture Lerma is a **standalone Phoenix application** (Elixir 1.20 / OTP 29) running on Hetzner VMs behind Cloudflare: - **Phoenix LiveView** — real-time dashboard UI, controllers, and JSON API under `/api`. - **Ecto / PostgreSQL** — the multi-tenant data layer. Production uses a dedicated Supabase PostgreSQL database per environment; local development runs PostgreSQL in Docker Compose. - **Cloudflare Access** — every browser request is gated by an Access JWT verified in `LermaWeb.Plugs.CfAccessAuth` (see [Local Auth Setup](#local-auth-setup-cloudflare-access)). - **Cloudflare API** — Lerma provisions per-tenant Workers, AI Gateways, Access applications, R2 buckets, and service tokens at onboarding time using `LERMA_API_TOKEN`. ``` Browser │ HTTPS (proxied) ▼ Cloudflare DNS (paso4.io zone) │ ├── lerma.paso4.io ──► Production Hetzner VM │ └── lerma-dev.paso4.io ──► Staging Hetzner VM │ │ │ ▼ │ Docker Compose ──► Phoenix release ──► Supabase (per-env) │ └── lerma-host.paso4.io (DNS-only) ──► Production VM :22 (maintainer SSH) lerma-staging-host.paso4.io (DNS-only) ──► Staging VM :22 ``` Tenant traffic is routed independently by `fouria-dispatch` on `*.fouria.io`; Lerma only hosts the dashboard on the `paso4.io` zone. The dashboard JSON API is served at `/api` on the dashboard domain — there is no separate API subdomain. ## Environments | Environment | Domain | Server | Image tag | Database | | ----------- | -------------------- | ------------ | ------------- | -------------------- | | Production | `lerma.paso4.io` | Hetzner CX23 | `:latest` | Supabase (prod) | | Staging | `lerma-dev.paso4.io` | Hetzner CX23 | `:latest-pre` | Supabase (staging) | | Local | `localhost:4000` | Docker | dev build | `postgres:16-alpine` | Each environment also publishes a **DNS-only** host record for maintainer SSH access (`lerma-host.paso4.io`, `lerma-staging-host.paso4.io`) — see [DEPLOY.md](/llms/platform/apps/lerma/deploy/index.md#maintainer-access). ## Quick Start ```bash # Phoenix dependencies + local database docker compose up -d db mix setup # Run the Phoenix app mix phx.server ``` Visit `http://localhost:4000` for the dashboard. ## API documentation The HexDocs-format API documentation is generated with [ExDoc](https://hexdocs.pm/ex_doc) from module docs and the README/DEPLOY/STYLE `extras`, and served by the Phoenix endpoint at: - Production: - Staging: `/docs` is served from `priv/static/docs` (see `docs/0` in [mix.exs](https://github.com/0xCAB0/fouria/blob/develop/apps/lerma/mix.exs)) and stays behind Cloudflare Access like the rest of the dashboard. Generate it locally with: ```bash mix docs # writes priv/static/docs, open priv/static/docs/index.html ``` `mix docs` is also run during the Docker image build, so the hosted docs always match the deployed release. Add new guides by listing them under `extras` in [mix.exs](https://github.com/0xCAB0/fouria/blob/develop/apps/lerma/mix.exs). ## Local Auth Setup (Cloudflare Access) The dashboard authenticates via Cloudflare Access JWT. In production, CF Access sits in front of the app. For local development, use a cloudflared tunnel to get real JWTs. ### One-Time Setup Create a Terraform workspace and apply the local dev configuration: ```bash cd apps/lerma/iac/tf # Ensure .env has TF_VAR_deploy_environment=local # Then create and select the "local" workspace tofu workspace new local tofu workspace select local # Source the .env (which has CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID, etc.) # and apply — this only creates CF Access resources, no Hetzner/Supabase (set +a && source ../../.env && set -a && tofu apply) ``` `CLOUDFLARE_API_TOKEN` remains Lerma's administrative provisioning credential. Set `CLOUDFLARE_AI_GATEWAY_API_KEY` separately to a real AI-scoped Cloudflare credential; the tenant provisioner copies that credential to User Workers and derives their `CLOUDFLARE_API_TOKEN` REST binding without exposing Lerma's token. Then copy the output values into your `.env`: ```bash tofu output -raw local_access_aud # → CF_ACCESS_AUD tofu output -raw local_service_token_id # → CF_ACCESS_SERVICE_TOKEN_ID tofu output -raw local_service_token_secret # → CF_ACCESS_SERVICE_TOKEN_SECRET tofu output -raw local_tunnel_token # → CLOUDFLARED_TUNNEL_TOKEN ``` ### Daily Development Start the tunnel in one terminal: ```bash CLOUDFLARED_TUNNEL_TOKEN= bin/local-tunnel.sh ``` Start the app in another: ```bash mix phx.server ``` Access via `https://local.lerma.paso4.io` — Cloudflare Access will prompt for OAuth login. ### Escape Hatch To run without the tunnel (auth bypassed): ```bash SKIP_AUTH=true mix phx.server ``` ## Infrastructure as Code All Lerma infrastructure lives under [`apps/lerma/iac/`](./iac/) and has two layers: ``` apps/lerma/iac/ ├── tf/ # OpenTofu — infrastructure provisioning │ ├── providers.tf # hcloud + cloudflare + supabase providers │ ├── main.tf # Hetzner server, firewall, SSH keys │ ├── supabase.tf # Supabase PostgreSQL project (prevent_destroy) │ ├── access.tf # Cloudflare Access app + Paso4/Partner root groups │ ├── variables.tf # Input variables (see terraform.tfvars.example) │ ├── outputs.tf # Hand-off values for Ansible + GitHub secrets │ ├── tests/*.tftest.hcl # `tofu test` suites (mocked providers) │ ├── justfile # Task aliases (plan/apply/output/destroy per env) │ └── .aliases # Shell-function fallback for the justfile └── ansible/ # Ansible — server configuration + deployment ├── inventory.yml # Production inventory (SERVER_IP) ├── inventory.staging.yml # Staging inventory (STAGING_SERVER_IP) ├── playbooks/site.yml # Master playbook ├── roles/ # common, firewall, docker, app-config, app ├── group_vars/all.yml # Shared playbook variables ├── templates/ # docker-compose.yml.j2 └── justfile # `just deploy-staging` / `just deploy-production` ``` ### OpenTofu (`iac/tf`) OpenTofu provisions everything *outside* the application: | File | Manages | | -------------- | ----------------------------------------------------------------------- | | `providers.tf` | `hcloud`, `cloudflare`, and `supabase` provider configuration | | `main.tf` | Hetzner server + firewall + uploaded SSH keys (env-suffixed per workspace) | | `supabase.tf` | One Supabase PostgreSQL project per environment (`prevent_destroy`) | | `access.tf` | Cloudflare Access application for the dashboard + the Paso4/Partner Root groups | | `variables.tf` | `hcloud_token`, `lerma_api_token`, `cloudflare_account_id/zone_id`, SSH keys, `deploy_environment`, server sizing, Supabase org/region, root email lists | | `outputs.tf` | `server_ipv4_address`, `database_url`, `host_fqdn`, `ansible_inventory`, `dashboard_access_aud`, root group IDs, local tunnel/token values | **Workspaces** isolate state per environment (they are *not* git-branched): | Workspace | Environment | Env file | Notes | | ------------ | ----------- | --------------------- | ------------------------------------------------ | | `default` | Production | `../../.env` | The default workspace **is** production | | `staging` | Staging | `../../.env.staging` | Set `TF_VAR_deploy_environment=staging` implicitly via workspace | | `local` | Local dev | `../../.env` | `TF_VAR_deploy_environment=local`; only CF Access + tunnel resources, no Hetzner/Supabase | Use the `justfile` (recommended) or source `.aliases`: ```bash cd apps/lerma/iac/tf just init # tofu init just test # tofu test (all mocked suites) just plan-production # tofu plan (default workspace, sources ../../.env) just apply-staging # tofu apply (staging workspace, sources ../../.env.staging) just output-production # inspect outputs just refresh-production # tofu apply -refresh-only (reconcile after state drift) just fmt-production # tofu fmt just help # list all targets ``` `tofu test` runs the suites in `tests/` with `mock_provider`, so it needs no real credentials — run it before every IaC change. ### Ansible (`iac/ansible`) Ansible configures each provisioned VM and deploys the app container. Roles run in order: | Role | Purpose | | ------------ | -------------------------------------------------------------- | | `common` | Update apt, install base packages (curl, gnupg, ufw) | | `firewall` | Open ports 22, 80, 443; enable UFW with deny-incoming default | | `docker` | Install Docker CE + Compose plugin, start the service | | `app-config` | Copy the compose file, write `.env` with secrets | | `app` | Log in to GHCR, pull the image, restart the container, health-check | Inventories are environment-specific and read the server IP from an env var. Compose files: `docker-compose.prod.yml` (`:latest`, container `lerma`), `docker-compose.staging.yml` / `docker-compose.pre.yml` (`:latest-pre`, container `lerma-staging`). ```bash cd apps/lerma/iac/ansible cp .env.example .env # fill in secrets just deploy-staging # deploy to lerma-dev.paso4.io just deploy-production # deploy to lerma.paso4.io ``` See [DEPLOY.md](/llms/platform/apps/lerma/deploy/index.md) for the complete OpenTofu + Ansible runbook, required GitHub secrets, and the full manual `ansible-playbook` invocations. ### Relationship between the two layers ``` OpenTofu (iac/tf) ──provisions──► Hetzner VM + firewall + Supabase + Access │ └── outputs (server IP, database_url, AUDs) ──► GitHub secrets │ Ansible (iac/ansible) ──consumes secrets + outputs──► configures VM, deploys container ``` ## Deployment ```bash # Staging (push to develop triggers CD after Lerma CI passes) git push origin develop # Production (publish a GitHub release; the pipeline promotes :latest-pre → :latest) gh release create ``` See [DEPLOY.md](/llms/platform/apps/lerma/deploy/index.md) for the full deployment guide, including migrations, container lifecycle, DNS, and monitoring. ## Configuration Lerma reads its runtime configuration from environment variables in [`config/runtime.exs`](https://github.com/0xCAB0/fouria/blob/develop/apps/lerma/config/runtime.exs) and its deployment credentials from `apps/lerma/.env` (IaC) and `apps/lerma/iac/ansible/.env` (server deploy). Never commit these files. | Variable | Required | Purpose | | ------------------ | -------- | ------------------------------------------------------------------- | | `DATABASE_URL` | yes | PostgreSQL (Supabase in prod; `postgres:16-alpine` locally) | | `SECRET_KEY_BASE` | yes | Phoenix cookie/session signing key (`mix phx.gen.secret`) | | `PHX_SERVER` | prod | Starts the Phoenix HTTP server | | `PHX_HOST` | prod | Hostname used for URL generation (`lerma.paso4.io`) | | `PHX_CHECK_ORIGIN` | prod | Allowed LiveView WebSocket origins | | `DNS_CLUSTER_QUERY`| optional | Cluster discovery query | | `SKIP_AUTH` | dev only | Bypasses Cloudflare Access JWT verification | IaC and deploy credentials (see [`iac/ansible/.env.example`](https://github.com/0xCAB0/fouria/blob/develop/apps/lerma/iac/ansible/.env.example) and the secrets manifest) include `CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_ZONE_ID`, `CLOUDFLARE_API_TOKEN` / `LERMA_API_TOKEN`, `CLOUDFLARE_AI_GATEWAY_API_KEY`, `CF_ACCESS_TEAM_DOMAIN`, `CF_ACCESS_AUD`, `CF_ACCESS_PLATFORM_AUD`, `GHCR_OWNER`/`GHCR_TOKEN`, `HCLOUD_TOKEN`, and `SUPABASE_*`. The complete cross-project env var reference is in [`docs/ENV-VARIABLES.md`](/llms/platform/reference/env-variables/index.md), and the machine-readable secrets manifest is [`docs/schemas/secrets-manifest.schema.json`](/assets/docs/schemas/secrets-manifest.schema.json). ### Health monitor service token The health monitor authenticates its tenant `/api/health` probes with a Cloudflare Access service token: | Variable | Source | | ---------------------------------- | ----------------------------------------------------- | | `HEALTH_MONITOR_ACCESS_CLIENT_ID` | `tofu output -raw local_service_token_id` *(local)* / GitHub secret *(prod)* | | `HEALTH_MONITOR_ACCESS_CLIENT_SECRET` | `tofu output -raw local_service_token_secret` | When both are empty (e.g. local dev) the probes are sent unauthenticated. ## Cloudflare API Token Permissions This project requires a Cloudflare API token (`LERMA_API_TOKEN`) with these minimum permissions: | Permission | Reason | | --------------------------------- | ------------------------------------------------------- | | `Workers Scripts:Edit` | Deploy and manage tenant worker scripts | | `DNS:Edit` | Manage DNS records for tenant domains | | `Access:Apps & Policies:Edit` | Create and manage Cloudflare Access applications | | `Access:Service Tokens:Edit` | Create service tokens for CI/CD auth | | `R2:Edit` | Create and manage per-tenant backup buckets | | `AI Gateway:Edit` | Configure AI Gateway for unified billing | | `Email Sending:Edit` | Send invitation emails via Cloudflare Email Service | | `Analytics:Read` | Query Analytics Engine for cost metrics | | `Account API Tokens:Edit` | Mint account-owned R2 S3 credentials (see below) | ## AI Gateway catalog and authentication **Cloudflare is the source of truth for which AI Gateways exist.** The dashboard (`/dashboard/ai-gateways`) and provisioning selection read the account's gateways directly from the Cloudflare API. Lerma's `ai_gateways` table is only an optional *policy override* store — `max_instances`, `enabled`, `environment`, and display metadata — merged on top of the Cloudflare catalog. A gateway that exists only in Cloudflare is still listed and selectable, and is always bound by its `gateway_id` (never the local primary key). Before a worker is deployed, provisioned, or upgraded, lerma ensures the bound gateway exists (create-only) so a redeploy never points at a deleted gateway. AI Gateways are created **authenticated by default** (`authentication: true`) so Cloudflare unified billing can attribute traffic to the gateway. An operator may opt into an unauthenticated **BYOK** gateway (tenants supply their own upstream key) only by explicitly setting `authentication: false` on the policy entry; the Cloudflare gateway is then created/updated to match that intent. To repair gateways that drifted to unauthenticated (e.g. ones created before authenticated gateways were enforced), run reconciliation from an IEx/`mix run` session: ```elixir Lerma.AiGateways.reconcile_all_authentication() ``` Reconciliation only **enables** authentication on policy entries recorded as `authentication: true`; explicit BYOK entries are left untouched. ## R2 Backup Credentials The fouria User Worker signs R2 presigned URLs for Sandbox SDK backups using `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` (plus `CLOUDFLARE_ACCOUNT_ID` and `BACKUP_BUCKET_NAME`). Cloudflare derives these S3 credentials from a regular API token: Access Key ID = token `id`, Secret Access Key = SHA-256 of the token `value`. Lerma can mint one via API: ```bash CLOUDFLARE_API_TOKEN= CLOUDFLARE_ACCOUNT_ID= \ mix lerma.r2_token --all-buckets --write ../../apps/fouria/.dev.vars ``` `--all-buckets` grants R2 object read/write across every bucket (covers the dynamic per-tenant `fouria-backup-*` buckets). Without it, pass the exact bucket names instead. The task also supports `--name` and `--write `. > The **parent** token must carry the **Account → Account API Tokens → Edit** > permission (the account-scoped `LERMA_API_TOKEN` qualifies). The task creates > an **account-owned** token — the right model for a platform R2 backup > credential, since it survives individual user changes. Rotate R2 credentials > on demand via this task. ### The credential is mandatory Since the backup credential is required by the Sandbox SDK on every tenant worker, lerma now treats it as **mandatory**: a provisioning run that cannot resolve one fails and rolls back (it no longer logs a warning and deploys a tenant that silently cannot back up). The credential is selected by strategy (`config :lerma, :r2_token_strategy`, env `R2_TOKEN_STRATEGY`, default `:auto`): | Strategy | Behaviour | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `:auto` (default) | Uses the known account token limit (`:r2_token_max_per_account`) to choose: `<= 250` → `:shared`, `> 250` or unknown → `:per_instance`. | | `:per_instance` | Mints a dedicated R2 S3 token scoped to the tenant's backup bucket at instance creation (`Lerma.Provisioning.R2Credentials`). Persisted on the instance record. | | `:shared` | Reuses the lerma-level `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` pair from the environment. | - Cloudflare exposes no API for the account's maximum token count (`GET /accounts/{account_id}/tokens` lists existing tokens only), so the limit defaults to **unknown**, which selects `:per_instance`. Pin a known limit with `R2_TOKEN_MAX_PER_ACCOUNT` (or `config :lerma, :r2_token_max_per_account`). - Generated per-instance credentials are stored on the instance (`r2_access_key_id`, `r2_secret_access_key`, `r2_credential_source`) so redeploys/upgrades reuse the same credential; a token generated for a failed brand-new provisioning is deleted during rollback. - Existing instances provisioned before this change have no stored credential; the next upgrade/redeploy backfills one. - When the effective strategy is `:shared`, both `R2_ACCESS_KEY_ID` and `R2_SECRET_ACCESS_KEY` must be set in the lerma environment — otherwise provisioning fails with `:missing_shared_r2_credentials`. ## Testing ```bash # Phoenix tests mix test # ExDoc generation mix docs # Full repository pre-commit (run from the repository root) cd ../.. mix precommit ``` ## Project Structure ``` lerma/ ├── lib/ │ ├── lerma/ # Business logic (Ecto/PostgreSQL) │ └── lerma_web/ # Web layer (LiveView, controllers, plugs) ├── assets/ # Phoenix frontend (JS/CSS) ├── config/ # Phoenix config (runtime.exs, prod.exs) ├── priv/ │ ├── repo/migrations/ # Ecto migrations │ └── static/docs/ # ExDoc output served at /docs (generated) ├── test/ # Phoenix tests ├── iac/ # Infrastructure as Code │ ├── tf/ # OpenTofu (Hetzner, DNS, Supabase, Access) │ └── ansible/ # Ansible (server config, deployment) ├── bin/local-tunnel.sh # cloudflared tunnel for local CF Access auth ├── Dockerfile # Elixir release image (also builds /docs) ├── docker-compose.yml # Local dev (PostgreSQL only) ├── docker-compose.prod.yml # Production compose ├── docker-compose.pre.yml # Staging compose └── mix.exs # Elixir project + ExDoc definition ```