Lerma — Deployment Guide
On this page
Architecture
Lerma is a standalone Phoenix application running on Hetzner VMs via Docker Compose, deployed with Ansible. Infrastructure is provisioned with OpenTofu.
Two environments:
| Environment | Domain | Server | Image tag | Trigger |
|---|---|---|---|---|
| Production | lerma.paso4.io | Hetzner CX23 | :latest | Release published |
| Staging | lerma-dev.paso4.io | Hetzner CX23 | :latest-pre | Push to develop |
Each environment has its own isolated Supabase PostgreSQL database (free tier), SSH key, and Hetzner firewall.
Browser
│
▼
Cloudflare DNS (proxied)
│
├── lerma.paso4.io ──► Production Hetzner VM
│ └── Docker Compose ──► Phoenix app ──► Supabase (prod)
│
└── lerma-dev.paso4.io ──► Staging Hetzner VM
└── Docker Compose ──► Phoenix app ──► Supabase (staging)
Maintainer SSH (DNS-only, not proxied):
lerma-host.paso4.io ──► Production Hetzner VM :22
lerma-staging-host.paso4.io ──► Staging Hetzner VM :22Deployment Flow
Push to develop Release published
│ │
▼ ▼
Build image Pull :sha-<sha>
Push as :latest-pre Tag as :latest + release tag
+ :sha-<sha> Push to GHCR
│ │
▼ │
Deploy to staging Deploy to production
(Ansible against (Ansible against
staging VM) production VM)
On every push to develop, the CI builds a fresh Docker image tagged :latest-pre (staging channel) plus an immutable :sha-<sha> (per-commit). On release publish, the CI promotes the :sha-<sha> image built for the released lerma commit to :latest + the release tag (no rebuild) and deploys it to production.
Manual deployments to production are available via workflow_dispatch.
Prerequisites
- OpenTofu 1.6+ (for infrastructure provisioning)
- Ansible (for server configuration and deployment)
- Docker (for local development)
- Elixir 1.20 with Erlang/OTP 29 (for local Phoenix development)
- Access to the Paso4 Cloudflare account (
35afea16440634aa2350331d2a736eec) - Hetzner Cloud API token
- Supabase access token (generate here)
Environment Provisioning
1. Set up API credentials
export HCLOUD_TOKEN="your-hcloud-api-token"
export LERMA_API_TOKEN="your-lerma-api-token"2. Provision infrastructure with OpenTofu
Each environment uses a separate OpenTofu workspace for state isolation and sources its own env file.
cd apps/lerma/iac/tf
# Create a terraform.tfvars (copy from .example) with your SSH keys
cp terraform.tfvars.example terraform.tfvars
# Edit: fill in ssh_public_key and ssh_public_key_staging
# Provision production (default workspace)
tofu init
(set +a && source ../../.env && set -a && tofu apply)
# Provision staging
tofu workspace new staging
(set +a && source ../../.env.staging && set -a && tofu apply)3. Collect outputs for GitHub secrets
# Production
tofu workspace select default
tofu output -raw server_ipv4_address # → SERVER_IP
tofu output -raw database_url # → DATABASE_URL
# Staging
tofu workspace select staging
tofu output -raw server_ipv4_address # → STAGING_SERVER_IP
tofu output -raw database_url # → STAGING_DATABASE_URL
Add these to the GitHub repository secrets along with SECRET_KEY_BASE and STAGING_SECRET_KEY_BASE (generate with mix phx.gen.secret). Invitation emails are sent through Cloudflare Email Service using LERMA_API_TOKEN as the SMTP password, so the token must include the Email Sending: Edit permission and the sender domain must be onboarded for Email Sending.
DEV_INSTANCE_SSH_PUBLIC_KEY is optional. For manual Ansible deploys, export
the single-line ssh-ed25519 public key in the Ansible controller environment
before running the playbook. Do not provide the private key to Lerma.
4. Database migrations and the staging tenant
The dashboard deployment workflow applies pending migrations before deploying
the new application image. Staging runs mix ecto.migrate with
CF_ENVIRONMENT=staging, then idempotently bootstraps the reserved client
tenant with Lerma.StagingTenant.ensure_client/0. Production runs the same
migrations with CF_ENVIRONMENT=production; the migration’s safe default
marker means it never creates or modifies a production client named client.
Database triggers prevent deleting, renaming, or truncating away staging’s
reserved tenant.
For a manual recovery or a newly provisioned environment, run migrations with the explicit platform environment:
# Production
MIX_ENV=prod CF_ENVIRONMENT=production DATABASE_URL="$DATABASE_URL" mix ecto.migrate -r Lerma.Repo
# Staging
MIX_ENV=prod CF_ENVIRONMENT=staging DATABASE_URL="$STAGING_DATABASE_URL" mix ecto.migrate -r Lerma.Repo
MIX_ENV=prod CF_ENVIRONMENT=staging DATABASE_URL="$STAGING_DATABASE_URL" \
mix run -e 'case Lerma.StagingTenant.ensure_client() do
{:ok, _client} -> :ok
{:error, reason} -> Mix.raise("staging tenant bootstrap failed: #{inspect(reason)}")
end'Ansible (Server Configuration)
Ansible configures each Hetzner VM: installs Docker, sets up the firewall, writes configuration, and deploys the app container.
Inventory
| File | Environment | IP source |
|---|---|---|
iac/ansible/inventory.yml | Production | SERVER_IP env var |
iac/ansible/inventory.staging.yml | Staging | STAGING_SERVER_IP env var |
Roles
| Role | Purpose |
|---|---|
common | Update apt, install base packages (curl, gnupg, ufw) |
firewall | Open ports 22, 80, 443; enable UFW with deny-incoming default |
docker | Install Docker CE + Compose plugin, start service |
app-config | Copy docker compose file, write .env with secrets |
app | Login to GHCR, pull image, restart container, health check |
Running Ansible manually
cd apps/lerma/iac/ansible
# Production
SERVER_IP=116.203.112.210 ansible-playbook \
-i inventory.yml \
--private-key ~/.ssh/lerma_hetzner \
--extra-vars "ghcr_owner=0xcab0" \
--extra-vars "ghcr_username=0xCAB0" \
--extra-vars "ghcr_token=$(gh auth token)" \
--extra-vars "database_url=$DATABASE_URL" \
--extra-vars "secret_key_base=$SECRET_KEY_BASE" \
--extra-vars "cloudflare_api_token=$LERMA_API_TOKEN" \
--extra-vars "cf_access_team_domain=$CF_ACCESS_TEAM_DOMAIN" \
--extra-vars "cf_access_aud=$CF_ACCESS_AUD" \
playbooks/site.yml
# Staging
STAGING_SERVER_IP=<staging-ip> ansible-playbook \
-i inventory.staging.yml \
--private-key ~/.ssh/lerma_staging \
--extra-vars "compose_source_file=docker-compose.staging.yml" \
--extra-vars "ghcr_owner=0xcab0" \
--extra-vars "ghcr_username=0xCAB0" \
--extra-vars "ghcr_token=$(gh auth token)" \
--extra-vars "database_url=$STAGING_DATABASE_URL" \
--extra-vars "secret_key_base=$STAGING_SECRET_KEY_BASE" \
--extra-vars "cloudflare_api_token=$LERMA_API_TOKEN" \
--extra-vars "cf_access_team_domain=$CF_ACCESS_TEAM_DOMAIN" \
--extra-vars "cf_access_aud=$CF_ACCESS_AUD" \
playbooks/site.ymlDocker Compose Files
| File | Environment | Image tag | Container name |
|---|---|---|---|
docker-compose.prod.yml | Production | :latest | lerma |
docker-compose.staging.yml | Staging | :latest-pre | lerma-staging |
Both expose port 80 → 4000, set CF_WORKER_MODE=false (standalone VM mode), and read secrets from .env.
API documentation (/docs)
The Dockerfile runs mix docs during the image build; ExDoc writes the
HexDocs-format HTML into priv/static/docs, which the release serves at
/docs (see the docs/0 config in mix.exs and the dedicated
Plug.Static plug in LermaWeb.Endpoint). /docs remains behind Cloudflare
Access like the rest of the dashboard. README/DEPLOY/STYLE are included as
ExDoc extras, so anything documented in the README — including the
Infrastructure as Code section — is also
reachable from the hosted docs.
GitHub Actions
Two workflows share the Lerma pipeline (one workflow per deployable asset):
- CI —
.github/workflows/lerma.ymlruns the quality gates on PRs and pushes todevelop(mix format --check-formatted,mix credo --strict,mix license_audit,mix test). - CD —
.github/workflows/deploy-dashboard.ymlbuilds and deploys. Staging runs only after Lerma CI completes successfully ondevelop(workflow_rungate), so failing quality gates block deployment.
| Trigger | Jobs |
|---|---|
Lerma CI success on develop | build → stage migrations/bootstrap → deploy-staging |
| Release published | promote → production migrations → deploy-production |
workflow_dispatch | production migrations → deploy-manual (build + deploy) |
Test build-and-push success on develop | .github/workflows/promote-staging.yml promotes unstable-<sha> to the staging client tenant |
Required GitHub Secrets
| Secret | Environment | Purpose |
|---|---|---|
DATABASE_URL | Production | Supabase PostgreSQL connection string |
STAGING_DATABASE_URL | Staging | Staging Supabase PostgreSQL connection string |
SECRET_KEY_BASE | Production | Phoenix secret key base |
STAGING_SECRET_KEY_BASE | Staging | Phoenix secret key base |
SERVER_IP | Production | Production Hetzner VM IP |
STAGING_SERVER_IP | Staging | Staging Hetzner VM IP |
SSH_PRIVATE_KEY | Production | SSH key for production VM |
STAGING_SSH_PRIVATE_KEY | Staging | SSH key for staging VM |
DEV_INSTANCE_SSH_PUBLIC_KEY | Both (optional) | ssh-ed25519 public key for Wrangler SSH to dev Containers; never the private key |
STAGING_PROVISIONING_API_KEY | Staging | Fouria application’s staging API key for the build promotion workflow |
LERMA_API_TOKEN | Both | Lerma API token |
OTEL_INGEST_TOKEN | Both | Shared bearer secret configured on the account-level Cloudflare Observability destinations (Workers Logs/Traces → Lerma). Optional; the ingest endpoints fail closed (401) when unset. |
WORKER_OBSERVABILITY_LOGS_DESTINATIONS | Both | Comma-separated destination names included in tenant worker upload metadata (staging lerma-worker-logs, production lerma-worker-logs-prod). Empty → no export. |
WORKER_OBSERVABILITY_TRACES_DESTINATIONS | Both | Comma-separated destination names (staging lerma-worker-traces, production lerma-worker-traces-prod). |
WORKER_OBSERVABILITY_LOGS_SAMPLING_RATE | Both | Logs head sampling rate (e.g. 0.1). Empty → 1.0. |
WORKER_OBSERVABILITY_TRACES_SAMPLING_RATE | Both | Traces head sampling rate (e.g. 0.1). Empty → 1.0. |
CF_ACCESS_TEAM_DOMAIN | Both | Cloudflare Access team domain |
CF_ACCESS_AUD | Both | Cloudflare Access audience tag (local env). CI maps the GitHub secrets CF_ACCESS_LERMA_AUD / STAGING_CF_ACCESS_LERMA_AUD into the CF_ACCESS_LERMA_AUD env var. |
E2E_CF_ACCESS_TEAM_DOMAIN | Both | Aliased CF Access team domain (for ansible) |
Generate STAGING_PROVISIONING_API_KEY from staging Lerma’s Applications →
Fouria page and copy the one-time value to the GitHub secret. The key is
application-scoped, stored in Lerma only as a digest, and used with the
X-FourIA-Application: fouria header. Rotating it immediately invalidates the
previous value; update the GitHub secret after generating a replacement.
The dashboard CD also runs mix lerma.observability.destinations --environment staging|production --warn-only before Ansible. It creates or PATCHes the
account-level Cloudflare Observability destinations that export Workers
Logs/Traces to Lerma (lerma-worker-logs[-prod],
lerma-worker-traces[-prod]). The credential must carry Workers
Observability Write: set CF_OBSERVABILITY_TOKEN, or add that permission to
LERMA_API_TOKEN. With --warn-only, a missing permission logs a warning
instead of failing the deploy; the destinations must then be created manually.
For dev Container SSH, generate a separate Ed25519 keypair on the operator
workstation and store only the .pub line as DEV_INSTANCE_SSH_PUBLIC_KEY.
The matching private key stays on the workstation. Lerma applies the public key
to dev Container applications only; it is not deployed to production apps or
passed into tenant Worker/container environment variables.
Local Development
# Start PostgreSQL
docker compose up -d db
# Setup and run
mix setup
mix phx.server
The Phoenix app runs on http://localhost:4000.
Testing
# Phoenix tests
mix test
# Generate the HexDocs-format API docs served at /docs
mix docs
# Full repository pre-commit check (run from the repository root)
cd ../..
mix precommitDNS & Routing
Cloudflare DNS records are provisioned by OpenTofu:
| Domain | Environment | Type | Proxied | Purpose |
|---|---|---|---|---|
lerma.paso4.io | Production | A record | Yes | Dashboard (Cloudflare Access) |
lerma-dev.paso4.io | Staging | A record | Yes | Dashboard (Cloudflare Access) |
lerma-host.paso4.io | Production | A + AAAA records | No | Maintainer SSH/direct access |
lerma-staging-host.paso4.io | Staging | A + AAAA records | No | Maintainer SSH/direct access |
Maintainer access
The lerma-host / lerma-staging-host records are DNS-only (grey cloud) so
they resolve for SSH. The dashboard records are proxied and cannot be used for
SSH — Cloudflare’s proxy terminates HTTP(S) only.
# Production (SSH key: ~/.ssh/lerma_hetzner)
ssh root@lerma-host.paso4.io
# Staging (SSH key: ~/.ssh/lerma_staging)
ssh root@lerma-staging-host.paso4.io
OpenTofu exposes the hostname as host_fqdn:
tofu output -raw host_fqdn # lerma-host.paso4.io (or lerma-staging-host.paso4.io)Dev Container SSH access
Dev instances run as Cloudflare Containers, not Hetzner VMs. Cloudflare does not expose their port 22 publicly: connect through Wrangler after authenticating to the Paso4 Cloudflare account. The Container must be running; visiting its dev URL first will wake an idle instance. See the Cloudflare Containers SSH guide for details.
# Generate a dedicated keypair; share only the public key with Lerma.
ssh-keygen -t ed25519 -f ~/.ssh/fouria-dev -C fouria-dev
# Add the contents of ~/.ssh/fouria-dev.pub to GitHub's
# DEV_INSTANCE_SSH_PUBLIC_KEY secret, then deploy Lerma.
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/fouria-dev
# Find the dev Container application and its UUID.
npx wrangler containers list
# Production Lerma app name: tenant-<slug>-dev-sandbox
# Staging Lerma app name: tenant-<slug>-dev-sandbox-staging
npx wrangler containers instances <APPLICATION_ID>
npx wrangler containers ssh <INSTANCE_ID>
# Or use OpenSSH with the dedicated private key explicitly:
ssh -i ~/.ssh/fouria-dev \
-o ProxyCommand="npx wrangler containers ssh %h" \
cloudchamber@<INSTANCE_ID>
The same secret is applied when Lerma creates or re-deploys a dev Container
application. Existing dev applications pick up key rotations or removal on
their next Lerma redeploy; changing authorized_keys itself does not require an
image rollout. Clearing the secret removes the authorized key and disables
Wrangler SSH for dev applications on that next redeploy.
The Phoenix endpoint’s check_origin is configured in config/runtime.exs:
check_origin: ["https://lerma.paso4.io", "https://lerma-dev.paso4.io"]Project Structure
lerma/
├── lib/ # Phoenix backend (Elixir)
│ ├── lerma/ # Business logic (Ecto/PostgreSQL)
│ └── lerma_web/ # Web layer (LiveView, controllers)
├── assets/ # Phoenix frontend (JS/CSS)
├── config/ # Phoenix config (runtime.exs, prod.exs)
├── priv/repo/migrations/ # Ecto migrations
├── test/ # Phoenix tests
├── iac/ # Infrastructure as Code
│ ├── tf/ # OpenTofu (Hetzner, DNS, Supabase)
│ └── ansible/ # Ansible (server config, deployment)
│ ├── inventory.yml # Production inventory
│ ├── inventory.staging.yml # Staging inventory
│ ├── playbooks/site.yml # Master playbook
│ ├── roles/ # common, firewall, docker, app-config, app
│ ├── templates/ # Jinja2 templates
│ └── group_vars/ # Shared variables
├── Dockerfile # Elixir release image
├── docker-compose.yml # Local dev (PostgreSQL only)
├── docker-compose.prod.yml # Production compose
├── docker-compose.staging.yml # Staging compose
└── mix.exs # Elixir project definitionTroubleshooting
Database connection errors
Check the .env file at /opt/fouria/.env on the server:
ssh root@lerma-host.paso4.io cat /opt/fouria/.env # production
ssh root@lerma-staging-host.paso4.io cat /opt/fouria/.env # staging
Verify the Supabase project is active and the transaction pooler is enabled (port 6543).
Authentication 401 errors
Ensure CF_ACCESS_AUD matches the Cloudflare Access application audience tag. Set SKIP_AUTH=true in .env to bypass auth for debugging.
Container not healthy
Check container logs on the server:
ssh root@lerma-host.paso4.io docker compose -f /opt/fouria/docker-compose.yml logs appRebuilding the Docker image
The Docker image is rebuilt on every push to develop. To force a rebuild without code changes, update the cache bust comment in the Dockerfile:
# Build cache bust: 2026-09-22-v6-exdoc-docsAdding a new environment
cd apps/lerma/iac/tf
tofu workspace new <env-name>
(tofu workspace select <env-name> && set +a && source ../../.env && set -a && tofu apply)