F FourIA latest 2026.10.29 GitHub ↗

Lerma — Deployment Guide

On this page

Architecture

Lerma is a standalone Phoenix application running on Hetzner VMs via Docker Compose, deployed with Ansible. Infrastructure is provisioned with OpenTofu.

Two environments:

EnvironmentDomainServerImage tagTrigger
Productionlerma.paso4.ioHetzner CX23:latestRelease published
Staginglerma-dev.paso4.ioHetzner CX23:latest-prePush to develop

Each environment has its own isolated Supabase PostgreSQL database (free tier), SSH key, and Hetzner firewall.

Browser
    │
    ▼
Cloudflare DNS (proxied)
    │
    ├── lerma.paso4.io     ──►  Production Hetzner VM
    │                                           └── Docker Compose ──► Phoenix app ──► Supabase (prod)
    │
    └── lerma-dev.paso4.io ──►  Staging Hetzner VM
                                               └── Docker Compose ──► Phoenix app ──► Supabase (staging)

Maintainer SSH (DNS-only, not proxied):
    lerma-host.paso4.io         ──►  Production Hetzner VM :22
    lerma-staging-host.paso4.io ──►  Staging Hetzner VM :22

Deployment Flow

Push to develop               Release published
      │                              │
      ▼                              ▼
 Build image                  Pull :sha-<sha>
 Push as :latest-pre          Tag as :latest + release tag
   + :sha-<sha>               Push to GHCR
      │                              │
      ▼                              │
 Deploy to staging             Deploy to production
 (Ansible against              (Ansible against
  staging VM)                   production VM)

On every push to develop, the CI builds a fresh Docker image tagged :latest-pre (staging channel) plus an immutable :sha-<sha> (per-commit). On release publish, the CI promotes the :sha-<sha> image built for the released lerma commit to :latest + the release tag (no rebuild) and deploys it to production.

Manual deployments to production are available via workflow_dispatch.

Prerequisites

  • OpenTofu 1.6+ (for infrastructure provisioning)
  • Ansible (for server configuration and deployment)
  • Docker (for local development)
  • Elixir 1.20 with Erlang/OTP 29 (for local Phoenix development)
  • Access to the Paso4 Cloudflare account (35afea16440634aa2350331d2a736eec)
  • Hetzner Cloud API token
  • Supabase access token (generate here)

Environment Provisioning

1. Set up API credentials

export HCLOUD_TOKEN="your-hcloud-api-token"
export LERMA_API_TOKEN="your-lerma-api-token"

2. Provision infrastructure with OpenTofu

Each environment uses a separate OpenTofu workspace for state isolation and sources its own env file.

cd apps/lerma/iac/tf

# Create a terraform.tfvars (copy from .example) with your SSH keys
cp terraform.tfvars.example terraform.tfvars
# Edit: fill in ssh_public_key and ssh_public_key_staging

# Provision production (default workspace)
tofu init
(set +a && source ../../.env && set -a && tofu apply)

# Provision staging
tofu workspace new staging
(set +a && source ../../.env.staging && set -a && tofu apply)

3. Collect outputs for GitHub secrets

# Production
tofu workspace select default
tofu output -raw server_ipv4_address    # → SERVER_IP
tofu output -raw database_url           # → DATABASE_URL

# Staging
tofu workspace select staging
tofu output -raw server_ipv4_address    # → STAGING_SERVER_IP
tofu output -raw database_url           # → STAGING_DATABASE_URL

Add these to the GitHub repository secrets along with SECRET_KEY_BASE and STAGING_SECRET_KEY_BASE (generate with mix phx.gen.secret). Invitation emails are sent through Cloudflare Email Service using LERMA_API_TOKEN as the SMTP password, so the token must include the Email Sending: Edit permission and the sender domain must be onboarded for Email Sending.

DEV_INSTANCE_SSH_PUBLIC_KEY is optional. For manual Ansible deploys, export the single-line ssh-ed25519 public key in the Ansible controller environment before running the playbook. Do not provide the private key to Lerma.

4. Database migrations and the staging tenant

The dashboard deployment workflow applies pending migrations before deploying the new application image. Staging runs mix ecto.migrate with CF_ENVIRONMENT=staging, then idempotently bootstraps the reserved client tenant with Lerma.StagingTenant.ensure_client/0. Production runs the same migrations with CF_ENVIRONMENT=production; the migration’s safe default marker means it never creates or modifies a production client named client. Database triggers prevent deleting, renaming, or truncating away staging’s reserved tenant.

For a manual recovery or a newly provisioned environment, run migrations with the explicit platform environment:

# Production
MIX_ENV=prod CF_ENVIRONMENT=production DATABASE_URL="$DATABASE_URL" mix ecto.migrate -r Lerma.Repo

# Staging
MIX_ENV=prod CF_ENVIRONMENT=staging DATABASE_URL="$STAGING_DATABASE_URL" mix ecto.migrate -r Lerma.Repo
MIX_ENV=prod CF_ENVIRONMENT=staging DATABASE_URL="$STAGING_DATABASE_URL" \
  mix run -e 'case Lerma.StagingTenant.ensure_client() do
    {:ok, _client} -> :ok
    {:error, reason} -> Mix.raise("staging tenant bootstrap failed: #{inspect(reason)}")
  end'

Ansible (Server Configuration)

Ansible configures each Hetzner VM: installs Docker, sets up the firewall, writes configuration, and deploys the app container.

Inventory

FileEnvironmentIP source
iac/ansible/inventory.ymlProductionSERVER_IP env var
iac/ansible/inventory.staging.ymlStagingSTAGING_SERVER_IP env var

Roles

RolePurpose
commonUpdate apt, install base packages (curl, gnupg, ufw)
firewallOpen ports 22, 80, 443; enable UFW with deny-incoming default
dockerInstall Docker CE + Compose plugin, start service
app-configCopy docker compose file, write .env with secrets
appLogin to GHCR, pull image, restart container, health check

Running Ansible manually

cd apps/lerma/iac/ansible

# Production
SERVER_IP=116.203.112.210 ansible-playbook \
  -i inventory.yml \
  --private-key ~/.ssh/lerma_hetzner \
  --extra-vars "ghcr_owner=0xcab0" \
  --extra-vars "ghcr_username=0xCAB0" \
  --extra-vars "ghcr_token=$(gh auth token)" \
  --extra-vars "database_url=$DATABASE_URL" \
  --extra-vars "secret_key_base=$SECRET_KEY_BASE" \
  --extra-vars "cloudflare_api_token=$LERMA_API_TOKEN" \
  --extra-vars "cf_access_team_domain=$CF_ACCESS_TEAM_DOMAIN" \
  --extra-vars "cf_access_aud=$CF_ACCESS_AUD" \
  playbooks/site.yml

# Staging
STAGING_SERVER_IP=<staging-ip> ansible-playbook \
  -i inventory.staging.yml \
  --private-key ~/.ssh/lerma_staging \
  --extra-vars "compose_source_file=docker-compose.staging.yml" \
  --extra-vars "ghcr_owner=0xcab0" \
  --extra-vars "ghcr_username=0xCAB0" \
  --extra-vars "ghcr_token=$(gh auth token)" \
  --extra-vars "database_url=$STAGING_DATABASE_URL" \
  --extra-vars "secret_key_base=$STAGING_SECRET_KEY_BASE" \
  --extra-vars "cloudflare_api_token=$LERMA_API_TOKEN" \
  --extra-vars "cf_access_team_domain=$CF_ACCESS_TEAM_DOMAIN" \
  --extra-vars "cf_access_aud=$CF_ACCESS_AUD" \
  playbooks/site.yml

Docker Compose Files

FileEnvironmentImage tagContainer name
docker-compose.prod.ymlProduction:latestlerma
docker-compose.staging.ymlStaging:latest-prelerma-staging

Both expose port 80 → 4000, set CF_WORKER_MODE=false (standalone VM mode), and read secrets from .env.

API documentation (/docs)

The Dockerfile runs mix docs during the image build; ExDoc writes the HexDocs-format HTML into priv/static/docs, which the release serves at /docs (see the docs/0 config in mix.exs and the dedicated Plug.Static plug in LermaWeb.Endpoint). /docs remains behind Cloudflare Access like the rest of the dashboard. README/DEPLOY/STYLE are included as ExDoc extras, so anything documented in the README — including the Infrastructure as Code section — is also reachable from the hosted docs.

GitHub Actions

Two workflows share the Lerma pipeline (one workflow per deployable asset):

  • CI — .github/workflows/lerma.yml runs the quality gates on PRs and pushes to develop (mix format --check-formatted, mix credo --strict, mix license_audit, mix test).
  • CD — .github/workflows/deploy-dashboard.yml builds and deploys. Staging runs only after Lerma CI completes successfully on develop (workflow_run gate), so failing quality gates block deployment.
TriggerJobs
Lerma CI success on developbuild → stage migrations/bootstrap → deploy-staging
Release publishedpromote → production migrations → deploy-production
workflow_dispatchproduction migrations → deploy-manual (build + deploy)
Test build-and-push success on develop.github/workflows/promote-staging.yml promotes unstable-<sha> to the staging client tenant

Required GitHub Secrets

SecretEnvironmentPurpose
DATABASE_URLProductionSupabase PostgreSQL connection string
STAGING_DATABASE_URLStagingStaging Supabase PostgreSQL connection string
SECRET_KEY_BASEProductionPhoenix secret key base
STAGING_SECRET_KEY_BASEStagingPhoenix secret key base
SERVER_IPProductionProduction Hetzner VM IP
STAGING_SERVER_IPStagingStaging Hetzner VM IP
SSH_PRIVATE_KEYProductionSSH key for production VM
STAGING_SSH_PRIVATE_KEYStagingSSH key for staging VM
DEV_INSTANCE_SSH_PUBLIC_KEYBoth (optional)ssh-ed25519 public key for Wrangler SSH to dev Containers; never the private key
STAGING_PROVISIONING_API_KEYStagingFouria application’s staging API key for the build promotion workflow
LERMA_API_TOKENBothLerma API token
OTEL_INGEST_TOKENBothShared bearer secret configured on the account-level Cloudflare Observability destinations (Workers Logs/Traces → Lerma). Optional; the ingest endpoints fail closed (401) when unset.
WORKER_OBSERVABILITY_LOGS_DESTINATIONSBothComma-separated destination names included in tenant worker upload metadata (staging lerma-worker-logs, production lerma-worker-logs-prod). Empty → no export.
WORKER_OBSERVABILITY_TRACES_DESTINATIONSBothComma-separated destination names (staging lerma-worker-traces, production lerma-worker-traces-prod).
WORKER_OBSERVABILITY_LOGS_SAMPLING_RATEBothLogs head sampling rate (e.g. 0.1). Empty → 1.0.
WORKER_OBSERVABILITY_TRACES_SAMPLING_RATEBothTraces head sampling rate (e.g. 0.1). Empty → 1.0.
CF_ACCESS_TEAM_DOMAINBothCloudflare Access team domain
CF_ACCESS_AUDBothCloudflare Access audience tag (local env). CI maps the GitHub secrets CF_ACCESS_LERMA_AUD / STAGING_CF_ACCESS_LERMA_AUD into the CF_ACCESS_LERMA_AUD env var.
E2E_CF_ACCESS_TEAM_DOMAINBothAliased CF Access team domain (for ansible)

Generate STAGING_PROVISIONING_API_KEY from staging Lerma’s Applications → Fouria page and copy the one-time value to the GitHub secret. The key is application-scoped, stored in Lerma only as a digest, and used with the X-FourIA-Application: fouria header. Rotating it immediately invalidates the previous value; update the GitHub secret after generating a replacement.

The dashboard CD also runs mix lerma.observability.destinations --environment staging|production --warn-only before Ansible. It creates or PATCHes the account-level Cloudflare Observability destinations that export Workers Logs/Traces to Lerma (lerma-worker-logs[-prod], lerma-worker-traces[-prod]). The credential must carry Workers Observability Write: set CF_OBSERVABILITY_TOKEN, or add that permission to LERMA_API_TOKEN. With --warn-only, a missing permission logs a warning instead of failing the deploy; the destinations must then be created manually.

For dev Container SSH, generate a separate Ed25519 keypair on the operator workstation and store only the .pub line as DEV_INSTANCE_SSH_PUBLIC_KEY. The matching private key stays on the workstation. Lerma applies the public key to dev Container applications only; it is not deployed to production apps or passed into tenant Worker/container environment variables.

Local Development

# Start PostgreSQL
docker compose up -d db

# Setup and run
mix setup
mix phx.server

The Phoenix app runs on http://localhost:4000.

Testing

# Phoenix tests
mix test

# Generate the HexDocs-format API docs served at /docs
mix docs

# Full repository pre-commit check (run from the repository root)
cd ../..
mix precommit

DNS & Routing

Cloudflare DNS records are provisioned by OpenTofu:

DomainEnvironmentTypeProxiedPurpose
lerma.paso4.ioProductionA recordYesDashboard (Cloudflare Access)
lerma-dev.paso4.ioStagingA recordYesDashboard (Cloudflare Access)
lerma-host.paso4.ioProductionA + AAAA recordsNoMaintainer SSH/direct access
lerma-staging-host.paso4.ioStagingA + AAAA recordsNoMaintainer SSH/direct access

Maintainer access

The lerma-host / lerma-staging-host records are DNS-only (grey cloud) so they resolve for SSH. The dashboard records are proxied and cannot be used for SSH — Cloudflare’s proxy terminates HTTP(S) only.

# Production (SSH key: ~/.ssh/lerma_hetzner)
ssh root@lerma-host.paso4.io

# Staging (SSH key: ~/.ssh/lerma_staging)
ssh root@lerma-staging-host.paso4.io

OpenTofu exposes the hostname as host_fqdn:

tofu output -raw host_fqdn   # lerma-host.paso4.io (or lerma-staging-host.paso4.io)

Dev Container SSH access

Dev instances run as Cloudflare Containers, not Hetzner VMs. Cloudflare does not expose their port 22 publicly: connect through Wrangler after authenticating to the Paso4 Cloudflare account. The Container must be running; visiting its dev URL first will wake an idle instance. See the Cloudflare Containers SSH guide for details.

# Generate a dedicated keypair; share only the public key with Lerma.
ssh-keygen -t ed25519 -f ~/.ssh/fouria-dev -C fouria-dev
# Add the contents of ~/.ssh/fouria-dev.pub to GitHub's
# DEV_INSTANCE_SSH_PUBLIC_KEY secret, then deploy Lerma.
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/fouria-dev

# Find the dev Container application and its UUID.
npx wrangler containers list
# Production Lerma app name: tenant-<slug>-dev-sandbox
# Staging Lerma app name:    tenant-<slug>-dev-sandbox-staging
npx wrangler containers instances <APPLICATION_ID>
npx wrangler containers ssh <INSTANCE_ID>

# Or use OpenSSH with the dedicated private key explicitly:
ssh -i ~/.ssh/fouria-dev \
  -o ProxyCommand="npx wrangler containers ssh %h" \
  cloudchamber@<INSTANCE_ID>

The same secret is applied when Lerma creates or re-deploys a dev Container application. Existing dev applications pick up key rotations or removal on their next Lerma redeploy; changing authorized_keys itself does not require an image rollout. Clearing the secret removes the authorized key and disables Wrangler SSH for dev applications on that next redeploy.

The Phoenix endpoint’s check_origin is configured in config/runtime.exs:

check_origin: ["https://lerma.paso4.io", "https://lerma-dev.paso4.io"]

Project Structure

lerma/
├── lib/                         # Phoenix backend (Elixir)
│   ├── lerma/        # Business logic (Ecto/PostgreSQL)
│   └── lerma_web/    # Web layer (LiveView, controllers)
├── assets/                      # Phoenix frontend (JS/CSS)
├── config/                      # Phoenix config (runtime.exs, prod.exs)
├── priv/repo/migrations/        # Ecto migrations
├── test/                        # Phoenix tests
├── iac/                         # Infrastructure as Code
│   ├── tf/                      # OpenTofu (Hetzner, DNS, Supabase)
│   └── ansible/                 # Ansible (server config, deployment)
│       ├── inventory.yml        # Production inventory
│       ├── inventory.staging.yml # Staging inventory
│       ├── playbooks/site.yml   # Master playbook
│       ├── roles/               # common, firewall, docker, app-config, app
│       ├── templates/           # Jinja2 templates
│       └── group_vars/          # Shared variables
├── Dockerfile                   # Elixir release image
├── docker-compose.yml           # Local dev (PostgreSQL only)
├── docker-compose.prod.yml      # Production compose
├── docker-compose.staging.yml   # Staging compose
└── mix.exs                      # Elixir project definition

Troubleshooting

Database connection errors

Check the .env file at /opt/fouria/.env on the server:

ssh root@lerma-host.paso4.io cat /opt/fouria/.env          # production
ssh root@lerma-staging-host.paso4.io cat /opt/fouria/.env  # staging

Verify the Supabase project is active and the transaction pooler is enabled (port 6543).

Authentication 401 errors

Ensure CF_ACCESS_AUD matches the Cloudflare Access application audience tag. Set SKIP_AUTH=true in .env to bypass auth for debugging.

Container not healthy

Check container logs on the server:

ssh root@lerma-host.paso4.io docker compose -f /opt/fouria/docker-compose.yml logs app

Rebuilding the Docker image

The Docker image is rebuilt on every push to develop. To force a rebuild without code changes, update the cache bust comment in the Dockerfile:

# Build cache bust: 2026-09-22-v6-exdoc-docs

Adding a new environment

cd apps/lerma/iac/tf
tofu workspace new <env-name>
(tofu workspace select <env-name> && set +a && source ../../.env && set -a && tofu apply)