Worker Health Probes

Kubernetes-style health probe API for the FourIA fouria. Defines the contract between worker instances and the lerma control plane.

Endpoint

GET /api/health

No authentication required. Returns a structured health snapshot with three probes plus the canonical gateway status.

Unified source of truth. GET /api/health is the single health endpoint shared by all consumers — internal (lerma HealthMonitor / HealthPoller) and external (loading page, admin dashboard, E2E). The legacy GET /api/status, GET /api/admin/gateway/status/polling and GET /api/admin/gateway/status/ws are thin aliases that delegate to the same snapshot.

Response Schema

{
  "ts": "2026-07-03T14:00:00.000Z",
  "startup": {
    "ok": true,
    "status": "success",
    "latencyMs": 12
  },
  "readiness": {
    "ok": true,
    "status": "success",
    "latencyMs": 8
  },
  "liveness": {
    "ok": true,
    "status": "success",
    "latencyMs": 5
  },
  "restoreStatus": {
    "hadBackup": true,
    "restored": true
  },
  "status": "running",
  "processId": "proc_123",
  "processType": "gateway"
}

ProbeResult

Field	Type	Values	Description
ok	boolean		true if the probe succeeded
status	string	"success", "failure", "unknown"	Outcome classification
latencyMs	number		Time the probe took in milliseconds
error	string?		Human-readable failure reason (present when ok: false)

RestoreStatus

Field	Type	Description
hadBackup	boolean	Whether a restorable R2 backup existed at startup (backups that are all expired/gone count as false — nothing to restore)
restored	boolean	Whether the most recent backup was restored successfully

Gateway Status

Field	Type	Description
status	running|starting|stopped	Canonical gateway status derived from the probes (see derivation below)
processId	string?	Id of the detected gateway binary / startup-script process, if any
processType	gateway|script?	Whether the detected process is the gateway binary or the startup script

Status derivation (deriveGatewayStatus in src/gateway/health-probes.ts):

Condition	status
readiness.ok	running
Readiness fails, a process exists	starting
Readiness fails, no process	stopped

---

Probe Definitions

Liveness Probe

What: Is the OpenClaw gateway alive?

How: Curls http://localhost:18789/health inside the sandbox container. Checks for HTTP 200 with {"ok": true}.

Frequency: Every /api/health call. This is the live heartbeat.

Failure meaning: The OpenClaw gateway process is dead or unresponsive. The worker container is still running but the application inside it has failed.

Readiness Probe

What: Is the worker ready to serve tenant traffic?

How: Checks three conditions:

1. Startup script exited — start-openclaw.sh is no longer in the sandbox process list. The script’s purpose is to bootstrap the gateway; it exits after confirming the gateway is reachable.
2. Gateway binary running — An openclaw gateway process exists with status running.
3. Gateway HTTP reachable — The gateway /health endpoint responds with HTTP 200.

Frequency: Every /api/health call.

Restart detection: If the startup script reappears in the process list (indicating a gateway restart), the readiness probe reports failure, and startupProbe re-validates its cached success against that fresh readiness result so the startup probe no longer serves a stale “ok” while the gateway is actually restarting.

Failure meaning: Traffic should NOT be routed to this worker. The gateway is either starting up, restarting, or has crashed.

Startup Probe

What: Has the worker completed its one-time initialization sequence?

How: Checks all readiness probe conditions PLUS explicit backup verification:

1. Liveness probe passes (gateway /health → ok: true)
2. Readiness probe passes (script exited + binary running + HTTP reachable)
3. If a backup existed (hadBackup: true), the backup must have been restored successfully (restored: true)

Caching: After the startup probe returns ok: true for the first time, the result is cached and subsequent calls return immediately with latencyMs: 0. Each call re-validates the cached success against the current snapshot’s readiness: if readiness reports failure (e.g. the startup script reappeared after the gateway went down), the stale success is dropped and a fresh result is computed, so a cached “ok” can never mask an outage. POST /api/admin/gateway/restart clears the cache explicitly.

Frequency: Evaluated on every /api/health call until first success; cached thereafter.

Failure meaning: The worker has not completed initial setup. Keep retrying.

---

Activating vs Passive

GET /api/health is activating by default: if the gateway is not running it triggers ensureGateway + restoreIfNeeded (and schedules an interval-gated backup). This is what keeps sleeping containers warm and recovers stopped instances when lerma polls.

Pass ?passive=1 to fail fast instead of warming:

• Never calls ensureGateway, restoreIfNeeded, or schedules a backup.
• Probes use a short timeout (PASSIVE_PROBE_TIMEOUT_MS, 3s) so a down container reports immediately instead of cold-starting for minutes.

The ROOT diagnostics dashboard uses passive probes so an operator can reach the recovery/debug surface even when the instance’s container is down.

GET /api/health           # activating (warm + recover)
GET /api/health?passive=1 # passive (fail fast, ROOT debug)

WebSocket Stream

GET /api/health/ws

Public WebSocket that pushes a fresh /api/health snapshot every ~5s (interval: HEALTH_WS_POLL_INTERVAL_MS). Supports the same ?passive=1 flag. The admin dashboard subscribes here and falls back to HTTP polling when the stream is unavailable.

---

State Transitions

State	startup.ok	readiness.ok	liveness.ok	Meaning
Cold start	false	false	false	Container just booted, nothing running yet
Starting	false	false	true	Gateway is running but startup script hasn’t exited yet
Starting (backup)	false	true	true	Readiness passes but backup not yet confirmed restored
Ready	true	true	true	Normal operation — route traffic
Degraded	true	false	true	Gateway restart in progress
Dead	true	false	false	Gateway crashed, no restart initiated
Stuck	false	false	true	Gateway running but startup script never exited (zombie script)

---

Consumer Contract (lerma)

Health Check Polling

The dashboard’s HealthPoller should poll GET https://{slug}.fouria.io/api/health and interpret results as:

startup.ok = true AND readiness.ok = true → Worker is healthy, ready for traffic
startup.ok = true AND readiness.ok = false → Degraded (restarting or backend issue), don't route new traffic
startup.ok = false → Still provisioning, keep polling

Polling Behavior

• During provisioning (instance.status = "provisioning"): Poll until startup.ok && readiness.ok, then transition to "running".
• During operation (instance.status = "running"): Poll periodically. If readiness.ok flips to false, the worker is restarting. Don’t immediately change status — wait for the restart to complete.
• On persistent failure: If liveness.ok stays false for an extended period (e.g., 5+ minutes), consider the worker dead and trigger an incident or re-deploy.

Health History

Recommendation: persist probe snapshots in an instance_health_snapshots table with:

Column	Type	Description
instance_id	UUID	FK to instances
startup_ok	boolean
readiness_ok	boolean
liveness_ok	boolean
had_backup	boolean
restored	boolean
latency_ms	integer	Combined probe time
inserted_at	timestamp

This enables health trending and alerting over time.

---

Implementation Notes

• The liveness probe is implemented in src/gateway/health-probes.ts → livenessProbe().
• The readiness probe is readinessProbe(). It uses findExistingGatewayProcess() and checkGatewayHealth() internally.
• The startup probe is startupProbe(). It calls liveness + readiness + checks getRestoreStatus().
• getRestoreStatus() reads from src/persistence.ts, which updates its state during restoreIfNeeded().
• getHealthSnapshot() fetches the gateway/startup-script process once and reuses it across all probes (readinessProbe + livenessProbe run concurrently, then startupProbe reuses their results). This keeps a snapshot to a single listProcesses call and two /health probes instead of one per probe — a meaningful reduction in Sandbox DO RPC volume.
• The probe cache uses module-level variables scoped to the Worker isolate. On container cold start (new isolate), the cache resets naturally.