# Worker Health Probes

Kubernetes-style health probe API for the FourIA fouria. Defines the contract between worker instances and the `lerma` control plane.

## Endpoint

```
GET /api/health
```

No authentication required. Returns a structured health snapshot with three probes plus the canonical gateway status.

> **Unified source of truth.** `GET /api/health` is the single health endpoint shared by all consumers —
> internal (`lerma` `HealthMonitor` / `HealthPoller`) and external (loading page, admin dashboard, E2E).
> The legacy `GET /api/status`, `GET /api/admin/gateway/status/polling` and `GET /api/admin/gateway/status/ws`
> are thin aliases that delegate to the same snapshot.

## Response Schema

```json
{
  "ts": "2026-07-03T14:00:00.000Z",
  "startup": {
    "ok": true,
    "status": "success",
    "latencyMs": 12
  },
  "readiness": {
    "ok": true,
    "status": "success",
    "latencyMs": 8
  },
  "liveness": {
    "ok": true,
    "status": "success",
    "latencyMs": 5
  },
  "restoreStatus": {
    "hadBackup": true,
    "restored": true
  },
  "status": "running",
  "processId": "proc_123",
  "processType": "gateway"
}
```

### ProbeResult

| Field       | Type    | Values                                | Description                                              |
| ----------- | ------- | ------------------------------------- | -------------------------------------------------------- |
| `ok`        | boolean |                                       | `true` if the probe succeeded                            |
| `status`    | string  | `"success"`, `"failure"`, `"unknown"` | Outcome classification                                   |
| `latencyMs` | number  |                                       | Time the probe took in milliseconds                      |
| `error`     | string? |                                       | Human-readable failure reason (present when `ok: false`) |

### RestoreStatus

| Field       | Type    | Description                                                                                                                   |
| ----------- | ------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `hadBackup` | boolean | Whether a _restorable_ R2 backup existed at startup (backups that are all expired/gone count as `false` — nothing to restore) |
| `restored`  | boolean | Whether the most recent backup was restored successfully                                                                      |

### Gateway Status

| Field         | Type                         | Description                                                              |
| ------------- | ---------------------------- | ------------------------------------------------------------------------ |
| `status`      | `running\|starting\|stopped` | Canonical gateway status derived from the probes (see derivation below)  |
| `processId`   | `string?`                    | Id of the detected gateway binary / startup-script process, if any       |
| `processType` | `gateway\|script?`           | Whether the detected process is the gateway binary or the startup script |

**Status derivation** (`deriveGatewayStatus` in `src/gateway/health-probes.ts`):

| Condition                         | `status`   |
| --------------------------------- | ---------- |
| `readiness.ok`                    | `running`  |
| Readiness fails, a process exists | `starting` |
| Readiness fails, no process       | `stopped`  |

---

## Probe Definitions

### Liveness Probe

**What**: Is the OpenClaw gateway alive?

**How**: Curls `http://localhost:18789/health` inside the sandbox container. Checks for HTTP 200 with `{"ok": true}`.

**Frequency**: Every `/api/health` call. This is the live heartbeat.

**Failure meaning**: The OpenClaw gateway process is dead or unresponsive. The worker container is still running but the application inside it has failed.

### Readiness Probe

**What**: Is the worker ready to serve tenant traffic?

**How**: Checks three conditions:

1. **Startup script exited** — `start-openclaw.sh` is no longer in the sandbox process list. The script's purpose is to bootstrap the gateway; it exits after confirming the gateway is reachable.
2. **Gateway binary running** — An `openclaw gateway` process exists with status `running`.
3. **Gateway HTTP reachable** — The gateway `/health` endpoint responds with HTTP 200.

**Frequency**: Every `/api/health` call.

**Restart detection**: If the startup script reappears in the process list (indicating a gateway restart), the readiness probe reports `failure`, and `startupProbe` re-validates its cached success against that fresh readiness result so the startup probe no longer serves a stale "ok" while the gateway is actually restarting.

**Failure meaning**: Traffic should NOT be routed to this worker. The gateway is either starting up, restarting, or has crashed.

### Startup Probe

**What**: Has the worker completed its one-time initialization sequence?

**How**: Checks all readiness probe conditions PLUS explicit backup verification:

1. Liveness probe passes (gateway `/health` → `ok: true`)
2. Readiness probe passes (script exited + binary running + HTTP reachable)
3. If a backup existed (`hadBackup: true`), the backup must have been restored successfully (`restored: true`)

**Caching**: After the startup probe returns `ok: true` for the first time, the result is cached and subsequent calls return immediately with `latencyMs: 0`. Each call re-validates the cached success against the current snapshot's readiness: if readiness reports failure (e.g. the startup script reappeared after the gateway went down), the stale success is dropped and a fresh result is computed, so a cached "ok" can never mask an outage. `POST /api/admin/gateway/restart` clears the cache explicitly.

**Frequency**: Evaluated on every `/api/health` call until first success; cached thereafter.

**Failure meaning**: The worker has not completed initial setup. Keep retrying.

---

## Activating vs Passive

`GET /api/health` is **activating** by default: if the gateway is not running it triggers `ensureGateway` + `restoreIfNeeded` (and schedules an interval-gated backup). This is what keeps sleeping containers warm and recovers stopped instances when `lerma` polls.

Pass `?passive=1` to **fail fast instead of warming**:

- Never calls `ensureGateway`, `restoreIfNeeded`, or schedules a backup.
- Probes use a short timeout (`PASSIVE_PROBE_TIMEOUT_MS`, 3s) so a down container reports immediately instead of cold-starting for minutes.

The ROOT diagnostics dashboard uses passive probes so an operator can reach the recovery/debug surface even when the instance's container is down.

```
GET /api/health           # activating (warm + recover)
GET /api/health?passive=1 # passive (fail fast, ROOT debug)
```

## WebSocket Stream

```
GET /api/health/ws
```

Public WebSocket that pushes a fresh `/api/health` snapshot every ~5s (interval: `HEALTH_WS_POLL_INTERVAL_MS`). Supports the same `?passive=1` flag. The admin dashboard subscribes here and falls back to HTTP polling when the stream is unavailable.

---

## State Transitions

| State             | startup.ok | readiness.ok | liveness.ok | Meaning                                                         |
| ----------------- | ---------- | ------------ | ----------- | --------------------------------------------------------------- |
| Cold start        | `false`    | `false`      | `false`     | Container just booted, nothing running yet                      |
| Starting          | `false`    | `false`      | `true`      | Gateway is running but startup script hasn't exited yet         |
| Starting (backup) | `false`    | `true`       | `true`      | Readiness passes but backup not yet confirmed restored          |
| **Ready**         | `true`     | `true`       | `true`      | **Normal operation — route traffic**                            |
| Degraded          | `true`     | `false`      | `true`      | Gateway restart in progress                                     |
| Dead              | `true`     | `false`      | `false`     | Gateway crashed, no restart initiated                           |
| Stuck             | `false`    | `false`      | `true`      | Gateway running but startup script never exited (zombie script) |

---

## Consumer Contract (`lerma`)

### Health Check Polling

The dashboard's `HealthPoller` should poll `GET https://{slug}.fouria.io/api/health` and interpret results as:

```
startup.ok = true AND readiness.ok = true → Worker is healthy, ready for traffic
startup.ok = true AND readiness.ok = false → Degraded (restarting or backend issue), don't route new traffic
startup.ok = false → Still provisioning, keep polling
```

### Polling Behavior

- **During provisioning** (instance.status = `"provisioning"`): Poll until `startup.ok && readiness.ok`, then transition to `"running"`.
- **During operation** (instance.status = `"running"`): Poll periodically. If `readiness.ok` flips to `false`, the worker is restarting. Don't immediately change status — wait for the restart to complete.
- **On persistent failure**: If `liveness.ok` stays `false` for an extended period (e.g., 5+ minutes), consider the worker dead and trigger an incident or re-deploy.

### Health History

Recommendation: persist probe snapshots in an `instance_health_snapshots` table with:

| Column         | Type      | Description         |
| -------------- | --------- | ------------------- |
| `instance_id`  | UUID      | FK to instances     |
| `startup_ok`   | boolean   |                     |
| `readiness_ok` | boolean   |                     |
| `liveness_ok`  | boolean   |                     |
| `had_backup`   | boolean   |                     |
| `restored`     | boolean   |                     |
| `latency_ms`   | integer   | Combined probe time |
| `inserted_at`  | timestamp |                     |

This enables health trending and alerting over time.

---

## Implementation Notes

- The liveness probe is implemented in `src/gateway/health-probes.ts` → `livenessProbe()`.
- The readiness probe is `readinessProbe()`. It uses `findExistingGatewayProcess()` and `checkGatewayHealth()` internally.
- The startup probe is `startupProbe()`. It calls liveness + readiness + checks `getRestoreStatus()`.
- `getRestoreStatus()` reads from `src/persistence.ts`, which updates its state during `restoreIfNeeded()`.
- `getHealthSnapshot()` fetches the gateway/startup-script process **once** and reuses it across all probes (`readinessProbe` + `livenessProbe` run concurrently, then `startupProbe` reuses their results). This keeps a snapshot to a single `listProcesses` call and two `/health` probes instead of one per probe — a meaningful reduction in Sandbox DO RPC volume.
- The probe cache uses module-level variables scoped to the Worker isolate. On container cold start (new isolate), the cache resets naturally.