# Cloudflare Memory Expense Report — Cutting Allocated-Memory Cost

**Date:** 2026-09-04 · **Account:** `35afea16440634aa2350331d2a736eec` (Paso4)
**Scope:** Last 31 days (2026-08-04 → 2026-09-04), sampled via the Cloudflare GraphQL
`workersInvocationsAdaptive` and `containersUsageAdaptiveGroups` datasets, cross-checked
against the Containers API and the deployment configs in this repo.

---

## 1. TL;DR

1. Plain Workers (dispatch + tenant dispatch workers) are **not** the memory problem —
   their duration/CPU sit comfortably inside the included tier. Their enormous wall-time
   (≈19.6 days of open connections) costs **$0** on the Standard usage model.
2. **The expense is Cloudflare Containers.** Every sandbox (OpenClaw gateway) is billed
   for every 10 ms it runs, on **provisioned** vCPU/memory/disk of its instance type.
3. Three cost levers, in order of impact:
   - **Instance-type overprovisioning** — several tenants are deployed at `standard-1`
     (4 GiB) / `standard-2` (6 GiB) while the platform default is `basic` (1 GiB) / `lite`
     (0.25 GiB). Right-sizing alone cuts memory billing ~44% (→ basic) or ~87% (→ lite).
   - **`max_instances` = 3 on every app** — including abandoned staging/test tenants.
     Provisioned capacity is 3× what most tenants need.
   - **Runtime (`SANDBOX_SLEEP_AFTER`)** — memory cost scales linearly with awake time;
     confirm every tenant worker actually has a sleep-after set (default is `never`).

---

## 2. Data: what the account actually spent

### 2.1 Plain Workers (`workersInvocationsAdaptive`, 31 days)

| Metric                   | Total (month) | Included (Standard) | Overage   |
| ------------------------ | ------------- | ------------------- | --------- |
| Requests                 | 28,940        | 10,000,000          | 0         |
| CPU time                 | 112,284 ms    | 30,000,000 ms       | 0         |
| Duration (GB·s)          | 29,877 GB·s   | 400,000 GB·s        | 0         |
| **Estimated extra cost** |               |                     | **$0.00** |

Breakdown by named worker (duration, i.e. GB·s):

| Worker                                        | Duration (GB·s) | Wall time          | Requests |
| --------------------------------------------- | --------------- | ------------------ | -------- |
| `fouria-dispatch`                             | 8,385           | 849,802 s (≈9.8 d) | 11,530   |
| `moltbot-sandbox-e2e-1787041788-…`            | 261             | 21,733 s           | 300      |
| `moltbot-sandbox-e2e-1786969669-…`            | 46              | 3,706 s            | 250      |
| `moltbot-sandbox-e2e-33785097313-1-base`      | 36              | 2,909 s            | 30       |
| `fouria-dispatch-staging`                     | 12              | 96,087 s           | 950      |
| `mila-gemini-proxy`                           | 0.5             | 1,724 s            | 1,120    |
| `__unknown__` (dispatch-tenant + sandbox e2e) | ~21,135         | ~18.7 d            | ~14,760  |

**Key observation:** 28,940 requests generate 112 CPU-seconds but **1.69 M seconds of wall
time** — the workers are alive almost entirely to hold idle WebSockets open. On the Standard
usage model duration is not billed, so this is free today, but it becomes expensive the moment
these are moved to unbound-style billing or Containers keep them warm. Fixing idle-connection
handling also shrinks container awake time (see §4.3).

### 2.2 Containers (`containersUsageAdaptiveGroups` / `containersMetricsAdaptiveGroups`, 31 days)

Billing model: **billed for every 10 ms actively running**, on provisioned resources.

| Component              | Total (month)                   | Free tier    | Est. cost    |
| ---------------------- | ------------------------------- | ------------ | ------------ |
| Memory                 | 4,242,369 GiB·s (1,178 GiB-h)   | 25 GiB-h     | **$10.38**   |
| CPU (allocated vCPU)   | 582,382 vCPU·s (9,706 vCPU-min) | 375 vCPU-min | **$11.20**   |
| Disk                   | 9,305,172 GB·s (2,585 GB-h)     | 200 GB-h     | **$0.60**    |
| **Total (this month)** |                                 |              | **≈ $22.20** |

> The GraphQL `containersUsageAdaptiveGroups` dataset is the same one the Cloudflare dashboard
> uses for billing estimates, so these are close to invoice figures (sampled; small months may
> drift). Memory and CPU dominate in roughly equal parts; disk is negligible.

#### Provisioned capacity is the real headline number

There are **14 container apps** (1 production tenant, several dev/test/staging tenants, 1 live
e2e app). All are configured with `max_instances: 3`. If every instance ran at capacity
simultaneously, the account is provisioned for:

| Resource | Total provisioned |
| -------- | ----------------- |
| Memory   | **68.3 GiB**      |
| vCPU     | 11.06             |
| Disk     | 186 GB            |

Worst-case (all 14 apps × max_instances × 24/7 × 30 d) would bill **≈ $1,050/mo**. The actual
bill is ~$22 because containers sleep most of the time — but that gap is the entire opportunity:
every hour a container stays awake costs money, and every GiB provisioned multiplies it.

#### Instance-type mix (from Containers API)

| App                                                                     | Instance type | Memory   | vCPU | 30 d @ 1 inst 24/7 |
| ----------------------------------------------------------------------- | ------------- | -------- | ---- | ------------------ |
| `tenant-tio-sandbox`                                                    | `standard-2`  | 6 GiB    | 1    | $92.90             |
| `tenant-tarjetero-sandbox`                                              | `standard-1`  | 4 GiB    | ½    | $53.29             |
| `tenant-tarjetero-dev-sandbox`                                          | `standard-1`  | 4 GiB    | ½    | $53.29             |
| `tenant-mcabo-sandbox`                                                  | `standard-1`  | 4 GiB    | ½    | $53.29             |
| `tenant-arua-sandbox`                                                   | `basic`       | 1 GiB    | ¼    | $20.17             |
| `tenant-test2-sandbox`                                                  | `basic`       | 1 GiB    | ¼    | $20.17             |
| `moltbot-sandbox-e2e-33865737117-1-base-sandbox`                        | `basic`       | 1 GiB    | ¼    | $20.17             |
| 7 × lite tenants (nocf-8/13/900, draft-11084/1220, sadsadsa, test2-dev) | `lite`        | 0.25 GiB | 1/16 | $5.22 each         |

Monthly multiplier vs `lite` (the platform default):

| Type         | $/mo (1 inst 24/7) | × lite |
| ------------ | ------------------ | ------ |
| `lite`       | $5.22              | 1×     |
| `basic`      | $20.17             | 3.9×   |
| `standard-1` | $53.29             | 10.2×  |
| `standard-2` | $92.90             | 17.8×  |
| `standard-3` | $158.42            | 30.4×  |
| `standard-4` | $288.75            | 55.3×  |

---

## 3. Root causes

1. **Instance type chosen at provisioning time overrides the platform default.**
   `apps/fouria/wrangler.jsonc` templates `basic`, and `lerma` defaults to `lite`
   (`tenant_provisioner.ex:1103`), but `tarjetero`, `tarjetero-dev` and `mcabo` were
   provisioned at `standard-1` (4 GiB) and `tio` at `standard-2` (6 GiB). Each 4–6 GiB
   tenant costs **10–18× a lite instance** for the same work. OpenClaw in a sandbox does not
   need 6 GiB; `basic` (1 GiB) or `lite` is enough for the gateway process.
2. **`max_instances: 3` everywhere** (`container_max_instances` default = 3, and every app in
   the account shows 3). Staging/test tenants (`nocf-*`, `draft-*`, `sadsadsa`, `test2*`) carry
   the same 3-instance provisioned capacity as production, tripling warm memory headroom for
   tenants with near-zero traffic.
3. **Idle WebSockets keep containers warm.** The 19.6 days of worker wall-time are open
   Control-UI / gateway connections. Each open connection prevents the container from sleeping,
   directly billing memory+CPU the whole time (`SANDBOX_SLEEP_AFTER` default is `never`).
4. **Stale e2e / test artifacts.** `moltbot-sandbox-e2e-*` workers/apps and the `-dev` sandbox
   apps persist after CI runs and keep allocated (if sleeping) capacity.

---

## 4. Recommendations (ordered by impact)

### 4.1 Right-size tenant instance types — **largest, immediate win**

| Tenant                                | Current              | Proposed        | Memory cut |
| ------------------------------------- | -------------------- | --------------- | ---------- |
| `tio`                                 | `standard-2` (6 GiB) | `basic` (1 GiB) | 83%        |
| `tarjetero`, `tarjetero-dev`, `mcabo` | `standard-1` (4 GiB) | `basic` (1 GiB) | 75%        |

Measured from this month's metrics, capping every instance at `basic` (1 GiB) cuts container
memory billing **≈ 44%**; capping at `lite` (0.25 GiB) cuts it **≈ 87%**.

Implementation:

- In lerma: `Instances.change_instance_type/2` (already wired to the Containers API) →
  set the four tenants above to `basic`.
- Guard rails: enforce `default_instance_type/1` (`lite`) and cap `standard-*` behind a
  plan gate so a `standard-2` never appears without explicit operator + plan justification.

### 4.2 Cut `max_instances` per tenant

- Production tenant: keep 2 (for a quick warm failover), not 3.
- Dev/test/staging tenants (`nocf-*`, `draft-*`, `sadsadsa`, `test2*`, `-dev` sandboxes):
  set `max_instances: 1`.
- Add a per-client `max_instances` field in lerma instead of a global default of 3
  (`container_max_instances`, `tenant_provisioner.ex:1147`).

### 4.3 Force aggressive sleep + close idle connections

- Ensure **every** tenant worker sets `SANDBOX_SLEEP_AFTER` (e.g. `10m`); audit the dispatch
  namespace bindings (the three production tenant scripts currently expose **zero** plain-text
  vars — confirming no sleep policy is deployed there).
- Add an idle WebSocket timeout in the worker proxy so long-lived gateway/Control-UI
  connections are closed after N minutes without traffic, letting the container sleep.
- Tighten `SANDBOX_SLEEP_AFTER` in `apps/fouria/wrangler.jsonc` from `15m` → `10m` and add a
  per-tenant override in the provisioning metadata.

### 4.4 Decommission stale apps

- Remove the abandoned tenants: `tenant-sadsadsa`, `tenant-draft-11084`, `tenant-draft-1220`,
  `tenant-nocf-8`, `tenant-nocf-13`, `tenant-nocf-900`, `tenant-test2-dev`, and
  `tenant-tarjetero-dev` if unused.
- Delete the old `moltbot-sandbox-e2e-1787041788-c89ea982` / `-1786969669-7d925be5` workers
  and apps; extend the `cleanup-stale-images.yml` workflow to also delete e2e container apps
  older than a day.

### 4.5 Monitoring

- Track `containersUsageAdaptiveGroups` memory/CPU per `applicationId` daily (GraphQL query in
  §5) so instance-type changes are verified against the invoice.
- The existing `container-active` Analytics Engine events already carry `instanceType` and
  `action` — wire the `estimateContainerCost/3` helper (in lerma `cost_calc.ex`) into a weekly
  drift report comparing provisioned vs. billed.

---

## 5. Appendix — reusable queries

```graphql
# Container memory + CPU per app (billing estimates)
query {
  viewer {
    accounts(filter: { accountTag: "35afea16440634aa2350331d2a736eec" }) {
      containersUsageAdaptiveGroups(
        limit: 50
        filter: { datetime_geq: "2026-08-04T00:00:00Z", datetime_lt: "2026-09-04T00:00:00Z" }
        orderBy: [sum_allocatedMemory_DESC]
      ) {
        sum {
          allocatedMemory
          allocatedDisk
          cpuTimeSec
        }
        dimensions {
          applicationId
          instanceId
        }
      }
    }
  }
}

# Worker duration (GB·s) — verify plain Workers stay inside the free tier
query {
  viewer {
    accounts(filter: { accountTag: "35afea16440634aa2350331d2a736eec" }) {
      workersInvocationsAdaptive(
        limit: 100
        filter: { datetime_geq: "2026-08-04T00:00:00Z", datetime_lt: "2026-09-04T00:00:00Z" }
        orderBy: [sum_duration_DESC]
      ) {
        sum {
          requests
          duration
          cpuTimeUs
          wallTime
        }
        dimensions {
          scriptName
          usageModel
        }
      }
    }
  }
}
```

**Pricing references** (Cloudflare docs, retrieved 2026-09-04):

- Workers Standard: 10 M requests/mo + $0.30/M, 30 M CPU ms/mo + $0.02/M, **no duration charge**.
- Containers (Workers Paid): 25 GiB-h/mo + **$0.0000025/GiB-s**; 375 vCPU-min/mo +
  $0.000020/vCPU-s; 200 GB-h/mo + $0.00000007/GB-s. Instance types: lite 256 MiB, basic 1 GiB,
  standard-1 4 GiB, standard-2 6 GiB, standard-3 8 GiB, standard-4 12 GiB.