---
title: "Usage metering"
description: "How usage is counted from response statuses, aggregated per minute, spooled to disk during an outage, rolled up daily, and turned into warnings and blocks."
---
> Documentation Index
> Fetch the complete documentation index at: https://queenmq.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Usage metering
Usage is counted **after** the response, from what the broker actually did, not from
what the request asked for. That single decision is what makes the numbers defensible:
a duplicate that stored nothing is not billed, a transaction that rolled back is not
billed, and a batch of 1000 items where 400 were rejected is billed for 600.
## The unit
One accumulator per `(cluster, op class, minute)`, held in a 16-way sharded map keyed by
cluster id, carrying four counters:
| Field | Meaning |
| --- | --- |
| `reqs` | Requests, one per proxied call |
| `msgs` | Billable messages, per the rules below |
| `bytes_in` | Buffered request bytes. Non-zero only for produce and `/configure`, which are the requests the proxy buffers |
| `bytes_out` | Buffered response bytes, or the upstream `Content-Length` when the response streams through |
Recording is deliberately silent: no log line on the per-request path. Rates and sizes
belong in aggregated blocks, not per-message lines.
## Op classes
`usage_minutes.op_class` is one of five values, constrained by the schema:
| Class | Which requests |
| --- | --- |
| `push` | `POST /api/v1/push` |
| `txn` | `POST /api/v1/transaction` |
| `delivery` | Pop routes (`/api/v1/pop/queue/…`) |
| `configure` | The queue-admin class: `/configure`, deletes, seeks, subscription changes |
| `read` | Everything else that is metered: reads, gated surfaces, operator surfaces, and the consume routes that are not pops (ack, lease extension) |
Ack and lease-extension calls metering as `read` is worth knowing when you read the
table: `delivery` counts deliveries, not acknowledgements.
## What counts as a billable message
Push: from the per-item statuses
A `201` response body is a top-level array of per-item results. Each item's `status`
decides:
- `queued` → billed.
- `buffered` → billed. The message was accepted; the cell is spooling to disk because it
is in maintenance or its database is unreachable.
- `duplicate` → **not** billed. Deduplication stored nothing.
- `error`, `failed` → **not** billed.
A response that will not parse bills `msgs = 0` with a warning. Under-billing something
unrecognised beats billing something that did not happen.
The same parse feeds one more signal: when a clear majority of items came back
`buffered`, the proxy logs (at most one line per 30 seconds, cell-wide) that the cell
is spooling pushes to disk. A single `buffered` item can be one item's bad luck; a
majority is the cell not writing to PostgreSQL right now.
Transaction: only when it committed
The push operations are counted on the way **in**, when the body is parsed for the size
caps and the message bucket. What decides whether they are billed is the response body,
not the status line: the broker answers `200` on a rollback too, with
`{transactionId, success:false, error, results:[]}`.
- `success: true` → bill the counted push operations, minus any that came back
`duplicate: true` (first-wins deduplication inside the batch stored nothing new).
- `success: false` → bill zero. The stored procedure raised and the whole transaction
was undone, so nothing was stored whatever the request contained.
- A shape the proxy cannot read → bill zero, with a warning.
Pop: delivered messages, and a bucket debit
A `200` response is buffered and the `messages` array counted. That count is both
metered as `delivery` messages **and** debited from the cluster's message token bucket,
the post-completion debit described in
[Quotas and rate limits](/selfhost/multi-tenant/quotas).
A `204` (nothing available, or the queue is paused) is `reqs = 1`, `msgs = 0`, and the
body is not buffered at all.
Errors: recorded, never billed
- **Upstream 5xx**: `reqs = 1`, `msgs = 0`, logged separately, body streamed straight
through with no buffering.
- **Upstream timeout** (the proxy's own `504`) and **upstream unreachable** (its own
`502`): the request is recorded with `msgs = 0` and a log line saying it was not
billed.
- **Refused at the proxy** (`401`, `403`, `413`, `429`, `421`) is not metered at all.
Those responses return before the metering step, so `usage_minutes` reflects work the
cell was actually asked to do.
Responses that need parsing are buffered with a 64 MiB ceiling, matching the broker's
own default body cap, so a legitimate pop batch of large messages still relays. A
response above it answers `502` rather than being silently truncated or mis-billed.
## From memory to the database
Every `QUEEN_PROXY_METER_FLUSH_MS` (default 15000) the flush task drains every
accumulator whose minute is **strictly before** the current one, and UPSERTs the rows
into `queen_proxy.usage_minutes` inside one transaction:
```sql
ON CONFLICT (cluster_id, minute, op_class) DO UPDATE SET
reqs = usage_minutes.reqs + EXCLUDED.reqs,
msgs = usage_minutes.msgs + EXCLUDED.msgs, …
```
The UPSERT **adds**, which is what makes the whole pipeline idempotent-by-addition and
the spool below safe. The current minute is left in place, still accumulating, so
concurrent requests for the same minute are not split across flushes for no reason.
At shutdown the drain takes **everything, including the open minute**. Otherwise every
restart would silently lose up to a minute of usage per cluster per op class. It is
bounded by `QUEEN_PROXY_SHUTDOWN_DRAIN_MS` and falls back to the spool, so a dead
control-plane database cannot turn a deploy into a hang.
## The disk spool
When the flush fails (pxdb down, failing over, mid-migration), the rows are appended as
JSONL to `QUEEN_PROXY_SPOOL_DIR`:
| Property | Behaviour |
| --- | --- |
| File name | `meter--.buf`, so lexical order is FIFO order |
| Rotation | Size-based at 5 MB |
| Durability | `flush`, not `fsync` per line: best-effort, because a lost tail of usage on an unclean crash is acceptable where a lost message would not be |
| Recovery | Once, at process start, strictly before this process can write anything of its own |
| Circuit breaker | After 3 consecutive recovery failures, sleep 30 s before the next file, so a startup recovery against a still-down database does not hammer it |
Recovery is at-least-once: a commit whose acknowledgement is lost is replayed on the
next start and re-added by the UPSERT. That is the deliberate trade: usage that can be
over-counted once is recoverable, usage that is dropped is gone.
If `QUEEN_PROXY_SPOOL_DIR` cannot be created, the proxy warns at construction and keeps
running. Usage is then lost whenever a flush fails, so put that directory on a volume
that survives a container restart.
## Roll-up and pruning
The roll-up task ticks every `QUEEN_PROXY_ROLLUP_MS` (default hourly, first tick
immediately) and does two things in a fixed order:
1. `queen_proxy.rollup_usage_days()` folds **closed** days out of `usage_minutes` into
`usage_days`, aggregated by `(cluster, day, op_class)`. The day is the UTC calendar
day, pinned to UTC rather than the server's timezone so a roll-up produces the same
rows on any instance and month boundaries are stable. It recomputes each day rather
than adding to it, so it is idempotent: extra runs cost an aggregate pass and change
nothing. That is why hourly is fine.
2. `queen_proxy.prune_usage_minutes(keep_days)` deletes minute rows whose day is
already in `usage_days` and older than `QUEEN_PROXY_USAGE_KEEP_DAYS` (default 90).
The order matters: the prune is gated on the day existing in `usage_days`, so running it
first would simply skip the days this pass just folded in. A failed roll-up therefore
makes the prune a no-op rather than a data loss, and the tick returns early instead of
paying for the round trip.
Minutes are the evidence behind a billing dispute, which is why the retention window is
generous. `usage_days` is kept indefinitely either way.
## The warn and block chain
The same tick evaluates the monthly allowance. One query returns, for every cluster with
an allowance, its calendar-month message count and the month itself, all from one
`now()`, so month and count cannot disagree across a boundary. The count comes from
`queen_proxy.cluster_month_msgs`, which reads `usage_days` **plus** the not-yet-rolled
`usage_minutes` remainder, so it includes traffic from the current hour.
| Level | Condition | Outbox event | Effect |
| --- | --- | --- | --- |
| Warn | `msgs ≥ quota × 80%` (`QUEEN_PROXY_QUOTA_WARN_PERCENT`) | `cluster_monthly_quota_warning` | None. The point is that a human can raise the plan before the tenant is stopped |
| Over | `msgs ≥ quota` | `cluster_monthly_quota_blocked` | Pushes blocked until the next calendar month |
Each level is announced once per cluster per month, and only upward. A repeat announces
nothing, and a count that wobbles back down announces nothing. Two things enforce that:
in-process state for the common case, and a lookup against the outbox for an existing
`(kind, cluster_id, month)` event, which is needed because a proxy restart mid-month
would otherwise re-announce every cluster already over the line. That lookup is
best-effort: if the query itself errors the event is emitted anyway, because a
duplicate control-plane event beats a swallowed one.
The block is released when the count falls back under, when the quota is lifted, when a
new month starts the count near zero, or when the cluster stops being listed at all.
A failed read releases nothing: having no evidence is not evidence that a block should
be lifted.
`queen_proxy.outbox` is insert-only from the proxy's side. Draining it (sending the
email, opening the ticket, calling the billing system) is a job for something else,
which is the whole point of the table.
## Reading usage
The cluster console exposes it per cluster at `GET /api/console/usage`, human sessions
only. Directly:
```sql
SELECT day, op_class, msgs, reqs, bytes_in, bytes_out
FROM queen_proxy.usage_days
WHERE cluster_id = '…'
ORDER BY day DESC, op_class;
```
```sql
SELECT queen_proxy.cluster_month_msgs('…cluster-uuid…', date_trunc('month', now() AT TIME ZONE 'UTC')::date);
```
## What metering is not
- **Not a storage measure.** Retained bytes come from the broker's own per-queue stats
through the registry reconciler, on a separate path, and are not in `usage_minutes`.
See [Quotas and rate limits](/selfhost/multi-tenant/quotas).
- **Not a per-queue breakdown.** The finest grain is `(cluster, minute, op class)`.
- **Not exactly-once.** Spool recovery can over-count a batch once. Design any invoice
on top of `usage_days` with that in mind.
- **Not a substitute for the broker's own metrics.** These numbers describe traffic
through the gateway, not what the storage engine did with it.
Source: https://queenmq.com/selfhost/multi-tenant/metering/index.mdx