Usage is counted after the response, from what the broker actually did, not from what the request asked for. That single decision is what makes the numbers defensible: a duplicate that stored nothing is not billed, a transaction that rolled back is not billed, and a batch of 1000 items where 400 were rejected is billed for 600.
The unit
One accumulator per (cluster, op class, minute), held in a 16-way sharded map keyed by
cluster id, carrying four counters:
| Field | Meaning |
|---|---|
reqs |
Requests, one per proxied call |
msgs |
Billable messages, per the rules below |
bytes_in |
Buffered request bytes. Non-zero only for produce and /configure, which are the requests the proxy buffers |
bytes_out |
Buffered response bytes, or the upstream Content-Length when the response streams through |
Recording is deliberately silent: no log line on the per-request path. Rates and sizes belong in aggregated blocks, not per-message lines.
Op classes
usage_minutes.op_class is one of five values, constrained by the schema:
| Class | Which requests |
|---|---|
push |
POST /api/v1/push |
txn |
POST /api/v1/transaction |
delivery |
Pop routes (/api/v1/pop/queue/…) |
configure |
The queue-admin class: /configure, deletes, seeks, subscription changes |
read |
Everything else that is metered: reads, gated surfaces, operator surfaces, and the consume routes that are not pops (ack, lease extension) |
Ack and lease-extension calls metering as read is worth knowing when you read the
table: delivery counts deliveries, not acknowledgements.
What counts as a billable message
A 201 response body is a top-level array of per-item results. Each item’s status
decides:
queued→ billed.buffered→ billed. The message was accepted; the cell is spooling to disk because it is in maintenance or its database is unreachable.duplicate→ not billed. Deduplication stored nothing.error,failed→ not billed.
A response that will not parse bills msgs = 0 with a warning. Under-billing something
unrecognised beats billing something that did not happen.
The same parse feeds one more signal: when a clear majority of items came back
buffered, the proxy logs (at most one line per 30 seconds, cell-wide) that the cell
is spooling pushes to disk. A single buffered item can be one item’s bad luck; a
majority is the cell not writing to PostgreSQL right now.
The push operations are counted on the way in, when the body is parsed for the size
caps and the message bucket. What decides whether they are billed is the response body,
not the status line: the broker answers 200 on a rollback too, with
{transactionId, success:false, error, results:[]}.
success: true→ bill the counted push operations, minus any that came backduplicate: true(first-wins deduplication inside the batch stored nothing new).success: false→ bill zero. The stored procedure raised and the whole transaction was undone, so nothing was stored whatever the request contained.- A shape the proxy cannot read → bill zero, with a warning.
A 200 response is buffered and the messages array counted. That count is both
metered as delivery messages and debited from the cluster’s message token bucket,
the post-completion debit described in
Quotas and rate limits.
A 204 (nothing available, or the queue is paused) is reqs = 1, msgs = 0, and the
body is not buffered at all.
- Upstream 5xx:
reqs = 1,msgs = 0, logged separately, body streamed straight through with no buffering. - Upstream timeout (the proxy’s own
504) and upstream unreachable (its own502): the request is recorded withmsgs = 0and a log line saying it was not billed. - Refused at the proxy (
401,403,413,429,421) is not metered at all. Those responses return before the metering step, sousage_minutesreflects work the cell was actually asked to do.
Responses that need parsing are buffered with a 64 MiB ceiling, matching the broker’s
own default body cap, so a legitimate pop batch of large messages still relays. A
response above it answers 502 rather than being silently truncated or mis-billed.
From memory to the database
Every QUEEN_PROXY_METER_FLUSH_MS (default 15000) the flush task drains every
accumulator whose minute is strictly before the current one, and UPSERTs the rows
into queen_proxy.usage_minutes inside one transaction:
ON CONFLICT (cluster_id, minute, op_class) DO UPDATE SET
reqs = usage_minutes.reqs + EXCLUDED.reqs,
msgs = usage_minutes.msgs + EXCLUDED.msgs, …The UPSERT adds, which is what makes the whole pipeline idempotent-by-addition and the spool below safe. The current minute is left in place, still accumulating, so concurrent requests for the same minute are not split across flushes for no reason.
At shutdown the drain takes everything, including the open minute. Otherwise every
restart would silently lose up to a minute of usage per cluster per op class. It is
bounded by QUEEN_PROXY_SHUTDOWN_DRAIN_MS and falls back to the spool, so a dead
control-plane database cannot turn a deploy into a hang.
The disk spool
When the flush fails (pxdb down, failing over, mid-migration), the rows are appended as
JSONL to QUEEN_PROXY_SPOOL_DIR:
| Property | Behaviour |
|---|---|
| File name | meter-<epoch_ms>-<seq>.buf, so lexical order is FIFO order |
| Rotation | Size-based at 5 MB |
| Durability | flush, not fsync per line: best-effort, because a lost tail of usage on an unclean crash is acceptable where a lost message would not be |
| Recovery | Once, at process start, strictly before this process can write anything of its own |
| Circuit breaker | After 3 consecutive recovery failures, sleep 30 s before the next file, so a startup recovery against a still-down database does not hammer it |
Recovery is at-least-once: a commit whose acknowledgement is lost is replayed on the next start and re-added by the UPSERT. That is the deliberate trade: usage that can be over-counted once is recoverable, usage that is dropped is gone.
If QUEEN_PROXY_SPOOL_DIR cannot be created, the proxy warns at construction and keeps
running. Usage is then lost whenever a flush fails, so put that directory on a volume
that survives a container restart.
Roll-up and pruning
The roll-up task ticks every QUEEN_PROXY_ROLLUP_MS (default hourly, first tick
immediately) and does two things in a fixed order:
-
queen_proxy.rollup_usage_days()folds closed days out ofusage_minutesintousage_days, aggregated by(cluster, day, op_class). The day is the UTC calendar day, pinned to UTC rather than the server’s timezone so a roll-up produces the same rows on any instance and month boundaries are stable. It recomputes each day rather than adding to it, so it is idempotent: extra runs cost an aggregate pass and change nothing. That is why hourly is fine. -
queen_proxy.prune_usage_minutes(keep_days)deletes minute rows whose day is already inusage_daysand older thanQUEEN_PROXY_USAGE_KEEP_DAYS(default 90).
The order matters: the prune is gated on the day existing in usage_days, so running it
first would simply skip the days this pass just folded in. A failed roll-up therefore
makes the prune a no-op rather than a data loss, and the tick returns early instead of
paying for the round trip.
Minutes are the evidence behind a billing dispute, which is why the retention window is
generous. usage_days is kept indefinitely either way.
The warn and block chain
The same tick evaluates the monthly allowance. One query returns, for every cluster with
an allowance, its calendar-month message count and the month itself, all from one
now(), so month and count cannot disagree across a boundary. The count comes from
queen_proxy.cluster_month_msgs, which reads usage_days plus the not-yet-rolled
usage_minutes remainder, so it includes traffic from the current hour.
| Level | Condition | Outbox event | Effect |
|---|---|---|---|
| Warn | msgs ≥ quota × 80% (QUEEN_PROXY_QUOTA_WARN_PERCENT) |
cluster_monthly_quota_warning |
None. The point is that a human can raise the plan before the tenant is stopped |
| Over | msgs ≥ quota |
cluster_monthly_quota_blocked |
Pushes blocked until the next calendar month |
Each level is announced once per cluster per month, and only upward. A repeat announces
nothing, and a count that wobbles back down announces nothing. Two things enforce that:
in-process state for the common case, and a lookup against the outbox for an existing
(kind, cluster_id, month) event, which is needed because a proxy restart mid-month
would otherwise re-announce every cluster already over the line. That lookup is
best-effort: if the query itself errors the event is emitted anyway, because a
duplicate control-plane event beats a swallowed one.
The block is released when the count falls back under, when the quota is lifted, when a new month starts the count near zero, or when the cluster stops being listed at all. A failed read releases nothing: having no evidence is not evidence that a block should be lifted.
queen_proxy.outbox is insert-only from the proxy’s side. Draining it (sending the
email, opening the ticket, calling the billing system) is a job for something else,
which is the whole point of the table.
Reading usage
The cluster console exposes it per cluster at GET /api/console/usage, human sessions
only. Directly:
SELECT day, op_class, msgs, reqs, bytes_in, bytes_out
FROM queen_proxy.usage_days
WHERE cluster_id = '…'
ORDER BY day DESC, op_class;SELECT queen_proxy.cluster_month_msgs('…cluster-uuid…', date_trunc('month', now() AT TIME ZONE 'UTC')::date);What metering is not
- Not a storage measure. Retained bytes come from the broker’s own per-queue stats
through the registry reconciler, on a separate path, and are not in
usage_minutes. See Quotas and rate limits. - Not a per-queue breakdown. The finest grain is
(cluster, minute, op class). - Not exactly-once. Spool recovery can over-count a batch once. Design any invoice
on top of
usage_dayswith that in mind. - Not a substitute for the broker’s own metrics. These numbers describe traffic through the gateway, not what the storage engine did with it.