Counters, rates and lag

Nothing in the log engine counts messages as they pass. There is no per-message row to count, and the counters you see are not sums: they are subtractions between watermarks, computed on a cadence and persisted into queen.stats. Understanding that is enough to predict every oddity on this page: why a number can be a few seconds stale, why it dips after a retention sweep, and why the manual refresh route does not help.

There are four independent sources of numbers, and they answer different questions.

The last row is not this page’s subject and is listed so nobody looks for it here. The KV and timer usage numbers are the sweeper’s third and slowest clock, not the reconciler’s work: they write queen.kv_usage, they are what the quota gauges and the 403 release decision read, and they are deliberately allowed to go stale under pressure, because they are precision rather than delivery. GET /api/v1/system/kv-timers reports their age as quotaAgeMs, and an age that keeps growing means the refresh is failing while the local write delta keeps enforcing.

The stats reconciler

stats.rs runs one cycle every STATS_INTERVAL_MS on one pooled connection, inside a transaction. The schedule is a durable claim row (queen.maintenance_leases, task stats_refresh): each replica polls the row and runs only when it wins the claim, so the configured interval is the true cluster cadence whatever the replica count, next_due_at survives restarts, a dead holder stalls the task for at most its lease, and enabled = false on the row pauses the loop cluster-wide. Inside the cycle, pg_try_advisory_xact_lock(737002) is kept as belt against writers that do not know the row: an old-image pod during a rollout, and the unlocked manual refresh route below. The lock id is deliberately distinct from retention’s 737001 so the two never serialize against each other. The transaction also sets SET LOCAL statement_timeout and idle_in_transaction_session_timeout, so an abandoned cycle can no longer pin the lock and an open snapshot until TCP keepalive gives up.

The cycle is one call: queen.log_refresh_all_stats_v1().

The arithmetic

Per partition, with worst = MIN(committed) across that partition’s consumer groups (no consumer rows means -1, so the whole retained range is pending):

total      = last_offset - log_start + 1
pending    = last_offset - GREATEST(worst, log_start - 1)
processing = SUM over live leases of (batch_end - committed)
completed  = total - pending - dlq          (floored at zero)

Every one of those is a subtraction between columns already on log_partitions and log_consumers. total is exact, not an estimate, because retention deletes only a contiguous prefix (there are never mid-log gaps), so last_offset - log_start + 1 is precisely the retained frame count. The refresh is therefore O(partitions), not O(segments). The retired engine’s refresher summed a message count over every live segment on every cycle.

processing is the leased-but-unacked span, summed only over leases that are still live, and it is capped at pending when written.

One timestamp cannot be derived from offsets, so it is the only log_segments touch the refresh makes, and it is a bounded primary-key probe rather than a scan:

• oldest_pending_at: queen.log_oldest_pending_at_v1(pid, wanted), which probes the segment covering the oldest pending offset and, if the cursor sits in a retention gap, the first segment past it. It runs only when something is actually pending. It is written as two probes rather than one filtered LIMIT 1 precisely so a miss cannot walk the partition’s dead prefix.

The stored newest_message_at column is dead: the refresh used to probe every partition’s tail for it on every cycle, but nothing reads the stored value (the newestMessage the wire reports is computed live per request), so the refresh now writes NULL and the probe is gone.

The retained-bytes lane

retained_bytes is SUM(octet_length(blob)) over the queue’s live segments. Because blob is STORAGE EXTERNAL, octet_length reads the length from the TOAST pointer without detoasting, but it is still a full heap scan of log_segments, and it feeds exactly one consequential reader: the proxy’s storage quota, which is hysteretic by contract. So it does not run inside the reconciler cycle. A separate loop in stats.rs runs queen.log_refresh_retained_bytes_v1() every RETAINED_BYTES_INTERVAL_MS (default 600000 ms per replica), gated by its own advisory lock (737003), and writes only the two columns it owns: retained_bytes and, counted inside the same scan, segment_count, the live log_segments rows behind each queue. The reconciler self-assigns both and never computes them. segment_count exists so the queue list stops counting segments per call: that count grew with the tenant’s segment population on every list request, while the queue detail keeps the exact live count per click.

The value measures the compressed bytes as stored (TOAST bookkeeping, indexes, WAL and the log_txns hash sidecar are all excluded). Both columns lag up to one slow-lane period, and a newly created queue reads 0 until the lane’s first pass over it.

Lag and rates

Lag is derived from the one timestamp above:

avg_lag_seconds = max_lag_seconds = EXTRACT(EPOCH FROM (now - oldest_pending_at))

At the queue level both come from the same value, so on this engine they agree.

Rates are per-snapshot deltas. queen.stats carries prev_total_messages, prev_completed_messages and prev_snapshot_at, and the upsert computes:

ingested_per_second  = (new total     - prev total)     / seconds since prev snapshot
processed_per_second = (new completed - prev completed) / seconds since prev snapshot

Both are floored at zero, and both require the window to be at least 3 seconds wide: replica timers are staggered rather than phase-locked, and a near-zero denominator would turn a handful of frames into a spike. A narrower window writes 0.

Rollups

The namespace, task and system rows are produced by the shared aggregators in 018_stats, which read the queue rows and never touch partition_id. The system aggregator counts queen.log_partitions for child_count, and the refresh then re-writes the same value from its own COUNT(queen.log_partitions). That overwrite is belt and braces rather than a correction: it is cheap, authoritative, and pins the value at this cycle’s snapshot.

The summary the procedure returns keeps the labels engine: "segments" and segPartitions. Those are hard-coded literals, the same compatibility echo as the storage: "segments" the queue wire still reports: there is one engine and no engine-selector column, so the label selects nothing.

The stats log target prints that summary verbatim with elapsed_ms. A cycle that updated no queues logs at debug so a quiet leader stops emitting info every ten seconds.

If the reconciler is not running

Every stats-backed reader (the system overview, the status endpoints, the queue detail) reads queen.stats. Without this loop those rows stay stale or empty and each of those surfaces returns zeros. That is not a hypothetical: it was a real regression, and this loop is the fix.

POST /api/v1/stats/refresh

The route forces the cycle above to run now instead of waiting out STATS_INTERVAL_MS. It calls the same queen.log_refresh_all_stats_v1() the loop calls, and returns its summary. It refreshes the counters only: retained_bytes belongs to the slow lane and keeps its last value here.

The metrics collector

syscollect.rs runs every METRICS_FLUSH_MS on one pooled connection and is deliberately not advisory-locked: every replica records its own rows, keyed by hostname, worker id and pid, and the readers aggregate across replicas.

Each cycle it:

• diffs the in-process operation counters since the last flush and inserts one queen.worker_metrics row of per-minute deltas (an AFTER INSERT trigger rolls it into queen.worker_metrics_summary, which is where the dashboard’s lifetime totals come from);
• samples host and process gauges (CPU from getrusage as a delta over the interval, RSS, deadpool’s pool status) and inserts one queen.system_metrics row whose metrics JSONB matches the shape the reader re-aggregates;
• flushes per-queue counters into queen.queue_lag_metrics.

Per-queue rates

queen.queue_lag_metrics is keyed (bucket_time, queue_id) on one-minute buckets: queue identity is the queen.queues id, the row carries a cascading foreign key to it (a deleted queue’s metric rows go with the queue), and readers join queen.queues when they need the name. The upsert sums on conflict, so several replicas writing the same minute aggregate into one cluster-wide row. Each bucket carries pop count, push request and message counts, empty-pop count, transaction count, ack request, success and failure counts, and a conflated count: log positions retired by conflation without a handler invocation, 0 on every queue whose groups do not conflate. The conflated count surfaces as queen_queue_conflated_per_minute beside the pop family, because the pair is the whole story of a conflating queue, what was handled and what was skipped.

Two columns merge differently because summing them would be wrong. Lag merges as a weighted average (SUM(avg × count) / SUM(count), the same identity the readers use across buckets) and the maximum merges with GREATEST. The parked-consumer count is a gauge and is summed across replicas.

Delivery lag itself is measured on the pop path: as each message is rendered the broker computes its age at delivery, which feeds both queen_queue_pop_lag_milliseconds and the lag columns here. So there are two different “lag” numbers in the system, and they are not the same measurement:

The first is a backlog age and exists even with no consumers running. The second only exists when messages are being delivered.

In-process counters and Prometheus

/metrics/prometheus mixes three kinds of series, and the prefix tells you which:

• queen_process_*: what this broker instance did since it started. Reset on restart.
• queen_cluster_*: lifetime totals read back out of PostgreSQL, so every instance reports the same value.
• Gauges are live in-process readings: the admission budget and per-lane inflight/waiting, pool active/idle/size, spool pending and failed, fusion items per batch, the pop fill-wait counters.

The full list is generated from the source; see the Prometheus reference.

Two collection details worth knowing because they affect what the numbers mean. autoAck deliveries are counted as acks: they are acknowledgements, so the ack throughput and completed totals reflect auto-acked consumption. And a push whose transaction failed is counted in the database-error series before the spool relabels those items buffered, so spooling never hides the failure.

Practical consequences

• Numbers are up to one interval stale. Counters lag by up to STATS_INTERVAL_MS; per-queue throughput lags by up to METRICS_FLUSH_MS and is bucketed by minute; retained_bytes lags by up to RETAINED_BYTES_INTERVAL_MS, and a queue created between lane passes reads 0.
• A rate dip right after a sweep is expected. See the retained-count note above.
• Zeros everywhere mean the reconciler is not running or, on an older build, something called the retired rows-engine refresh route.
• pending counts retained data only. Messages deleted by retention are not pending and not completed; they are gone, and total shrinks with them.
• completed is a residual, not an event count: total - pending - dlq, floored at zero. It is not a tally of acks.
• One queue-detail field is honestly zero. cursor.batchesConsumed reads 0 because queen.log_consumers does not carry the retired per-batch counter, and lastActivity is MAX(lease_acquired_at), the closest activity signal the log schema keeps.