Prometheus metrics

GET /metrics/prometheus returns exposition text with content type text/plain; version=0.0.4; charset=utf-8 and Cache-Control: no-cache. Every family carries # HELP and # TYPE.

GET /metrics is a different endpoint and is not Prometheus text: it returns a small JSON object with uptime, aggregate request and message counts, pool size and RSS. Scraping it will fail to parse.

The endpoint cannot be protected by broker authentication

/health, /metrics and /metrics/prometheus resolve to access level public in the route table, and the authentication middleware short-circuits on public before it ever looks for a token. They are also in the default JWT_SKIP_PATHS. Removing them from that list changes nothing, because the level check wins.

Process versus cluster

Two namespaces answer two different questions, and mixing them is the most common mistake in a Queen dashboard.

queen_process_* counts what this one broker instance did since it started. The counters live in memory, so they reset to zero on every restart, and each instance behind a load balancer reports its own numbers. Use them for rate queries and for attributing load to an instance.

queen_cluster_* are lifetime totals read back out of PostgreSQL, from queen.worker_metrics_summary, which the metrics collector maintains. They survive restarts, and every instance in the cluster reports the same value because they all read the same row. Summing them across instances multiplies your traffic by the number of brokers. Aggregate them with max, never sum.

The in-process families beyond queen_process_* (pool gauges, spool gauges, the maintenance flag, the admission families, the fusion batch families, parked long-polls) are also per-instance and also reset on restart.

How a scrape is assembled

The response is built in one pass and in a fixed order:

1. In-process gauges and counters, always emitted.
2. The admission families: queen_admission_budget, queen_admission_inflight{lane}, queen_admission_waiting{lane}, queen_admission_trains_per_s, queen_admission_txn_per_train and queen_admission_cycle_ms. These replaced queen_seg_push_vegas_limit and queen_seg_pop_vegas_limit in 1.0.0-beta.2, which no longer exist.
3. Pool gauges (queen_db_pool_size, _idle, _active), the push maintenance-mode flag, and the disk spool gauges.
4. One call to the database aggregator, whose JSON becomes the queen_cluster_* families, the per-queue minute rates, and the DLQ depths.

Step 4 is best-effort. If the pool has no connection or the read fails, the DB-backed families are silently absent from that scrape and the in-process block is still returned with HTTP 200. An alert on the absence of queen_cluster_* therefore detects a database problem; an alert on non-200 does not.

Because step 4 is a database round trip per scrape, the scrape interval is a load knob. Its per-queue and per-worker reads are bounded to buckets from the last five minutes; the lifetime totals are a single-row read and the dead-letter depths are counts.

Label sets

queen_batch_rtt_milliseconds is a gauge that happens to carry a quantile label. It is not a Prometheus summary, and quantile here is a plain label computed from a bounded in-process ring of recent round trips. Queue names are escaped for backslash, double quote and newline.

Freshness of the per-queue families

Per-queue series do not come from live counters. Each instance flushes its per-queue deltas into minute buckets in PostgreSQL every METRICS_FLUSH_MS (60 s by default), and the aggregator reads the most recent bucket per queue within a five-minute window. Consequences:

• a per-queue value can be up to one flush interval stale;
• queen_queue_metrics_age_seconds is the age of the bucket the value came from (use it to distinguish “zero traffic” from “stale bucket”);
• a queue that stops receiving traffic disappears from the exposition about five minutes later, rather than reporting zero.

queen_parked_long_polls is the exception: it is the live, instantaneous per-queue count of parked long-poll pops on this instance.

Per-queue series are not tenant-scoped

No per-queue series carries a tenant label, and exactly one series on the whole endpoint does: queen_timers_fire_lag_seconds, described under the cardinality rule below. When QUEEN_TENANCY_HEADER is on and two tenants hold the same queue name, their series collide:

• queen_dlq_depth_by_queue groups dead-letter rows by queue name, so the count is the sum across tenants;
• queen_parked_long_polls explicitly sums the per-tenant gauges into one line per queue name;
• the per-queue minute rates are stored per tenant but the aggregator selects one row per queue name, so one tenant’s bucket is reported and the others are hidden.

The rule about the tenant label

Every label on this endpoint is a fixed enumeration in Rust, so no call site can mint a new label value. That is not an accident of style, it is the rule:

The tenant label is allowed only on occupancy gauges, where it is one series per tenant written by a background job and bounded by the control plane. It is forbidden on per-operation counters, where it would be tenant times operation times outcome, that is, a cardinality chosen by whoever calls the API.

The key/value routes are the first surface in this product whose call rate is decided by somebody else’s web traffic, so they are the first place that rule could really have been broken. The per-tenant view of a hot path lives in the broker’s top-N log lines and in the JSON status endpoint, not here.

There is one motivated exception. queen_timers_fire_lag_seconds carries tenant alongside quantile, because it is an occupancy gauge and because a timer backlog that does not name its culprit puts every tenant on the same alert. It is capped at 1024 distinct tenants, and past the cap it denies the new tenant rather than evicting an old one: the tenant id is opaque and validated against nothing, so an evicting map is an unbounded map with extra steps. queen_timers_fire_lag_tenants_dropped_total says when the cap is biting, and a non-zero value means the gauge is no longer complete.

The two alarm gauges

Both are emitted only while the feature that owns them is on, and both report a condition that is invisible in every other series.

queen_kv_expired_not_pruned, with queen_kv_expiry_lag_seconds beside it. An expired key is never returned and never counts as existing, whether or not the sweeper has deleted its row: expiry is a predicate, not the physical absence of a row. So a sweeper that falls behind produces a failure mode that disguises itself as success. Reads stay perfectly correct while the table grows, table bloat follows, and nothing else on this endpoint moves. Alert on 50 000 rows or an expiry lag above 600 seconds, whichever comes first.

queen_timers_oldest_late_seconds, with queen_timers_due beside it. A timer backlog is promised work that has not been delivered, and a customer reads it as a lost message rather than as a late one. Warn on a minute of sustained lateness, page on five.

Both queen_kv_expired_not_pruned and queen_timers_due are capped at the SQL level, and each has a _capped companion gauge that is 1 when the cap was hit. When the companion is 1, the value is a floor and not a count: an exact count is proportional to the backlog precisely in the failure it exists to detect. Alert on the value, then read the companion before quoting the number.

A third signal is worth a panel even though it is not a fault: queen_kv_read_rejected_total{reason="rate_limited"} becoming non-zero for a tenant that was at zero. It is the earliest warning of the one new failure mode these features introduce, a customer who has just put key/value reads on their own end users’ request path. Panel it as a list of tenants rather than a total, because a total does not say who to call.

The reason label on that counter is read from the verdict rather than from the status code, which matters for one value: disabled is the operator’s runtime kill switch and nothing else. Classifying it from the status would file it under pool, and an operator who had just pulled the lever would be told the cell was out of database connections.

Two families that are gone

queen_queue_depth_total and queen_queue_depth_pending were formatted from a queue_depth key in the aggregator’s JSON that no stored procedure in this tree has ever produced. Neither family was ever present on a real scrape, only in this reference, because the generator reads help strings out of the exposition code and cannot tell live code from dead. The exposition code was removed in 1.1.0. Nothing an operator could see changes: a family with no samples is not in the exposition either way.

There is no per-queue depth family to replace them with, and that is on purpose: depth under a delivery policy is a per-group number, not a per-queue one, and a conflating group’s remaining work is not the queue’s message count. Ask GET /api/v1/resources/queues/:queue/depth for it, per group, and alert on queen_queue_pop_lag_milliseconds and the consumer-group lag endpoints for backlog.

queen_dlq_depth and queen_dlq_depth_by_queue do work: they count queen.log_dlq rows, cluster total and per queue name, with an index-only scan per scrape. (Until 2026-07-31 they read the retired rows engine’s dead-letter table and were pinned at 0; if a dashboard predates that, its DLQ panel was lying.) queen_cluster_dlq_total{scope="cluster"} remains the lifetime counter of dead-letter transitions; the depth gauges shrink when DLQ rows are deleted or replayed, the counter does not.

The families

GET /metrics/prometheus exposes 83 families. queen_process_* counts what this one broker instance did since it started; queen_cluster_* are lifetime totals read back out of PostgreSQL, so every instance reports the same value.

The queen_kv_*, queen_timers_* and queen_sweeper_* families are exposed by every broker, at zero on one that has never seen a key or a timer, because those surfaces are part of the engine rather than a feature a cell opts into. A dashboard panel or an alert built on them can therefore be written before the first tenant arrives, and a family that goes missing is a broker that is gone, not a feature that is off.

Process (this broker instance)

Cluster lifetime totals (from PostgreSQL)

Per-queue rates and depth

Engine internals

Key/value state, timers and the sweeper

Other

Related pages