Prometheus metrics

GET /metrics/prometheus returns exposition text with content type text/plain; version=0.0.4; charset=utf-8 and Cache-Control: no-cache. Every family carries # HELP and # TYPE.

GET /metrics is a different endpoint and is not Prometheus text: it returns a small JSON object with uptime, aggregate request and message counts, pool size and RSS. Scraping it will fail to parse.

The endpoint cannot be protected by broker authentication

/health, /metrics and /metrics/prometheus resolve to access level public in the route table, and the authentication middleware short-circuits on public before it ever looks for a token. They are also in the default JWT_SKIP_PATHS. Removing them from that list changes nothing, because the level check wins.

Process versus cluster

Two namespaces answer two different questions, and mixing them is the most common mistake in a Queen dashboard.

queen_process_* counts what this one broker instance did since it started. The counters live in memory, so they reset to zero on every restart, and each instance behind a load balancer reports its own numbers. Use them for rate queries and for attributing load to an instance.

queen_cluster_* are lifetime totals read back out of PostgreSQL, from queen.worker_metrics_summary, which the metrics collector maintains. They survive restarts, and every instance in the cluster reports the same value because they all read the same row. Summing them across instances multiplies your traffic by the number of brokers. Aggregate them with max, never sum.

The in-process families beyond queen_process_* (pool gauges, spool gauges, the maintenance flag, the Vegas limits, the fusion batch families, parked long-polls) are also per-instance and also reset on restart.

How a scrape is assembled

The response is built in one pass and in a fixed order:

1. In-process gauges and counters, always emitted.
2. queen_seg_push_vegas_limit and queen_seg_pop_vegas_limit, the adaptive concurrency limits.
3. Pool gauges (queen_db_pool_size, _idle, _active), the push maintenance-mode flag, and the disk spool gauges.
4. One call to the database aggregator, whose JSON becomes the queen_cluster_* families, the per-queue minute rates, and the DLQ depths.

Step 4 is best-effort. If the pool has no connection or the read fails, the DB-backed families are silently absent from that scrape and the in-process block is still returned with HTTP 200. An alert on the absence of queen_cluster_* therefore detects a database problem; an alert on non-200 does not.

Because step 4 is a database round trip per scrape, the scrape interval is a load knob. Its per-queue and per-worker reads are bounded to buckets from the last five minutes; the lifetime totals are a single-row read and the dead-letter depths are counts.

Label sets

queen_batch_rtt_milliseconds is a gauge that happens to carry a quantile label. It is not a Prometheus summary, and quantile here is a plain label computed from a bounded in-process ring of recent round trips. Queue names are escaped for backslash, double quote and newline.

Freshness of the per-queue families

Per-queue series do not come from live counters. Each instance flushes its per-queue deltas into minute buckets in PostgreSQL every METRICS_FLUSH_MS (60 s by default), and the aggregator reads the most recent bucket per queue within a five-minute window. Consequences:

• a per-queue value can be up to one flush interval stale;
• queen_queue_metrics_age_seconds is the age of the bucket the value came from (use it to distinguish “zero traffic” from “stale bucket”);
• a queue that stops receiving traffic disappears from the exposition about five minutes later, rather than reporting zero.

queen_parked_long_polls is the exception: it is the live, instantaneous per-queue count of parked long-poll pops on this instance.

Per-queue series are not tenant-scoped

No series on this endpoint carries a tenant label. When QUEEN_TENANCY_HEADER is on and two tenants hold the same queue name, their series collide:

• queen_dlq_depth_by_queue groups dead-letter rows by queue name, so the count is the sum across tenants;
• queen_parked_long_polls explicitly sums the per-tenant gauges into one line per queue name;
• the per-queue minute rates are stored per tenant but the aggregator selects one row per queue name, so one tenant’s bucket is reported and the others are hidden.

Two families you cannot alert on

queen_queue_depth_total and queen_queue_depth_pending appear in the exposition code, gated on a queue_depth key in the aggregator’s JSON. No stored procedure in this tree produces that key, so neither family is present on a real scrape. Do not build a backlog alert on them; use queen_queue_pop_lag_milliseconds and the consumer-group lag endpoints instead.

queen_dlq_depth and queen_dlq_depth_by_queue do work: they count queen.log_dlq rows, cluster total and per queue name, with an index-only scan per scrape. (Until 2026-07-31 they read the retired rows engine’s dead-letter table and were pinned at 0; if a dashboard predates that, its DLQ panel was lying.) queen_cluster_dlq_total{scope="cluster"} remains the lifetime counter of dead-letter transitions; the depth gauges shrink when DLQ rows are deleted or replayed, the counter does not.

The families

GET /metrics/prometheus exposes 50 families. queen_process_* counts what this one broker instance did since it started; queen_cluster_* are lifetime totals read back out of PostgreSQL, so every instance reports the same value.

Process (this broker instance)

Cluster lifetime totals (from PostgreSQL)

Per-queue rates and depth

Engine internals

Other

Related pages