Skip to content

Prometheus metrics reference

Every Prometheus metric family a node exposes on GET /metrics/prometheus, with its type and help text, and how to scrape a cluster.

Updated View as Markdown

Every node serves Prometheus metrics in the text format on GET /metrics/prometheus. Each node describes itself: what it served, how far its log and its apply have got, how long each stage of its pipeline takes. So scrape every node of a cluster, and read a cluster as the set of its nodes. Which of these to alert on, and at what thresholds, is on monitoring.

curl -s http://localhost:6632/metrics/prometheus | grep '^queen_raft_index'
scrape_configs:
  - job_name: queen
    metrics_path: /metrics/prometheus
    static_configs:
      - targets: ["queen-0:6632", "queen-1:6632", "queen-2:6632"]

On the broker port the route needs no authentication (it is in the default JWT_SKIP_PATHS). With the embedded proxy on the public port (QUEEN_PROXY_EMBEDDED=true), /metrics and /metrics/prometheus answer only a request carrying the control-plane token (x-queen-cp-token, or Authorization: Bearer with the same value, set by QUEEN_PROXY_CP_TOKEN), and 404 to everyone else, because they describe every tenant’s queues. The other way is to give the proxy its own port with QUEEN_PROXY_PORT and scrape PORT from inside your network.

GET /metrics is a JSON summary for the dashboard: uptime, request and message totals, the process’s resident memory, and a raft block with this node’s role, whether it knows a leader, its term, applied and commit indexes, its lag and its quorumAckMs (monitoring). It is not Prometheus.

Reading the replicated log

queen_raft_index carries one series per kind: log (the last entry this node holds), committed, applied and durable (the last entry its store checkpoint covers). The series differ only by that label, so subtracting one from another needs ignoring(kind); without it PromQL matches nothing and the expression stays empty.

# entries this node knows are committed but has not applied yet
queen_raft_index{kind="committed"} - ignoring(kind) queen_raft_index{kind="applied"}

# entries committed or proposed but not applied, as the node counts them itself
queen_raft_inflight

Traffic

Three counters say what a node moves, in bytes since it started. Their rate is what the dashboard’s Cluster page shows for each node.

Family Labels What it counts
queen_raft_traffic_bytes_total channel, direction (sent, received) The bodies this node wrote and read on the Raft RPC port. Every channel is there from the start, at 0
queen_network_bytes_total direction (received, sent) Every byte on the node’s network interfaces since they came up: in a container its own, loopback, bridges, veth pairs and tunnels left out. Linux only
queen_process_disk_bytes_total direction (read, written) What the broker process read from and wrote to storage. On Linux, /proc/self/io: a read the page cache serves is not a disk read
channel Carries
replication The leader’s appends to its followers, and their answers
forward The client commands a follower hands to the leader, and their answers
snapshot A whole store, sent to a member too far behind the leader’s log
link A standby cluster’s reads of this one
control Votes, membership changes, the members view and the dashboard’s gathers

A byte is sent by one node and received by another: summed over the nodes of a cluster, sent counts each byte once.

# bytes per second on the wire, by channel, for the whole cluster
sum by (channel) (rate(queen_raft_traffic_bytes_total{direction="sent"}[1m]))

# the share of a node's network the Raft port takes
sum(rate(queen_raft_traffic_bytes_total[1m])) / sum(rate(queen_network_bytes_total[1m]))

Consumer group lag

Three gauges say how far each consumer group is behind on each queue, so an alert can fire when a group stops or cannot keep up. The raft leader writes them at every metrics bucket (METRICS_FLUSH_MS, 60 s by default) and the other nodes export none, so a query over every node sees one series per group, wherever the leader is.

Family Labels What it says
queen_consumer_group_lag_messages queue, group The messages the group has not consumed on the queue, over all its partitions
queen_consumer_group_lag_seconds queue, group How old the oldest of those messages is, in seconds; 0 when the group is caught up
queen_consumer_group_partitions_behind queue, group The partitions where the group has messages left

A group counts on every partition of the queue from the instant its subscription starts (everything for all, its first pop for new, the instant for subscriptionFrom), also on a partition it has never read: the first message of a new entity while every consumer is down is lag, and a message from before the group subscribed is not. The consumer groups list (GET /api/v1/consumer-groups) counts only the partitions the group has read from. Queue mode is the group __QUEUE_MODE__, and like every per-queue family a tenant label appears for every tenant except the default one.

Reading them is one pass over every group’s position on every partition. Past QUEEN_METRICS_GROUP_LAG_MAX_CURSORS positions (200,000 by default; 0 turns the gauges off) the leader skips the bucket rather than export a part: queen_consumer_group_lag_complete is 0 and the per-group series are absent. queen_consumer_group_lag_cursors is how many positions it read.

# a group more than five minutes behind, on any queue
max by (tenant, queue, group) (queen_consumer_group_lag_seconds) > 300

# one group's backlog, over its queues
sum by (group) (queen_consumer_group_lag_messages{group="billing"})

The repository ships a Grafana board and Prometheus alert rules on these series: deploy/grafana/queen-consumer-lag.json and deploy/prometheus/queen-consumer-lag.rules.yml.

Connections, commits and transactions

How many connections a node holds, how many commits its log takes in a second, and how many transactions it answers. The dashboard’s Cluster page charts the same three for each node.

Family Labels What it says
queen_connections listener The connections open now
queen_connections_max listener How many the listener serves at once: QUEEN_MAX_CONNECTIONS (20,000) on PORT, the proxy’s own cap on its port
queen_connections_accepted_total listener The connections accepted since the process started
queen_connections_refused_total listener The connections closed as soon as they were accepted, because the listener was at its cap
queen_raft_committed_entries_total The log entries committed, as this node knows it
queen_process_transactions_total outcome The POST /api/v1/transaction requests this node answered, by how they ended
listener Is
broker PORT: the apps inside your network, and the other nodes’ relays
proxy The embedded proxy’s own port (QUEEN_PROXY_PORT): the clients from outside. Present only when the proxy has its own port
outcome The answer
committed success: true: every push, ack and KV write of it is in the log
rolled_back HTTP 200 with success: false: a lost required KV precondition, a refused ack or item. Nothing of it was written
refused HTTP 4xx: a body the broker refuses before it plans anything
error No verdict: a standby, a scope denial, a timeout or an internal error. The outcome is unknown, and the client retries

Why a commit is not a write. One log entry is one commit: one atomic step, made durable at once. The leader packs many commands into one entry, pushes, acks and transactions from many clients together, so under load the commits a second are far fewer than the commands a second. Commands per entry is how much the batching saves (queen_raft_planner_commands_total is there unless QUEEN_RAFT_METRICS is off). Every node of a group knows the same committed index: take the max over the nodes, never the sum.

# commits a second, for the cluster
max(rate(queen_raft_committed_entries_total[1m]))

# commands per commit: what the leader planned over what the log took in
sum(rate(queen_raft_planner_commands_total[1m])) / max(rate(queen_raft_committed_entries_total[1m]))

# committed transactions a second, for the cluster, and the share that rolled back
sum(rate(queen_process_transactions_total{outcome="committed"}[1m]))
sum(rate(queen_process_transactions_total{outcome="rolled_back"}[1m])) / sum(rate(queen_process_transactions_total[1m]))

# connections open on each node, and how close a listener is to its cap
sum by (instance) (queen_connections)
queen_connections / queen_connections_max

With a client in front of a load balancer or the proxy, PORT sees the connections that hop holds, not the end clients. A refused connection is one the client sees closed at once: any rate of queen_connections_refused_total above 0 means a cap is too low or a client opens a connection per request.

Families

With QUEEN_RAFT_GROUPS above 1, each group has its own log and store, so the families that describe them (queen_raft_index, queen_raft_inflight, queen_raft_log_storage, queen_raft_proposals_total and the queen_raft_store_* families) carry a group label.

GET /metrics/prometheus exposes 152 families. Every one describes the node that answered: queen_process_* counts what this node did since it started, and the queen_raft_* families describe its replicated log, its store and its pipeline. Scrape every node. The pipeline timing, admission and forwarding families are skipped when QUEEN_RAFT_METRICS is 0, false, off or no; it is on by default. The per-queue families are written only for queues that have something to report. 27 families are declared but not written by this release; their Help says so.

Process (this broker instance)

Family Type Help
queen_event_loop_lag_avg_milliseconds gauge Mean scheduler (event-loop) lag since start
queen_malloc_bytes gauge glibc heap: in use, free inside the arenas, arena total, mmapped chunks
queen_memguard_level gauge Memory guard level: 0 ok, 1 soft (heavy reads refused), 2 hard (storage-growing writes refused too)
queen_memguard_limit_bytes gauge Memory limit the guard works against (cgroup memory.max or QUEEN_MEMORY_LIMIT_BYTES); 0 = guard off
queen_memguard_refused_total counter Requests refused by the memory guard: memory_pressure, or busy (no heavy-read slot)
queen_memguard_rss_anon_bytes gauge Anonymous resident memory of this process at the guard’s last tick
queen_parked_long_polls gauge Currently parked long-poll pops on this process
queen_parked_pops gauge Long-poll pops parked on this node’s consumption engine (capped by QUEEN_MAX_PARKED_POPS)
queen_process_ack_messages_total counter Messages acked by this process
queen_process_ack_requests_total counter Ack API requests handled by this process
queen_process_disk_bytes_total counter Bytes this process read from and wrote to storage (on Linux /proc/self/io; a read the page cache serves is not counted)
queen_process_pop_messages_total counter Messages popped by this process
queen_process_pop_requests_total counter Pop API requests handled by this process
queen_process_push_messages_total counter Messages pushed by this process
queen_process_push_requests_total counter Push API requests handled by this process
queen_process_resident_memory_bytes gauge Resident memory of this process
queen_process_transactions_total counter Transaction requests answered by this process, by outcome
queen_uptime_seconds gauge Process uptime in seconds

Replicated log and storage

Family Type Help
queen_raft_index gauge Raft/RSM indexes
queen_raft_inflight gauge Entries committed or proposed but not applied
queen_raft_log_storage gauge Queue-log files and bytes
queen_raft_proposals_total counter Proposals accepted since open
queen_raft_storage_full gauge Disk/map admission gate
queen_raft_store_commit_seconds summary Store commit duration
queen_raft_store_map_bytes gauge LMDB map capacity and use
queen_raft_store_operations_total counter Embedded-store operations by kind
queen_raft_store_ram_rows gauge Rows per RAM keyspace: live, and keys dirty since the last checkpoint
queen_raft_store_readers gauge LMDB reader slots

Admission

Family Type Help
queen_raft_admit_bytes gauge Admission budget and bytes held by storage-growing commands in the pipeline
queen_raft_admit_forwarded_total counter Commands followers forwarded to this node that it admitted, and that it refused as overloaded
queen_raft_admit_total counter Commands that waited for admission room, and that were refused with 429
queen_raft_admit_waiting gauge Commands waiting for admission room now

Forwarding between nodes

Family Type Help
queen_raft_forward_total counter Batched forwarding: commands this node sent to the leader, commands it served as the leader, streams it opened, and leaders it found without batched forwarding

Standby cluster

Family Type Help
queen_link_entries_total counter Source entries this node replayed since it started
queen_link_expired_at_source_total counter Links whose message its source no longer keeps, skipped on read
queen_link_follower_state gauge What this node’s follower of the source is doing: 1 on its state (it follows on the standby’s leader only)
queen_link_holding gauge Whether this node keeps its log for its standbys: 0 while its data volume is too full (QUEEN_LINK_HOLD_DISK_PCT)
queen_link_lag_entries gauge Entries of the source’s log this standby had still to read when the source last answered
queen_link_lag_seconds gauge The age of the last source entry this standby applied, while the source is known to hold more; 0 when it holds everything the source had
queen_link_last_answer_seconds gauge Seconds since the source last answered this node’s follower
queen_link_position_index gauge The index, in the source’s log, of the last source entry this cluster applied as a standby
queen_link_reader_behind_entries gauge Entries of this node’s log a standby had still to apply when it last said where it was
queen_link_reader_idle_seconds gauge Seconds since a standby last said where it was
queen_link_readers gauge Standbys this node keeps its log for
queen_link_reads_total counter Links read through to their message on this node
queen_link_role gauge This cluster’s part in a cluster link: 1 on the role it has

Pipeline timing

Family Type Help
queen_raft_apply_channel_depth summary Apply channel depth at receive
queen_raft_apply_entry_seconds summary Per-entry apply duration (total)
queen_raft_apply_other_seconds summary Per-entry apply remainder (store puts + dispatch + derived)
queen_raft_apply_receives_total counter Apply-channel receives
queen_raft_apply_segment_seconds summary Per-entry segment write_all duration
queen_raft_apply_stats counter Apply thread counters (ApplyStats)
queen_raft_arrival_to_proposed_seconds summary Command arrival to its entry proposed
queen_raft_checkpoint_rows summary Rows per async checkpoint cut
queen_raft_checkpoint_write_seconds summary Async durable point: the checkpoint cut written, committed and synced (checkpoint thread)
queen_raft_committed_entries_total counter Raft log entries committed, as this node knows it (each is one atomic commit of a batch of commands)
queen_raft_drain_commands summary Drain size in commands
queen_raft_drain_messages summary Drain size in messages
queen_raft_drain_to_propose_seconds summary Cycle drain to its entry proposed (PERF-G leg 2)
queen_raft_durable_dir_fsync_seconds summary Durable point directory fsync duration
queen_raft_durable_point_seconds summary Durable point duration (total)
queen_raft_durable_seg_fsync_seconds summary Durable point segment-file fsync duration
queen_raft_group_bytes summary Log group size in bytes
queen_raft_group_entries summary Log group size in entries
queen_raft_keep_overlay_total counter Planning cycles by what the kept overlay did
queen_raft_log_fsync_seconds summary Log group fsync duration
queen_raft_plan_cpu_seconds summary Planner command loop, thread CPU time
queen_raft_plan_deferred summary Commands drained but deferred unplanned by the plan budget cut
queen_raft_plan_seconds summary Planner cycle duration
queen_raft_plan_whole_cpu_seconds summary Whole plan_cycle_blocking call, thread CPU
queen_raft_plan_whole_wall_seconds summary Whole plan_cycle_blocking call, wall
queen_raft_planner_commands_total counter Commands planned, by kind
queen_raft_planner_cycles_total counter Planning cycles by how the footprint scheduler planned them
queen_raft_planner_plan_seconds summary Per-command planning duration, by kind
queen_raft_planner_slow_total counter Slow-planned commands, by kind (O18)
queen_raft_pop_h_total_seconds summary PERF-J: whole raft pop handler (entry to response), pop-only
queen_raft_pop_read_seconds summary Pop payload read (segment render) latency
queen_raft_propose_roundtrip_seconds summary Entry proposed to its waiters answered (commit+apply+notify)
queen_raft_proposed_to_committed_seconds summary Entry proposed to its group fsynced
queen_raft_push_h_await_seconds summary PERF-J: push reply wait (propose+commit+apply+answer), push-only
queen_raft_push_h_prep_seconds summary PERF-J: push pre-submit (parse+pack), entry to first channel send
queen_raft_push_h_submit_seconds summary PERF-J: push channel enqueue (cmd_tx.send await), back-pressure
queen_raft_push_h_total_seconds summary PERF-J: whole raft push handler (entry to response), push-only
queen_raft_qlog_dirx gauge The queue logs’ cross-file directory: runs on disk, and sealed files no run covers yet (a lookup probes those one by one)
queen_raft_qlog_early_codec_hits_total counter Append blobs whose compression the facade started before planning
queen_raft_qlog_early_codec_waiting gauge Early compression jobs not yet taken by a proposal
queen_raft_qlog_index_maps gauge Sealed log files whose index is memory-mapped (small indexes are held in memory), the maps the kernel allows this process (vm.max_map_count, 0 when unknown), and the maps it has refused since the start (those indexes are held in memory too)
queen_raft_queue_wait_seconds summary Command arrival to the cycle that drained it (PERF-G leg 1)
queen_raft_slow_commands_total counter Commands over QUEEN_RAFT_SLOW_COMMAND_MS
queen_raft_traces_map_bytes gauge Trace store (LMDB) map capacity and use
queen_raft_traces_stored gauge Traces in the trace store
queen_raft_traffic_bytes_total counter Bytes this node sent and received on the Raft RPC port, by channel (bodies only)
queen_raft_writer_pickup_seconds summary Entry proposed to the writer starting its group fsync (PERF-G)

Per-queue rates and depth

Family Type Help
queen_dlq_depth_by_queue gauge Dead letters per queue (exported by the raft leader only)
queen_queue_conflated_per_minute gauge Messages conflated away per queue in the last metrics bucket (METRICS_FLUSH_MS)
queen_queue_pop_lag_milliseconds gauge Per-queue pop lag (delivery time minus creation time) over the last metrics bucket (METRICS_FLUSH_MS), on this process

Engine internals

Family Type Help
queen_batch_items_fired_total counter Items flushed across fusion batches. Nothing in this release writes it, so it reads 0.
queen_batch_rtt_milliseconds gauge Fusion batch round-trip latency. Nothing in this release writes it, so it reads 0.
queen_batches_fired_total counter Fusion batches flushed. Nothing in this release writes it, so it reads 0.
queen_fusion_items_per_batch gauge Mean items per fusion batch. Nothing in this release writes it, so it reads 0.
queen_pop_fill_wait_microseconds_total counter Total time pops spent fattening an under-full batch. Nothing in this release writes it, so it reads 0.
queen_pop_fill_wait_total counter Pops that held an under-full batch back (minPopWaitTime). Nothing in this release writes it, so it reads 0.
queen_pop_targeted_total counter Hinted targeted single-partition pops issued. Nothing in this release writes it, so it reads 0.
queen_pop_wildcard_total counter Wildcard candidate-scan pops issued. Nothing in this release writes it, so it reads 0.

Ephemeral queues

Family Type Help
queen_ephemeral_bytes gauge Bytes held by this broker’s ephemeral rings
queen_ephemeral_dropped_total counter Ephemeral messages dropped, by cause
queen_ephemeral_forwarded_total counter Ephemeral requests relayed to the partition’s rendezvous owner
queen_ephemeral_messages_total counter Ephemeral messages by verb
queen_ephemeral_queues gauge Ephemeral queues on this broker (declared + live implicit)
queen_ephemeral_wipes_total counter Ephemeral rings dropped because their partition moved owner

Key/value state, timers and the sweeper

Family Type Help
queen_kv_bytes_total counter KV value bytes written / read
queen_kv_expired_not_pruned gauge Expired KV rows the sweeper has not pruned yet (capped). Nothing in this release writes it, so it reads 0.
queen_kv_expired_not_pruned_capped gauge 1 when the unpruned count hit its cap and is a floor. Nothing in this release writes it, so it reads 0.
queen_kv_expiry_lag_seconds gauge Age of the oldest expired, unpruned KV row. Nothing in this release writes it, so it reads 0.
queen_kv_op_duration_milliseconds gauge KV operation latency
queen_kv_ops_total counter KV operations by code path and outcome
queen_kv_pool gauge Dedicated KV connection pool. Nothing in this release writes it, so it reads 0.
queen_kv_read_rejected_total counter KV reads refused before reaching the database
queen_kv_singleflight_coalesced_total counter KV reads that shared an in-flight query. Nothing in this release writes it, so it reads 0.
queen_sweeper_cycle_milliseconds gauge Sweeper phase duration. Nothing in this release writes it, so it reads 0.
queen_sweeper_phase_skipped_total counter Phases shed under pressure (the degradation ladder, made visible). Nothing in this release writes it, so it reads 0.
queen_sweeper_rows_total counter Rows handled by each sweeper phase. Nothing in this release writes it, so it reads 0.
queen_sweeper_skip_locked_total counter Rows another broker was already holding. Nothing in this release writes it, so it reads 0.
queen_sweeper_sleep_milliseconds gauge Sleep the sweeper chose after the last cycle. Nothing in this release writes it, so it reads 0.
queen_timers_dlq_total counter Timers dead-lettered after exhausting attempts. Nothing in this release writes it, so it reads 0.
queen_timers_due gauge Timers due now, from the sweep probe (capped). Nothing in this release writes it, so it reads 0.
queen_timers_due_capped gauge 1 when the due count hit its cap and is a floor. Nothing in this release writes it, so it reads 0.
queen_timers_fire_failures_total counter Failed fire transactions by SQLSTATE class. Nothing in this release writes it, so it reads 0.
queen_timers_fire_lag_seconds gauge Delivery lateness of fired timers, per tenant. Nothing in this release writes it, so it has no samples.
queen_timers_fire_lag_tenants_dropped_total counter Fire-lag samples dropped because the tenant cap was reached. Nothing in this release writes it, so it reads 0.
queen_timers_fired_total counter Fired timer segments by outcome. Nothing in this release writes it, so it reads 0.
queen_timers_oldest_late_seconds gauge Lateness of the oldest due timer. Nothing in this release writes it, so it reads 0.
queen_timers_poisoned_total counter Batches replayed one segment per call after a permanent error. Nothing in this release writes it, so it reads 0.
queen_timers_schedule_rejected_total counter Timer schedules refused

Other

Family Type Help
queen_connections gauge Connections open now, by listener: broker = PORT (clients in the cluster and the other nodes’ relays), proxy = the embedded proxy’s own port
queen_connections_accepted_total counter Connections accepted since the process started, by listener
queen_connections_max gauge Connections the listener serves at once (QUEEN_MAX_CONNECTIONS on PORT, the proxy’s cap on its port)
queen_connections_refused_total counter Connections closed at accept because the listener’s cap was reached
queen_consume_cold_claims_total counter Claims of messages whose index row had left the store: their frames were read from the queue log, with no consume lock held
queen_consume_engine gauge What the consumption engine holds in memory (only the leader holds any): groups, (group, partition) parts, partitions
queen_consume_parts_total counter Parts the engine loaded (first contacts and loads back) and unloaded (idle)
queen_consumer_group_lag_complete gauge 1 when the last metrics bucket (METRICS_FLUSH_MS) read every consumer group’s cursors; 0 when QUEEN_METRICS_GROUP_LAG_MAX_CURSORS stopped the read and the per-group families are left out (exported by the raft leader only)
queen_consumer_group_lag_cursors gauge Cursors (one per consumer group and partition) read for the per-group families at the last metrics bucket (exported by the raft leader only)
queen_consumer_group_lag_messages gauge Messages a consumer group has not consumed on a queue, summed over the queue’s partitions, at the last metrics bucket (exported by the raft leader only)
queen_consumer_group_lag_seconds gauge Age of the oldest message a consumer group has not consumed on a queue, 0 when it is caught up, at the last metrics bucket (exported by the raft leader only)
queen_consumer_group_partitions_behind gauge Partitions of a queue where a consumer group has messages left to consume, at the last metrics bucket (exported by the raft leader only)
queen_holder_floor_age_seconds gauge Per source queue, how old the oldest message its reading queues, filtered groups and link targets keep unconsumed is: how far back they hold its completed retention
queen_locks_op_duration_milliseconds gauge Lock operation latency
queen_locks_ops_total counter Lock operations by kind and outcome (rejected = held by another, or a token that is no longer the row’s)
queen_network_bytes_total counter Bytes received and sent on this node’s network interfaces since they came up (in a container its own; loopback, bridges, veth pairs and tunnels left out)
queen_route_links_total counter Links this node’s planner appended to reading queues and filtered groups
queen_route_unrouted_total counter Messages a reader’s where took but whose partitionBy field was missing, so not routed

The per-queue families carry a queue label, and a tenant label for every tenant except the default one; queen_queue_pop_lag_milliseconds adds stat (avg or max), measured over the last metrics bucket (METRICS_FLUSH_MS, 60 s by default). queen_raft_forward_total is split by kind: sent (commands this node sent to the leader), served (commands it served as the leader), streams_opened, and legacy_leader (leaders it found that do not take batched forwarding).

Some families come from earlier engines and nothing in 2.0 writes them, among them most of the queen_timers_* and every queen_sweeper_* family: the table says so on each one, so do not build an alert on them. A queen_raft_dbg family may also appear: those are temporary diagnostic counters, and they may change or disappear in any release.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close