---
title: "Metrics"
description: "Every Prometheus metric family a node exposes on GET /metrics/prometheus, with its type and help text, and how to scrape a cluster."
---

> Queen MQ documentation, for AI agents
> Complete self-contained summary of Queen MQ: https://queenmq.com/llms-brief.txt
> Fetch that first when the question is about the product rather than about this page.
> Index of all pages: https://queenmq.com/llms.txt

# Metrics

Every node serves Prometheus metrics in the text format on `GET /metrics/prometheus`. Each node
describes itself: what it served, how far its log and its apply have got, how long each stage of
its pipeline takes. So scrape every node of a cluster, and read a cluster as the set of its
nodes. Which of these to alert on, and at what thresholds, is on [monitoring](/operate/monitoring/).

```bash
curl -s http://localhost:6632/metrics/prometheus | grep '^queen_raft_index'
```

```yaml
scrape_configs:
  - job_name: queen
    metrics_path: /metrics/prometheus
    static_configs:
      - targets: ["queen-0:6632", "queen-1:6632", "queen-2:6632"]
```

On the broker port the route needs no authentication (it is in the default `JWT_SKIP_PATHS`). With
the embedded proxy on the public port (`QUEEN_PROXY_EMBEDDED=true`), `/metrics` and
`/metrics/prometheus` answer only a request carrying the control-plane token (`x-queen-cp-token`,
or `Authorization: Bearer` with the same value, set by `QUEEN_PROXY_CP_TOKEN`), and 404 to
everyone else, because they describe every tenant's queues. The other way is to give the proxy its
own port with `QUEEN_PROXY_PORT` and scrape `PORT` from inside your network.

`GET /metrics` is a JSON summary for the dashboard: `uptime`, request and message totals, the
process's resident memory, and a `raft` block with this node's `role`, whether it knows a
`leader`, its `term`, `applied` and `commit` indexes and its `lag`. It is not Prometheus.

## Reading the replicated log

`queen_raft_index` carries one series per `kind`: `log` (the last entry this node holds),
`committed`, `applied` and `durable` (the last entry its store checkpoint covers). The series
differ only by that label, so subtracting one from another needs `ignoring(kind)`; without it
PromQL matches nothing and the expression stays empty.

```text
# entries this node knows are committed but has not applied yet
queen_raft_index{kind="committed"} - ignoring(kind) queen_raft_index{kind="applied"}

# entries committed or proposed but not applied, as the node counts them itself
queen_raft_inflight
```

## Families

With `QUEEN_RAFT_GROUPS` above 1, each group has its own log and store, so the families that
describe them (`queen_raft_index`, `queen_raft_inflight`, `queen_raft_log_storage`,
`queen_raft_proposals_total` and the `queen_raft_store_*` families) carry a `group` label.

`GET /metrics/prometheus` exposes **107 families**. Every one describes the node that answered: `queen_process_*` counts what this node did since it started, and the `queen_raft_*` families describe its replicated log, its store and its pipeline. Scrape every node. The pipeline timing, admission and forwarding families are skipped when `QUEEN_RAFT_METRICS` is `0`, `false`, `off` or `no`; it is on by default. The per-queue families are written only for queues that have something to report. 27 families are declared but not written by this release; their Help says so.

### Process (this broker instance)

| Family | Type | Help |
| --- | --- | --- |
| `queen_event_loop_lag_avg_milliseconds` | gauge | Mean scheduler (event-loop) lag since start |
| `queen_malloc_bytes` | gauge | glibc heap: in use, free inside the arenas, arena total, mmapped chunks |
| `queen_parked_long_polls` | gauge | Currently parked long-poll pops on this process |
| `queen_process_ack_messages_total` | counter | Messages acked by this process |
| `queen_process_ack_requests_total` | counter | Ack API requests handled by this process |
| `queen_process_pop_messages_total` | counter | Messages popped by this process |
| `queen_process_pop_requests_total` | counter | Pop API requests handled by this process |
| `queen_process_push_messages_total` | counter | Messages pushed by this process |
| `queen_process_push_requests_total` | counter | Push API requests handled by this process |
| `queen_process_resident_memory_bytes` | gauge | Resident memory of this process |
| `queen_uptime_seconds` | gauge | Process uptime in seconds |

### Replicated log and storage

| Family | Type | Help |
| --- | --- | --- |
| `queen_raft_index` | gauge | Raft/RSM indexes |
| `queen_raft_inflight` | gauge | Entries committed or proposed but not applied |
| `queen_raft_log_storage` | gauge | Queue-log files and bytes |
| `queen_raft_proposals_total` | counter | Proposals accepted since open |
| `queen_raft_storage_full` | gauge | Disk/map admission gate |
| `queen_raft_store_commit_seconds` | summary | Store commit duration |
| `queen_raft_store_map_bytes` | gauge | LMDB map capacity and use |
| `queen_raft_store_operations_total` | counter | Embedded-store operations by kind |
| `queen_raft_store_ram_rows` | gauge | Rows per RAM keyspace: live, and keys dirty since the last checkpoint |
| `queen_raft_store_readers` | gauge | LMDB reader slots |

### Admission

| Family | Type | Help |
| --- | --- | --- |
| `queen_raft_admit_bytes` | gauge | Admission budget and bytes held by storage-growing commands in the pipeline |
| `queen_raft_admit_forwarded_total` | counter | Commands followers forwarded to this node that it admitted, and that it refused as overloaded |
| `queen_raft_admit_total` | counter | Commands that waited for admission room, and that were refused with 429 |
| `queen_raft_admit_waiting` | gauge | Commands waiting for admission room now |

### Forwarding between nodes

| Family | Type | Help |
| --- | --- | --- |
| `queen_raft_forward_total` | counter | Batched forwarding: commands this node sent to the leader, commands it served as the leader, streams it opened, and leaders it found without batched forwarding |

### Pipeline timing

| Family | Type | Help |
| --- | --- | --- |
| `queen_raft_apply_channel_depth` | summary | Apply channel depth at receive |
| `queen_raft_apply_entry_seconds` | summary | Per-entry apply duration (total) |
| `queen_raft_apply_other_seconds` | summary | Per-entry apply remainder (store puts + dispatch + derived) |
| `queen_raft_apply_receives_total` | counter | Apply-channel receives |
| `queen_raft_apply_segment_seconds` | summary | Per-entry segment write_all duration |
| `queen_raft_apply_stats` | counter | Apply thread counters (ApplyStats) |
| `queen_raft_arrival_to_proposed_seconds` | summary | Command arrival to its entry proposed |
| `queen_raft_checkpoint_rows` | summary | Rows per async checkpoint cut |
| `queen_raft_checkpoint_write_seconds` | summary | Async durable point: the checkpoint cut written, committed and synced (checkpoint thread) |
| `queen_raft_drain_commands` | summary | Drain size in commands |
| `queen_raft_drain_messages` | summary | Drain size in messages |
| `queen_raft_drain_to_propose_seconds` | summary | Cycle drain to its entry proposed (PERF-G leg 2) |
| `queen_raft_durable_dir_fsync_seconds` | summary | Durable point directory fsync duration |
| `queen_raft_durable_point_seconds` | summary | Durable point duration (total) |
| `queen_raft_durable_seg_fsync_seconds` | summary | Durable point segment-file fsync duration |
| `queen_raft_group_bytes` | summary | Log group size in bytes |
| `queen_raft_group_entries` | summary | Log group size in entries |
| `queen_raft_keep_overlay_total` | counter | Planning cycles by what the kept overlay did |
| `queen_raft_log_fsync_seconds` | summary | Log group fsync duration |
| `queen_raft_plan_cpu_seconds` | summary | Planner command loop, thread CPU time |
| `queen_raft_plan_deferred` | summary | Commands drained but deferred unplanned by the plan budget cut |
| `queen_raft_plan_seconds` | summary | Planner cycle duration |
| `queen_raft_plan_whole_cpu_seconds` | summary | Whole plan_cycle_blocking call, thread CPU |
| `queen_raft_plan_whole_wall_seconds` | summary | Whole plan_cycle_blocking call, wall |
| `queen_raft_planner_commands_total` | counter | Commands planned, by kind |
| `queen_raft_planner_plan_seconds` | summary | Per-command planning duration, by kind |
| `queen_raft_planner_slow_total` | counter | Slow-planned commands, by kind (O18) |
| `queen_raft_pop_h_total_seconds` | summary | PERF-J: whole raft pop handler (entry to response), pop-only |
| `queen_raft_pop_read_seconds` | summary | Pop payload read (segment render) latency |
| `queen_raft_propose_roundtrip_seconds` | summary | Entry proposed to its waiters answered (commit+apply+notify) |
| `queen_raft_proposed_to_committed_seconds` | summary | Entry proposed to its group fsynced |
| `queen_raft_push_h_await_seconds` | summary | PERF-J: push reply wait (propose+commit+apply+answer), push-only |
| `queen_raft_push_h_prep_seconds` | summary | PERF-J: push pre-submit (parse+pack), entry to first channel send |
| `queen_raft_push_h_submit_seconds` | summary | PERF-J: push channel enqueue (cmd_tx.send await), back-pressure |
| `queen_raft_push_h_total_seconds` | summary | PERF-J: whole raft push handler (entry to response), push-only |
| `queen_raft_qlog_early_codec_hits_total` | counter | Append blobs whose compression the facade started before planning |
| `queen_raft_qlog_early_codec_waiting` | gauge | Early compression jobs not yet taken by a proposal |
| `queen_raft_queue_wait_seconds` | summary | Command arrival to the cycle that drained it (PERF-G leg 1) |
| `queen_raft_slow_commands_total` | counter | Commands over QUEEN_RAFT_SLOW_COMMAND_MS |
| `queen_raft_writer_pickup_seconds` | summary | Entry proposed to the writer starting its group fsync (PERF-G) |

### Per-queue rates and depth

| Family | Type | Help |
| --- | --- | --- |
| `queen_dlq_depth_by_queue` | gauge | Dead letters per queue (exported by the raft leader only) |
| `queen_queue_conflated_per_minute` | gauge | Messages conflated away per queue in the last metrics bucket (METRICS_FLUSH_MS) |
| `queen_queue_pop_lag_milliseconds` | gauge | Per-queue pop lag (delivery time minus creation time) over the last metrics bucket (METRICS_FLUSH_MS), on this process |

### Engine internals

| Family | Type | Help |
| --- | --- | --- |
| `queen_batch_items_fired_total` | counter | Items flushed across fusion batches. Nothing in this release writes it, so it reads 0. |
| `queen_batch_rtt_milliseconds` | gauge | Fusion batch round-trip latency. Nothing in this release writes it, so it reads 0. |
| `queen_batches_fired_total` | counter | Fusion batches flushed. Nothing in this release writes it, so it reads 0. |
| `queen_fusion_items_per_batch` | gauge | Mean items per fusion batch. Nothing in this release writes it, so it reads 0. |
| `queen_pop_fill_wait_microseconds_total` | counter | Total time pops spent fattening an under-full batch. Nothing in this release writes it, so it reads 0. |
| `queen_pop_fill_wait_total` | counter | Pops that held an under-full batch back (minPopWaitTime). Nothing in this release writes it, so it reads 0. |
| `queen_pop_targeted_total` | counter | Hinted targeted single-partition pops issued. Nothing in this release writes it, so it reads 0. |
| `queen_pop_wildcard_total` | counter | Wildcard candidate-scan pops issued. Nothing in this release writes it, so it reads 0. |

### Ephemeral queues

| Family | Type | Help |
| --- | --- | --- |
| `queen_ephemeral_bytes` | gauge | Bytes held by this broker's ephemeral rings |
| `queen_ephemeral_dropped_total` | counter | Ephemeral messages dropped, by cause |
| `queen_ephemeral_forwarded_total` | counter | Ephemeral requests relayed to the partition's rendezvous owner |
| `queen_ephemeral_messages_total` | counter | Ephemeral messages by verb |
| `queen_ephemeral_queues` | gauge | Ephemeral queues on this broker (declared + live implicit) |
| `queen_ephemeral_wipes_total` | counter | Ephemeral rings dropped because their partition moved owner |

### Key/value state, timers and the sweeper

| Family | Type | Help |
| --- | --- | --- |
| `queen_kv_bytes_total` | counter | KV value bytes written / read |
| `queen_kv_expired_not_pruned` | gauge | Expired KV rows the sweeper has not pruned yet (capped). Nothing in this release writes it, so it reads 0. |
| `queen_kv_expired_not_pruned_capped` | gauge | 1 when the unpruned count hit its cap and is a floor. Nothing in this release writes it, so it reads 0. |
| `queen_kv_expiry_lag_seconds` | gauge | Age of the oldest expired, unpruned KV row. Nothing in this release writes it, so it reads 0. |
| `queen_kv_op_duration_milliseconds` | gauge | KV operation latency |
| `queen_kv_ops_total` | counter | KV operations by code path and outcome |
| `queen_kv_pool` | gauge | Dedicated KV connection pool. Nothing in this release writes it, so it reads 0. |
| `queen_kv_read_rejected_total` | counter | KV reads refused before reaching the database |
| `queen_kv_singleflight_coalesced_total` | counter | KV reads that shared an in-flight query. Nothing in this release writes it, so it reads 0. |
| `queen_sweeper_cycle_milliseconds` | gauge | Sweeper phase duration. Nothing in this release writes it, so it reads 0. |
| `queen_sweeper_phase_skipped_total` | counter | Phases shed under pressure (the degradation ladder, made visible). Nothing in this release writes it, so it reads 0. |
| `queen_sweeper_rows_total` | counter | Rows handled by each sweeper phase. Nothing in this release writes it, so it reads 0. |
| `queen_sweeper_skip_locked_total` | counter | Rows another broker was already holding. Nothing in this release writes it, so it reads 0. |
| `queen_sweeper_sleep_milliseconds` | gauge | Sleep the sweeper chose after the last cycle. Nothing in this release writes it, so it reads 0. |
| `queen_timers_dlq_total` | counter | Timers dead-lettered after exhausting attempts. Nothing in this release writes it, so it reads 0. |
| `queen_timers_due` | gauge | Timers due now, from the sweep probe (capped). Nothing in this release writes it, so it reads 0. |
| `queen_timers_due_capped` | gauge | 1 when the due count hit its cap and is a floor. Nothing in this release writes it, so it reads 0. |
| `queen_timers_fire_failures_total` | counter | Failed fire transactions by SQLSTATE class. Nothing in this release writes it, so it reads 0. |
| `queen_timers_fire_lag_seconds` | gauge | Delivery lateness of fired timers, per tenant. Nothing in this release writes it, so it has no samples. |
| `queen_timers_fire_lag_tenants_dropped_total` | counter | Fire-lag samples dropped because the tenant cap was reached. Nothing in this release writes it, so it reads 0. |
| `queen_timers_fired_total` | counter | Fired timer segments by outcome. Nothing in this release writes it, so it reads 0. |
| `queen_timers_oldest_late_seconds` | gauge | Lateness of the oldest due timer. Nothing in this release writes it, so it reads 0. |
| `queen_timers_poisoned_total` | counter | Batches replayed one segment per call after a permanent error. Nothing in this release writes it, so it reads 0. |
| `queen_timers_schedule_rejected_total` | counter | Timer schedules refused |

The per-queue families carry a `queue` label, and a `tenant` label for every tenant except the
default one; `queen_queue_pop_lag_milliseconds` adds `stat` (`avg` or `max`), measured over the last
metrics bucket (`METRICS_FLUSH_MS`, 60 s by default). `queen_raft_forward_total` is split by
`kind`: `sent` (commands this node sent to the leader), `served` (commands it served as the
leader), `streams_opened`, and `legacy_leader` (leaders it found that do not take batched
forwarding).

Some families come from earlier engines and nothing in 2.0 writes them, among them most of the
`queen_timers_*` and every `queen_sweeper_*` family: the table says so on each one, so do not build
an alert on them. A `queen_raft_dbg` family may also appear: those are temporary diagnostic
counters, and they may change or disappear in any release.

Source: https://queenmq.com/reference/metrics/index.mdx