Analytics

Ten routes live under /api/v1/analytics/. All are read-only, all dispatch one stored procedure and serve its JSON verbatim. The handlers add no envelope. None of them compute anything live: they read tables the metrics collector fills every METRICS_FLUSH_MS (default 60000 ms) on every replica, which is why a freshly started broker answers with empty series for about a minute.

Seven are scoped to the caller’s tenant. Three are cell-wide: they aggregate across every tenant on the broker and must not be exposed to a tenant.

The bucket rule

Every time-series route here derives its bucket width from the length of the window you ask for, so a wide range does not return one point per minute per queue:

Unless stated otherwise, from defaults to now() - 1 hour and to to now(). The envelope carries bucketMinutes; never assume 1-minute points.

Gauges and counters are rolled up differently, on purpose. Counters are summed. avg_lag_ms is merged as a pop-weighted average (SUM(avg × count) / SUM(count)). The parked-consumer gauge is summed across replicas within a bucket and averaged across buckets.

GET /api/v1/analytics/queue-lag

Tenant-scoped. Filters: from, to, queue. The response is a bare JSON array. There is no envelope, so the bucket width travels with every point.

[
  {
    "queueName": "orders",
    "popCount": 41200,
    "avgLagMs": 18,
    "maxLagMs": 412,
    "bucketMinutes": 1,
    "bucketTime": "2026-07-30T10:41:00Z"
  }
]

Ordered newest bucket first, then by queue name. avgLagMs and maxLagMs are null when the bucket held no pop sample.

GET /api/v1/analytics/queue-ops

Tenant-scoped. Filters: from, to, queue. The per-queue operations series the dashboard’s queue-operations view renders.

{
  "timeRange": { "from": "2026-07-30T09:44:00.000Z", "to": "2026-07-30T10:44:00.000Z" },
  "bucketMinutes": 1,
  "series": [
    {
      "bucket": "2026-07-30T10:41:00Z",
      "queueName": "orders",
      "pushRequests": 412,
      "pushMessages": 41200,
      "popMessages": 40980,
      "popEmpty": 31,
      "ackRequests": 402,
      "ackSuccess": 40970,
      "ackFailed": 10,
      "transactions": 0,
      "partitionsCreated": 0,
      "partitionsDeleted": 0,
      "partitionCount": 8,
      "avgLagMs": 18,
      "maxLagMs": 412,
      "pushPerSecond": 686.67,
      "popPerSecond": 683.0,
      "ackPerSecond": 682.97,
      "emptyPerSecond": 0.52,
      "parkedCount": 12.5
    }
  ],
  "queues": ["orders"]
}

The *PerSecond fields are the counters divided by the bucket width, computed in SQL so a client cannot get the division wrong; parkedCount is a gauge and is deliberately not rate-normalised. avgLagMs and maxLagMs are null for a bucket with no pop sample. queues lists the queue names present in the window, so a chart can build its legend before the first datapoint arrives. Points are ordered oldest first.

GET /api/v1/analytics/queue-parked-replicas

Tenant-scoped. Filters: from, to, queue. The same parked-consumer gauge as queue-ops, broken out by replica instead of aggregated.

{
  "timeRange": { "from": "2026-07-30T09:44:00.000Z", "to": "2026-07-30T10:44:00.000Z" },
  "bucketMinutes": 1,
  "series": [
    {
      "bucket": "2026-07-30T10:41:00Z",
      "queueName": "orders",
      "hostname": "queen-0",
      "workerId": 0,
      "parkedCount": 12.5
    }
  ],
  "replicas": [ { "hostname": "queen-0", "workerId": 0 } ]
}

Buckets line up exactly with queue-ops, so the two can share a time axis. parkedCount is a within-bucket average, rounded to two decimals: a broker that parks and wakes long-polls constantly will show fractional values. workerId is always 0: one async process per replica. Rows are written only for a queue that actually had parked pops in the interval.

GET /api/v1/analytics/retention

Tenant-scoped. Filters: from, to, queue. What retention and eviction actually deleted, split by rule.

{
  "timeRange": { "from": "2026-07-30T09:44:00.000Z", "to": "2026-07-30T10:44:00.000Z" },
  "bucketMinutes": 1,
  "series": [
    {
      "bucket": "2026-07-30T10:41:00Z",
      "retentionMsgs": 40960,
      "completedRetentionMsgs": 0,
      "evictionMsgs": 0,
      "totalMsgs": 40960,
      "eventCount": 5
    }
  ],
  "totals": { "retentionMsgs": 40960, "completedRetentionMsgs": 0, "evictionMsgs": 0, "totalMsgs": 40960, "eventCount": 5 }
}

The three columns map to the three rules: retentionMsgs is age-based cleanup (retentionSeconds), completedRetentionMsgs is cleanup of segments every group has consumed past (completedRetentionSeconds), and evictionMsgs is maxWaitTimeSeconds eviction, which deletes messages regardless of whether anyone consumed them, see queues. Partition create/delete events are written to the same table with reserved type names and are filtered out here.

A history row whose partition no longer resolves to a queue (the queue was deleted after the sweep) cannot be attributed and stays in the result rather than being dropped, so totals can exceed the sum of what the queue filter would show.

queen.retention_history ages out with the other metrics tables at METRICS_RETENTION_DAYS (default 90).

Grouping the retention totals

groupBy (namespace, task or queue; anything else is a 400 with {"error":"bad groupBy"}) adds a rows array next to series and totals: one entry per group with key (the empty string for queues without a namespace or task), queues (the queues that had at least one retention event in the window) and the same five totals figures for that group, ordered by totalMsgs descending. series and totals are unchanged. This is what attributes an eviction to a queue instead of to the tenant.

GET /api/v1/analytics/workload

Tenant-scoped. Filters: from, to, groupBy, namespace, task, queue. Who is doing the work, how much of it, and what is stuck right now, rolled up per namespace or per task rather than per queue. It is the one route here that answers with a group and its parts in the same payload.

groupBy takes namespace (the default), task or queue. Any other value is a 400 with {"error":"bad groupBy"}; the route does not fall back to the default. The three name filters are exact matches on the queue’s own columns and combine with AND. Bucket widths follow the same rule as the routes above.

The example is trimmed: one row of many, two of its buckets, and the tenant block collapsed. Every row carries the same three blocks.

{
  "timeRange": { "from": "2026-09-08T09:44:00.000Z", "to": "2026-09-09T09:44:00.000Z" },
  "bucketMinutes": 15,
  "groupBy": "namespace",
  "buckets": ["2026-09-08T09:45:00Z", "2026-09-08T10:00:00Z"],
  "rows": [
    {
      "key": "smartchat",
      "queues": 21,
      "window": {
        "pushMessages": 412000, "pushRequests": 4120,
        "popMessages": 409800, "popEmpty": 310,
        "ackRequests": 4020, "ackSuccess": 409700, "ackFailed": 100,
        "transactions": 0, "conflated": 0,
        "partitionsCreated": 0, "partitionsDeleted": 0,
        "parkedAvg": 12.5, "avgLagMs": 18, "maxLagMs": 412
      },
      "series": {
        "push": [41200, null], "pop": [40980, null], "popEmpty": [31, null],
        "ackFailed": [10, null], "parked": [12.5, null],
        "avgLagMs": [18, null], "maxLagMs": [412, null]
      },
      "now": {
        "pending": 1204, "processing": 8, "deadLetter": 0,
        "retainedBytes": 91230144, "partitions": 168,
        "groups": 24, "queuesWithoutGroup": 2, "pendingWithoutGroup": 0,
        "queuesTouched": 19, "queuesActive": 17
      }
    }
  ],
  "tenant": { "queues": 68, "window": {}, "series": {}, "now": {} }
}

How the three blocks are built:

• window counters are summed over every metrics row the group’s queues wrote inside [from, to]: pushMessages, pushRequests, popMessages, popEmpty, ackRequests, ackSuccess, ackFailed, transactions, conflated, partitionsCreated and partitionsDeleted.
• window.avgLagMs is pop-weighted, SUM(avg × pops) / SUM(pops) over the rows that hold a pop sample, and null when the group had no pops in the window. maxLagMs is the maximum over those same rows, and null on the same condition. A busy window with no pops is unmeasured, not zero.
• window.parkedAvg is a gauge, not a counter. It is averaged per queue across the window first, then summed across the group’s queues, so it reads as “parked consumers in this group” rather than as a total that grows with the number of buckets.
• series follows the buckets axis, one value per bucket, with push, pop, popEmpty and ackFailed summed, parked averaged per queue and then summed, and avgLagMs pop-weighted within the bucket. A bucket where the group wrote no row is null in every series, never 0. buckets runs from the floored from to the floored to whether or not each one has data, so the axis is complete even where the series is not.
• now is a snapshot and says nothing about the window. pending, processing, deadLetter, retainedBytes and partitions are the per-queue figures from queen.stats that the status routes report, summed over the group. groups, queuesWithoutGroup and pendingWithoutGroup come from queen.consumer_groups_metadata: a queue with no consumer group is a queue nobody is reading, and its pending messages are counted separately for that reason. queuesTouched and queuesActive are the only two that look at the window, counting the group’s queues that wrote any row and those that actually pushed or popped.
• queues counts the group, not the traffic: every queue matching the filters, whether or not it was touched in the window.
• namespace and task appear on a row only when groupBy is queue. A queue with an empty namespace or task groups under "", not under null.
• tenant ignores the namespace, task and queue filters and aggregates every queue of the tenant, so a row’s share is always against the whole tenant and does not move when you narrow the filters.

Every number is a JSON number and never a string. parkedAvg is rounded to two decimals, avgLagMs is integer milliseconds.

GET /api/v1/analytics/dlq-signatures

Tenant-scoped. Filters: queue (required; missing is a 400 with {"error":"queue required"}), limit (default 200, at most 1000). Why the messages of one queue are in its dead-letter queue, without transferring a single payload: the newest limit rows by failedAt are read, their error messages are folded, and the folded texts are counted.

Folding replaces UUIDs and long hex runs with <id>, dates and timestamps with <date>, standalone integers with <n>, collapses whitespace and keeps the first 120 characters, so “connection 3f2a… unresolvable” and “connection 9c1d… unresolvable” are one signature. avgBytes is the average payload length of the sample, measured server-side; the payload itself never leaves the database on this route.

{
  "queue": "channel.opsync_flush",
  "rowsNow": 3205,
  "sample": 150,
  "retryCounts": [5],
  "groups": ["channel.opsync_flush.worker"],
  "oldest": "2026-09-02T12:32:01.813Z",
  "newest": "2026-09-02T19:20:05.431Z",
  "avgBytes": 187,
  "signatures": [
    { "text": "flush: resolve connection <id>: syncwire: connection <id> unresolvable", "n": 149, "share": 0.993 }
  ],
  "byDay": [ { "day": "2026-09-02", "n": 150 } ]
}

rowsNow is the queue’s dead-letter count as /api/v1/status/queues reports it, so it can exceed sample. Signatures are the top eight by count.

GET /api/v1/analytics/partition-liveness

Tenant-scoped. Filters: queue, namespace, task (exact matches; an explicit empty namespace= selects the queues without one), limit (default 20, at most 200). How many of a queue’s partitions are alive, counted from last_write_at in one pass over queen.log_partitions: no partition row crosses the wire, which is the difference from the queue detail route on a queue with twelve thousand partitions.

{
  "capturedAt": "2026-09-09T09:34:23.098Z",
  "rows": [
    {
      "queue": "smartchat.router.history", "namespace": "smartchat", "task": "core",
      "partitions": 12611, "live1h": 812, "live24h": 7252, "live7d": 12610,
      "created24h": 4823,
      "oldestWriteAt": "2026-09-02T08:41:00Z", "newestWriteAt": "2026-09-09T09:34:00Z",
      "pending": 0
    }
  ]
}

Rows are ordered by partitions descending. pending is the queue’s figure from queen.stats, the same one /api/v1/status/queues serves. Partitions deleted in a window are not here: that is partitionsDeleted on the workload route.

GET /api/v1/analytics/system-metrics

Cell-wide. Filters: from, to, hostname, workerId. Host and process gauges per replica, re-aggregated into the window’s buckets. Every numeric leaf is a {avg, min, max, last} object.

Populated blocks: cpu.user_us and cpu.system_us (percent × 100, so divide by 100 to plot a percentage), memory.rss_bytes, database.pool_size / pool_idle / pool_active, and uptime_seconds.

GET /api/v1/analytics/worker-metrics

Cell-wide. Filters: from, to, queue, hostname, workerId. Per-replica throughput and health as a time series.

{
  "timeRange": { "from": "2026-07-30T09:44:00.000Z", "to": "2026-07-30T10:44:00.000Z" },
  "bucketMinutes": 1,
  "pointCount": 60,
  "timeSeries": [],
  "workers": [],
  "queues": [],
  "summary": {}
}

Each timeSeries point carries timestamp, the interval deltas (pushMessages, popMessages, ackMessages, pushRequests, popRequests, ackRequests, ackSuccess, ackFailed, dlqCount, dbErrors), their pushPerSecond / popPerSecond / ackPerSecond normalisations, the scheduler (“event loop”) lag pair avgEventLoopLagMs / maxEventLoopLagMs, and the pop-lag triple avgLagMs / maxLagMs / lagCount.

Four fields carry a different meaning than their names suggest, because the column set predates this broker: dbConnections is the number of active pooled connections at flush time, avgFreeSlots and minFreeSlots are both the idle pool count at that same instant, and jobsDone is written from the push-request counter, so it equals pushRequests. avgJobQueueSize, maxJobQueueSize and backoffSize have no producer at all and stay at their column default of 0. Do not chart them.

workers enumerates the replicas seen in the window, so a rolling deployment shows the old and new pods side by side. Rows are written by every replica independently, without leader election, so this is where per-replica skew that the cluster totals hide becomes visible.

GET /api/v1/analytics/postgres-stats

Cell-wide, no parameters. A direct read of PostgreSQL’s own statistics views for the current database, intended for debugging a slow broker.

Keys: timestamp, database, databaseCache (block reads, cache hits, hit ratio), tableCache and indexCache (per relation in the queen schema), cacheSummary, deadTuples, hotUpdates, activeQueries, autovacuumStatus, bufferConfig, bufferUsage and tableSizes.

Access levels

read-only here is the route level, not a tenancy boundary: the three cell-wide routes will happily serve one tenant’s read-only token the whole cell’s numbers. Restrict them at the proxy or the network. The aggregate status routes are on Status, health and metrics.