Queues and resources

A queue does not have to be created before it is used: the first push to a queue name creates the queue, one partition, and a queen.queues configuration row with column defaults. It also derives namespace and task from a dotted queue name (billing.invoice becomes namespace billing, task invoice), so a push-only queue is still visible to namespace and task discovery pops. POST /api/v1/configure exists to set options that differ from the defaults, and to pre-create a queue so a consumer can attach before any producer runs.

A pop can create the row too: a queue-scoped pop on a name that does not exist creates the queen.queues row with column defaults, because a consumer group may legitimately subscribe before the first push and its durable subscription record hangs off the queue’s id. The practical consequence is that a consumer-side typo now creates an empty queue rather than nothing.

Two routes mutate queue state: POST /api/v1/configure (read-write) and DELETE /api/v1/resources/queues/:queue (admin). Everything else on this page is a read at the read-only access level.

POST /api/v1/configure

queue is the only required field. Options may be nested under options or sent at the top level; the broker normalises them into one bag before calling queen.configure_queue_v1. Top-level namespace and task are folded into the options when they are non-empty strings. Two other top-level keys are routing, not configuration: options itself, and mode, which decides whether the call merges or replaces.

{
  "queue": "orders",
  "namespace": "billing",
  "task": "invoice",
  "options": {
    "leaseTime": 120,
    "retryLimit": 5,
    "dedupWindowSeconds": 600
  }
}

An empty string is a valid queue name: queen.configure_queue_v1 creates a row named '', and the JavaScript client’s own load test relies on it. A missing or non-string queue is a 400.

Configure merges, unless you ask it to replace

An edit to an existing queue keeps every option the body does not mention. Four rules cover all 21 options, namespace and task included, and queen.configure_queue_v1 applies them one option at a time:

A queue that does not exist yet is created from defaults whichever mode you send: there is nothing to keep. So on a create the two modes are the same call, and on an edit they are opposites.

Merge is the default because every client sends only the options its caller set. Before 1.6.0 every call replaced, so queenctl queue configure orders --lease-time 60 reset dedupWindowSeconds to 3600, retention to off and the sink hold to off, silently, on a queue the operator meant only to nudge. An editor built on that is a reset button with a friendlier face. What changed is the default, not the capability: "mode": "replace" is the old behaviour, and it is what queenctl apply -f sends, because a manifest means the whole configuration and anything it leaves out should go back to the default.

Concurrent edits of one queue serialise: the stored procedure takes SELECT ... FOR UPDATE on the queue row first, so the second caller merges onto the first one’s result rather than onto the row it read before waiting. Concurrent creates take no lock (there is no row yet) and stay on the ON CONFLICT path they always had.

Enforced options

These are the options this engine reads. Each row names what enforces it, so you can predict the behaviour change from setting it.

Values are stored as given. The broker does not validate ranges, apart from the SQL clamp on minPopWaitTime, a max(0) floor on dedupWindowSeconds, and the two sink-hold options, which are rejected rather than clamped.

Seventeen rows for twenty-one keys. The four this table leaves out, priority, ttl, maxSize and retryDelay, are stored, echoed back and enforced by nothing at all: no push, pop, ack or maintenance path reads any of them. They are listed with what does read them, which is only ever an echo, under Queue options.

Disabling the dead-letter queue takes both flags

005_log_ack.sql computes the DLQ switch as COALESCE(dead_letter_queue, true) OR COALESCE(dlq_after_max_retries, true). Setting only one of them to false leaves dead-lettering on. Send both as false to turn it off, at which point an exhausted retry budget drops the poison frame instead of quarantining it.

maxWaitTimeSeconds deletes messages

maxWaitTimeSeconds is not a delivery deadline and it does not dead-letter. queen.log_evict_max_wait_step_v1 delegates to the same delete-and-advance path retention uses, with the cutoff taken from queen.queues.max_wait_time_seconds: whole segments older than the cutoff are removed for every consumer group, including groups holding a live lease on them. The SQL calls this data loss by design. A cursor left below the new log start resumes at the next existing offset; the pop scan tolerates the gap. Use it only where dropping stale work is the intent.

The sink hold is rejected out of range, never clamped

retentionSinkHold and retentionSinkHoldMaxSeconds are the only two options 012_configure.sql refuses. A name outside [A-Za-z0-9._-]{0,64}, or a cap outside 60 to 31,536,000 seconds, answers 400 with {"error":"...","invalid":"<the option>"} and the queue keeps the configuration it had; nothing partial is written. invalid is what tells the handler this is a bad request rather than a broken broker: without it the envelope would take the stored-procedure error path below and arrive as a 500, which reaches an operator as an outage and a metering proxy as an unbilled upstream failure. It is a key of the error envelope only, never of a successful echo. The difference from the clamps above is deliberate, because these two govern deletion: a silently clamped 0 would floor the hold at a minute ago and hand the lake a hole, and a silently clamped 999999999 would park retention for thirty-one years. The character set is narrow for a second reason: the name is the middle segment of the key/value key the retention cycle probes (s3:<sink>:<queue>:committed), built by concatenation, so a name containing : would compose the same key as a different (sink, queue) pair.

With a hold set, the two segment-deleting cutoffs are floored at

GREATEST(committed window end - 60 s, now() - retentionSinkHoldMaxSeconds)

Both halves earn their place. The minute of slack covers the sub-second skew of a watermark-derived bound, paid in kept bytes and never in lost ones. The cap is what makes the option safe to ship: without it a stopped sink is unbounded retention, and it also means the hold works from the moment it is set, before the sink has committed anything, because a queue with no pointer yet keeps everything younger than the cap.

Two limits worth knowing. The hold floors the cutoffs that delete segments, and deliberately not the dedup-hash purge or the maxWaitTimeSeconds eviction: those delete by different rules, and a copy in a lake says nothing about when a dedup hash may go. And the pointer read degrades rather than fails: if it cannot be read, every held queue falls back to its cap-only floor, because that value is tenant-writable and a malformed one must never stop deletion cluster-wide. The operator’s view of all this is on the S3 sink page.

Response

200 with the stored-procedure JSON passed through verbatim:

{
  "configured": true,
  "queueId": "0198...",
  "partitionId": null,
  "queue": "orders",
  "namespace": "billing",
  "task": "invoice",
  "storage": "segments",
  "options": { "leaseTime": 120, "retryLimit": 5, "dedupWindowSeconds": 600 }
}

storage is a hard-coded literal. There is no engine-selector column behind it (there is only one engine); the key is kept so the response shape does not change.

partitionId is always null. Configuring a queue does not create a partition: partitions are created by the first push that names one. The key used to carry the id of a Default partition the engine never used, and it was removed on 2026-07-30 along with the rows-engine table that held it. The key itself is kept so the response shape does not change.

The options object echoes the effective row, not the request: all 19 option keys, built from the values the upsert has just written. A caller that sent three options gets all nineteen back and needs no second read to render the queue’s whole configuration, which is exactly what makes a merging edit safe to build a form on. namespace and task stay top level, where they have always been. There is no replace key anywhere in the response: the flag is a directive of the request, read once and never stored.

queen.configure_queue_v1 owns every configuration write: one row, queen.queues, carries the whole configuration including lease_time and dedup_window_seconds. After the stored procedure returns, the handler drops this queue’s cached lease time and its cached encryption flag, locally and on mesh peers via a QUEUE_CONFIG_SET frame, so both a leaseTime and an encryptionEnabled change are visible on the next pop and the next push. If the stored procedure returns a body carrying an error key, the handler short-circuits before those invalidations.

Four failure shapes, and the status is the difference between them. A malformed body, a missing or non-string queue, and an unrecognised mode are 400, and nothing is written. The two sink-hold option refusals are 400 as well, marked with invalid, described above. Any other error the stored procedure reports is passed back verbatim at 500, or at 404 when its message contains “not found”. And a failure of the call itself rather than a verdict inside it, an exhausted pool or a database error, is 500 with {"error":"configure failed: ..."}.

GET /api/v1/resources/queues

Lists the tenant’s queues.

{
  "queues": [
    {
      "id": "0198...",
      "name": "orders",
      "namespace": "billing",
      "task": "invoice",
      "createdAt": "2026-07-30T09:12:44.011Z",
      "partitions": 8,
      "retainedBytes": 194883,
      "segments": { "segments": 41, "messages": 12904 },
      "messages": { "total": 12904, "pending": 310, "processing": 24 }
    }
  ]
}

Three different clocks feed one object, and the difference matters when you read these numbers:

• partitions and messages.total are computed live per request from queen.log_partitions watermark arithmetic (log_queue_stats_all_v1), at a cost proportional to the tenant’s partition count, never its segment count. messages.total is the count of retained frames, not the backlog.
• segments is read from queen.stats.segment_count, which the retained-bytes lane fills out of the one log_segments scan it already makes. It is as stale as that lane’s cadence and reads 0 on a queue the lane has not passed over yet; the per-queue detail route below keeps the exact live count.
• messages.pending and messages.processing come from queen.stats, which a background reconciler refreshes every STATS_INTERVAL_MS (default 10000 ms) on whichever replica claims the stats_refresh row in queen.maintenance_leases. They are the same watermark values the overview sums, so the list agrees with the rest of the dashboard rather than with a fresher count.

?stats=cached skips the live enrichment entirely and serves the cached queen.stats view as it stands, so nothing is computed per call. It exists for pollers whose fields are all in the cached view anyway, such as the proxy’s reconciler, which sends it unconditionally; a broker older than the parameter ignores it and enriches as before.

GET /api/v1/resources/queues/:queue

One queue in detail: identity, its whole configuration, per-partition stats, and totals.

{
  "id": "0198...",
  "name": "orders",
  "namespace": "billing",
  "task": "invoice",
  "createdAt": "2026-07-30T09:12:44.011Z",
  "options": {
    "namespace": "billing",
    "task": "invoice",
    "priority": 0,
    "leaseTime": 120,
    "retryLimit": 5,
    "retryDelay": 1000,
    "maxSize": 0,
    "ttl": 3600,
    "deadLetterQueue": true,
    "dlqAfterMaxRetries": true,
    "delayedProcessing": 0,
    "windowBuffer": 0,
    "retentionSeconds": 0,
    "completedRetentionSeconds": 0,
    "retentionEnabled": false,
    "encryptionEnabled": false,
    "maxWaitTimeSeconds": 0,
    "minPopWaitTime": 0,
    "dedupWindowSeconds": 600,
    "retentionSinkHold": "",
    "retentionSinkHoldMaxSeconds": 604800
  },
  "partitions": [
    {
      "id": "0198...",
      "name": "Default",
      "createdAt": "2026-07-30T09:12:44.031Z",
      "stats": {
        "total": 1600, "pending": 40, "processing": 8,
        "completed": 1552, "failed": null, "deadLetter": 0
      },
      "oldestMessage": "2026-07-30T09:12:45.100Z",
      "newestMessage": "2026-07-30T10:44:02.900Z"
    }
  ],
  "totals": { "total": 12904, "pending": 310, "processing": 24, "completed": 12570, "failed": null, "deadLetter": 0 },
  "retainedBytes": 194883,
  "segments": { "segments": 41, "messages": 12904 }
}

options carries all 21 keys /configure accepts, spelled exactly as configure_queue_v1 parses and echoes them, so a GET, an edit and a POST round-trip without a mapping table. namespace and task appear inside it as well as at the top level, where readers have always found them. This is the read an editor prefills from, and the reason it is the whole set rather than a selection is the merge rule above: a form that shows some options and silently keeps the rest cannot tell you what the queue is. The narrower six-key config block on GET /api/v1/status/queues/:name is unchanged and is a different object.

Notes that follow from the log engine: pending is measured against the worst (lowest) committed cursor across the queue’s consumer groups, so a single lagging group holds the number up; failed is always null because there is no per-message failure state; and retainedBytes is a queen.stats value, refreshed by the retained-bytes lane (RETAINED_BYTES_INTERVAL_MS, default 10 minutes, on whichever replica claims the retained_bytes row in queen.maintenance_leases), so it can lag by up to one lane period and reads 0 on a queue created since the lane’s last pass. A queue with no queen.queues row answers 404 with {"error":"Queue not found"}.

GET /api/v1/resources/queues/:queue/depth

Access level read-only. The minimal backlog read for relays and schedulers: one call to queen.log_queue_depth_v1 (011_log_stats), which computes per-partition pending with the watermark arithmetic only. No log_segments scan, no timestamps, no DLQ join: against the console-grade detail route above this is one index-only read, computed live per request (never stale by the stats cadence).

{
  "queue": "orders",
  "group": null,
  "pending": 310,
  "processing": 24,
  "ready": 286,
  "partitionsPending": 4,
  "partitionsReady": 3,
  "conflation": false,
  "effectivePending": 310,
  "effectiveReady": 286,
  "partitions": [
    { "partition": "Default", "pending": 310, "processing": 24, "ready": 286 }
  ]
}

Without group, pending uses the same worst-cursor precedence the detail route and the dashboard publish, so the numbers agree. With ?group=<name> each partition reports that group’s own backlog against its committed cursor; a group with no cursor on a partition owes the partition’s whole retained range. A queue with no queen.queues row answers 404 with {"error":"Queue not found"}.

partitionsPending counts the partitions with anything pending. It is useful for every group: queenctl queue depth used to compute it client-side and call it partitionsNonEmpty.

processing is the portion of pending work covered by live leases, clamped to pending; ready is pending - processing, and partitionsReady counts partitions whose ready value is non-zero. These fields are computed from the same consumer rows as pending, not from a message or segment scan.

conflation and effectivePending describe last-value delivery. conflation is the group’s stored delivery policy (false without ?group=, which addresses no group). effectivePending is partitionsPending for a conflating group and pending for every other; effectiveReady similarly selects partitionsReady or ready. The distinction is the one that decides whether you are looking at an incident:

For a conflating group, pending is log depth, the positions still to retire, and effectivePending is work depth, the handler invocations that remain. A conflating queue at pending: 4000000, effectivePending: 12 is healthy. The same two numbers on a non-conflating group are a page.

DELETE /api/v1/resources/queues/:queue

Access level admin. One call to queen.delete_queue_v1 (013_analytics) owns the whole delete. It removes, in order, the queue’s queen.log_txns and queen.log_dlq rows (neither carries a foreign key by design, because the purge path must never pay FK-trigger cost, so they are deleted explicitly while the partitions still resolve), then the single queen.queues row. That one delete cascades to everything else: queen.log_partitions and, under it, log_segments and log_consumers, plus the consumer watermarks, queue-scoped consumer-group metadata, lag metrics and queen.stats rows. Before the queue-identity merge this was two non-atomic steps (a stored procedure plus a broker-side teardown of the engine tables); it is now one transaction.

The response is 200 with a JSON body, never 204: the SDKs read res.deleted === true and a bodyless response would make them see null.

{ "deleted": true, "queue": "orders", "existed": true }

Deleting a queue that does not exist is also 200, with the body made self-consistent rather than misleading:

{
  "deleted": false,
  "queue": "nope",
  "existed": false,
  "message": "Queue not found, nothing was deleted"
}

The status stays 200 because the SDKs use delete-before-create as a cleanup idiom; a 404 would turn a no-op into a thrown error for them. Check deleted, not the status code.

Message traces are not deleted. queen.message_traces had its foreign keys dropped so it could hold traces for both engines, so a queue delete leaves its traces behind. See Traces.

Aggregate listings

Three sibling reads, all read-only, all passing their stored procedure’s JSON through unchanged.

statsAge is the age in seconds of the numbers you are reading, or -1 when nothing has been reconciled yet. Treat a large statsAge as “the reconciler is not running”, not as “the queue is idle”.

POST /api/v1/partitions/changed

The only route that lists partition names. Everything else answers a per-queue count, which is enough for a queue whose partitions are numbered 0..N-1 and useless for one partitioned by entity, where the names are order ids or customer ids and nothing enumerates them. It is the discovery half of what POST /api/v1/fetch is the read half of, and it answers lastOffset and logStart alongside each name so a caller can start fetching without a second round trip.

Like the fetch, it is not a pop: no lease, no cursor, no claim, nothing written anywhere. It reads two tables, queen.queues and queen.log_partitions, plus pg_stat_activity for the watermark below. Two callers asking the same question get the same answer and neither disturbs a consumer group. Its access level is read-only, and it takes no admission permit: this is one indexed scan of a small table, so the connection pool is the backpressure and the statement timeout is the ceiling.

The one client that exists for it today is the S3 sink.

Request

{ "entries": [ { "queue": "orders",
                 "since": "2026-09-04T10:00:00.000000Z",
                 "after": "t|1788515700123456|cust-0419",
                 "limit": 1000 } ] }

The answer

{ "safeTime": "2026-09-04T10:04:57.412331Z",
  "safeTimeDegraded": false,
  "entries": [ { "queue": "orders",
                 "partitions": [ { "name": "cust-0420",
                                   "lastOffset": 1811,
                                   "logStart": 1400,
                                   "lastWriteAt": "2026-09-04T10:04:12.000000Z" } ],
                 "next": null } ] }

lastOffset is the allocator’s last offset, so the high watermark is lastOffset + 1. logStart is the oldest offset retention still holds. Timestamps are rendered UTC at microsecond precision.

next is non-null only when the page filled, which is the rule to page on: a short page is the end of the sweep. The cost is one extra empty round trip when the partition count is an exact multiple of the limit; a look-ahead row would buy nothing a caller can act on.

The cursor is opaque and mode-tagged. Do not parse it. Its shape is the broker’s, and it carries which kind of sweep produced it, so a cursor from an enumeration sent with a since (or the reverse) is answered BAD_CURSOR for that entry rather than paging silently against the wrong column. Restarting quietly would loop a paging client for ever on its own first page.

Two per-entry errors replace partitions and next:

Paging is sound because lastWriteAt only moves up. The push path quantizes it to at most one real change per second per partition and never writes it backwards, so a row bumped between two pages of one sweep moves forward past the cursor and is seen a second time on a later page. Seen twice is free, since the caller re-reads bounds it already had; missed would be a silent hole in whatever the caller is mirroring. That asymmetry is the whole argument.

safeTime

Every answer carries a watermark, and it is the reason the route exists in this shape rather than as a listing:

safeTime is an instant such that no segment with an earlier created_at can still become visible. A segment’s created_at is stamped while the partition allocator’s row lock is held, so it is at or after the inserting transaction’s xact_start; the oldest xact_start among in-transaction sessions is therefore a floor under every created_at still to arrive, and anything strictly below it is settled. A guard of five seconds is subtracted, covering the skew between one backend’s snapshot of pg_stat_activity and another’s.

That is what lets a reader treat a time range as a closed set: re-reading [A, B) for B at or below safeTime yields exactly the same rows, which is what makes a reader’s retry idempotent without naming offsets.

Three consequences worth knowing:

• An idle session pins nothing, because its xact_start is null.
• A long read-only transaction, a stats refresh for instance, pins safeTime and holds the watermark back for its own duration. That is latency, never incorrectness.
• HA needs nothing. pg_stat_activity is cluster-wide, so a second broker’s sessions pin correctly with no mesh involvement.

safeTime is computed once per call, before any partition row is read, and it is answered even for an empty entries array: a caller whose queues are all idle still needs the watermark, and an empty batch is how it asks for one.

The order of that first read is load-bearing, and so is VOLATILE

queen.log_partitions_changed_v1 writes nothing, so STABLE is what its body looks like it deserves. STABLE is wrong here, and the reason has nothing to do with writes. A STABLE function’s statements all run under the snapshot the calling query took, but pg_stat_activity is not read through a snapshot at all: it reports live backend state. The two therefore observe the database at different instants, and a transaction can slip through the gap: the caller takes its snapshot while a long write transaction T is in flight, T commits, the body reads pg_stat_activity and finds nothing to hold the watermark back, and the partition scan still runs under the older snapshot that cannot see T’s row. T is then in neither the enumeration nor the watermark, and a sink can close a window above records it was never told about. Silent loss, of exactly the rows the protocol exists not to lose. It was found by the engine’s randomized test, not by reading the code.

VOLATILE closes it, because a VOLATILE function takes a fresh snapshot for each statement of its body instead of inheriting the caller’s. Combined with reading pg_stat_activity first, before the queue is resolved and before any partition is scanned, both cases are covered: a partition whose transaction committed before the scan’s snapshot is enumerated and read normally, and one that had not committed by then either was already in flight at the activity read (so its xact_start is inside the minimum) or had not started (so its xact_start is later than the read). Either way its records are at or above safeTime and belong to a later window. Reading the activity after the scan, or putting any table read before it, re-opens the gap; the five second guard is belt and not the argument.

One precondition holds it up, and the broker meets it: fresh-snapshot-per-statement is READ COMMITTED behaviour. A caller that wrapped this call in an explicit REPEATABLE READ or SERIALIZABLE transaction would pin one snapshot for the whole transaction and reinstate the hole, VOLATILE or not. The broker issues it as a single autocommit statement at the default isolation.

Access levels

Access levels are a role set, not a ladder. A WriteOnly token passes /api/v1/push and is rejected on every route on this page, including the listings.

Every route here resolves a tenant from the request, so with QUEEN_TENANCY_HEADER on they are scoped to the caller’s tenant. The full route table, generated from the router and the authorization function, is in the reference index.