Messages and the dead-letter queue

These are inspection routes, not a consumption path. Live payloads live inside immutable, zstd-compressed segments, so reading one message means resolving an address to an offset, fetching the covering segment and decoding one frame. Plan for them to be used by an operator or a dashboard, not by a hot loop.

Every per-message route is addressed by two path segments, :partitionId/:transactionId. The partitionId is the partition UUID echoed by push, pop and ack responses.

GET /api/v1/messages

Browse messages with filters. All filters are optional.

{
  "messages": [
    {
      "id": "0198...",
      "transactionId": "order-4471",
      "txnHash": "9f2c...",
      "partitionId": "0198...",
      "queuePath": "orders/eu-1",
      "queue": "orders",
      "partition": "eu-1",
      "namespace": "billing",
      "task": "invoice",
      "status": "pending",
      "queueStatus": "pending",
      "busStatus": { "consumedBy": 0, "totalGroups": 2 },
      "traceId": null,
      "producerSub": null,
      "payloadAvailable": true,
      "segment": { "seq": 8192, "frameIdx": 44 },
      "createdAt": "2026-07-30T10:41:02.010Z",
      "leaseExpiresAt": null,
      "data": { "amount": 41 },
      "payload": { "amount": 41 }
    }
  ],
  "mode": { "hasQueueMode": true, "busGroupsCount": 2, "type": "hybrid" },
  "total": 1
}

How that body is produced, and what it implies:

• Per-message metadata comes from the hash sidecar. SQL cannot see inside a segment blob, so the listing is built from queen.log_txns exploded to frames. segment.seq carries the covering segment’s base offset and segment.frameIdx carries offset - base_offset. Treat both as opaque.
• The broker fills in the payload. Rows come back from SQL with payloadAvailable: false, id: null and transactionId: null; the handler then fetches each referenced segment once (cached per (partitionId, seq) so a page inside one segment decodes it exactly once), decodes the frame, decrypts it if a key is configured, and fills data, payload, id, transactionId, traceId, producerSub and isEncrypted, flipping payloadAvailable to true. If the segment has since been deleted by retention, the entry stays at payloadAvailable: false with those fields null.
• Entries expire from the listing before the messages do. A frame is listed only while its queen.log_txns row survives, which is GREATEST(dedupWindowSeconds, completedRetentionSeconds, 900) seconds after the push. Older messages are still stored, still consumable, and no longer browsable here.
• total is the size of the page, not of the result set. The handler sets it to the length of the returned array. There is no total count; page with offset until you get a short page.
• status is derived per frame: dead_letter when a queen.log_dlq row exists at that offset, else completed when the queue-mode cursor has passed it, else processing when a lease is live, else pending.
• mode.type is queue, bus, hybrid or none, detected from which consumer groups have cursors on the partition.

GET /api/v1/messages/:partitionId/:transactionId

Resolves one message and returns it with its management detail.

Resolution order:

1. Hash the transaction id (xxh3-128, big-endian; SQL never hashes) and probe queen.log_txns for the offset.
2. Find the covering segment for that offset, decode frame offset - base_offset.
3. If the queen.log_txns rows have been purged, fall back to a newest-first scan of up to 5000 of the partition’s segment blobs, decoding each until the transaction id matches.

The fallback is why an old message still resolves; it is also why this route can be expensive on a large partition. A message whose covering segment has been deleted by retention is a 404: resolvable position, unrecoverable frame.

{
  "id": "0198...",
  "transactionId": "order-4471",
  "data": { "amount": 41 },
  "payload": { "amount": 41 },
  "traceId": null,
  "producerSub": "svc-billing",
  "createdAt": "2026-07-30T10:41:02.010Z",
  "partitionId": "0198...",
  "partition": "eu-1",
  "isEncrypted": false,
  "queue": "orders",
  "queuePath": "orders/eu-1",
  "namespace": "billing",
  "task": "invoice",
  "status": "processing",
  "errorMessage": null,
  "retryCount": 0,
  "leaseExpiresAt": "2026-07-30T10:46:02.010Z",
  "consumerGroups": [
    { "name": "invoicer", "group": "invoicer", "consumed": false, "leaseExpiresAt": null }
  ],
  "mode": { "hasQueueMode": true, "busGroupsCount": 1, "type": "bus" }
}

Details worth knowing:

• data and payload are the same value, decoded from the frame. The broker decrypts by sniffing the envelope shape whenever a key is configured, regardless of the queue’s stored encryptionEnabled flag, so messages written before the flag was set still decode. isEncrypted reports the flag stored on the frame.
• retryCount is 0 for any live message. The log engine counts retries per (partition, group), not per message; the only per-message value that exists is the counter snapshotted onto the DLQ row at dead-letter time, which is what this field reports for a dead-lettered address.
• leaseExpiresAt is the queue-mode group’s lease. Per-group state is in consumerGroups, where consumed is committed >= offset for that group.
• namespace and task fall back to the dot-split of the queue name when the queue row carries empty values.
• The response also carries a queueConfig object with configuration read off the queue row, including leaseTime and retryLimit.
• payload is null for an empty frame; a frame that fails to decode is a 500 (frame decode failed).

DELETE /api/v1/messages/:partitionId/:transactionId

Access level read-write. This deletes a dead-letter row, nothing else. Live payloads are in immutable segments and cannot be deleted individually.

{
  "success": true,
  "partitionId": "0198...",
  "transactionId": "order-4471",
  "message": "Message deleted successfully"
}

Nothing matched is a 404, not a 200 carrying success: false. A caller that ignores the body must not read a no-op as a deletion:

{
  "success": false,
  "partitionId": "0198...",
  "transactionId": "order-4471",
  "error": "Message not found",
  "message": "No dead-letter row for this address. Live messages live in immutable segments and cannot be deleted"
}

POST /api/v1/messages/:partitionId/:transactionId/retry

Access level read-write. Replays a dead-lettered message. It works only for dead-lettered addresses; a live message has nothing to replay and returns the same 404 shape as DELETE.

Since 1.6.0 the act underneath is a move, and the difference is the whole section. queen.log_dlq_move_v1 claims the dead-letter row with SELECT ... FOR UPDATE, pushes one packed frame through the ordinary push allocator, and deletes the row, all in one transaction. The broker’s half is what a transaction cannot do: decrypt the snapshot, pack the frame, and announce the landing afterwards so a parked long poll wakes on it. The implementation this replaced minted an id per attempt, read the row without a lock, deleted it in a second statement that could fail on its own, and removed every consumer group’s record while replaying one of them. The dashboard had dropped its replay button for exactly those reasons, and it is back because the primitive can no longer do any of them.

Read it as four guarantees, each one a property of the primitive rather than something the caller arranges:

1. One transaction, one row, one lock. The push and the delete commit together or not at all, so there is no “replayed but still dead-lettered” state to recover from, and two concurrent callers serialise on the row lock: the loser is told the row is gone rather than pushing a second copy.
2. A deterministic transaction id. The replayed frame carries dlq:<dead-letter row id>, derived from the row rather than minted per attempt. In practice a second call never gets that far, because the row went with the first move and the call answers 404; the id is the belt to that transaction’s braces, and it is also what links a replayed message back to the record it came from.
3. Only the addressed record is removed. An address can carry one row per consumer group. This route resolves it to the newest of them (failed_at DESC) and moves that row id, so the other groups’ dead-letter records survive the replay. That is the one place it differs from DELETE /api/v1/messages/:partitionId/:transactionId, which really does delete every group’s record for the address.
4. A foreign partition cannot be replayed. The ownership gate runs before the row is read, so with QUEEN_TENANCY_HEADER enabled a partition belonging to another tenant gets exactly the same 404 as an address that does not exist, and no payload is touched on the way to that answer. The move’s own SQL repeats the tenant predicate under the lock.

Encryption is handled the way it always was, with one change of address: the snapshot is stored verbatim, so on an encryption-enabled queue it is the {encrypted, iv, authTag} envelope. The broker decrypts it and then asks whether the destination queue wants encryption, which is the same queue here and can be a different one on the row-id route below.

{
  "success": true,
  "result": "moved",
  "queue": "orders",
  "partition": "eu-1",
  "consumerGroup": "invoicer",
  "dlqId": "0198...",
  "originalTransactionId": "order-4471",
  "replayedAs": {
    "index": 0,
    "message_id": "0198...",
    "transaction_id": "dlq:0198...",
    "queueName": "orders",
    "status": "queued",
    "offset": 41902
  },
  "dlqRowRemoved": true
}

replayedAs is still a push result, so the five SDK wrappers and queenctl keep parsing it; offset joined it because the move knows where the message landed and a caller that wants to read it back needs it. The four fields beside it are new: result, the verdict; consumerGroup, whose record was moved; dlqId, the row it was; and originalTransactionId, the id the dead-lettered message carried, which is the only link between that id and the dlq: one.

The verdicts

That last row is the one asymmetry between the two replay routes. On the row-id route below, a failed lookup is answered by the same builder as a failed move, so it carries dlqRowRemoved: false like every other refusal; here the arm returns a bare {"error": ...} instead. A client that reads the field rather than testing for it will see undefined, not false, and both mean the same thing.

duplicate is the one that repays reading twice. It means the destination’s dedup window already holds a frame under this replay’s transaction id, so the push wrote nothing. The row is kept anyway, and deliberately: the dedup identity is the transaction id hash alone, and dlq:<row id> is derivable by anyone who can read the dead-letter listing, so deleting on that verdict would turn a producer credential into a way to destroy a dead-letter record without replaying it. replayedAs.offset names where the existing frame is, replayedAs.status is duplicate, and replayedAs.message_id is the all-zero UUID, the same “original unknown” sentinel a duplicate push uses: the copy already in the log carries its own id inside a segment blob this route does not read.

The 503 is the one behaviour a move gives up compared with a push. A maintenance-mode push is answered buffered and replayed from the on-disk spool later; a move cannot be, because the spool carries frames and not the deletion of a row, and a spooled move would be exactly the state this primitive exists to make impossible. So it is refused instead, which costs nothing: the row is still there and the same replay works the moment the switch is off.

Five admin clients wrap this route, spelling it the way their language does: retryMessage in JavaScript and PHP, RetryMessage in Go, retry_message in Rust and Python. Their paths and bodies are unchanged. queenctl dlq retry <partitionId> <transactionId> wraps it too and still requires --yes, now because it appends a message and removes a record rather than because it might do either twice. The Go client keeps sending it with WithoutFailoverRetry(): it is a write, and a blind resend by the transport would hide a verdict the caller has to read.

POST /api/v1/dlq/:id/replay

Access level read-write. The same move, addressed by the dead-letter row id that GET /api/v1/dlq already returns in each row’s id. This is the form the dashboard’s Replay button uses, and the one a redrive wants: a row id names exactly one record, so a message dead-lettered under two consumer groups loses only the record that was replayed, with no “newest” rule to reason about.

The body is optional. Absent, empty, whitespace, or {} replays in place. When it is present it may name a destination:

curl -sS -X POST \
  http://localhost:6632/api/v1/dlq/0198abcd-0000-4000-8000-00000000cafe/replay \
  -H 'Content-Type: application/json' \
  -d '{"queue":"orders.retry","partition":"eu-2"}'

Both fields are optional and independent at the broker: an omitted half keeps the source row’s own value, so {"queue":"orders.retry"} moves the message to another queue and keeps its partition name. Names are trimmed, and a present-but-blank one is a 400, because the allocator provisions exactly the text it is given and a queue named " orders " is one nobody can address. A destination that does not exist yet is created, queue and partition, by the same allocator a first-contact producer push goes through, under the caller’s own tenant.

The 200 body is byte-identical to the retry route’s, and so is the 503. The two 500s the move itself produces match as well; only a failure of the row lookup in front of it differs, which this route reports with dlqRowRemoved: false and the retry route as a bare {"error":"dlq lookup failed: ..."}. The 404 differs only in shape, carrying the verdict explicitly:

{
  "success": false,
  "result": "gone",
  "dlqId": "0198...",
  "error": "Message not found"
}

A message field carries the human sentence beside it, as on every other refusal on this page. Branch on result, not on the sentence.

A malformed :id answers the same 404 rather than a 500. Postgres is the decider: its UUID input accepts more spellings than a rule written in the broker would, so a guard there would answer gone to ids the database would have resolved.

Three caveats belong to appending to a log rather than to this route, and they apply to both forms. The frame lands at the tail of the destination partition, so it is out of order with respect to its own key and the offset the original occupied stays committed: a move is a re-push, not a revival. Its createdAt is stamped at the destination, so the replayed message’s lag clock starts at zero. And it is encrypted for the destination queue, which an override can change: a move out of an encrypted queue into a plaintext one stores plaintext.

No SDK wraps this route yet. The five retry wrappers and queenctl dlq retry use the address-keyed form above, which reads no body.

GET /api/v1/dlq

Lists dead-lettered messages. Payloads are stored as snapshots on the DLQ row, so no segment decode is involved.

{
  "messages": [
    {
      "id": "0198...",
      "transactionId": "order-4471",
      "partitionId": "0198...",
      "queue": "orders",
      "partition": "eu-1",
      "consumerGroup": "invoicer",
      "errorMessage": "Test error",
      "retryCount": 3,
      "data": { "amount": 41 },
      "producerSub": null,
      "createdAt": "2026-07-30T10:52:11.400Z",
      "failedAt": "2026-07-30T10:52:11.400Z"
    }
  ],
  "pagination": { "limit": 100, "offset": 0 },
  "total": 1
}

• Ordered by failedAt descending; limit/offset page the rows inside the query, not the aggregate.
• createdAt equals failedAt. The original enqueue time of a frame is not recoverable from SQL, because blobs are opaque to it.
• producerSub is always null for log-engine rows: it is not tracked on the DLQ snapshot.
• retryCount is the retry budget consumed at dead-letter time, falling back to the queue’s retryLimit for rows written before that snapshot existed.
• data is decrypted on read when a key is configured. A payload that does not sniff as an envelope, or fails to decrypt with the current key, is returned exactly as stored (showing the envelope beats inventing a payload), and a decrypted row is flagged with isEncrypted: true.
• total, again, is the length of this page.

Retention never purges queen.log_dlq. Dead-lettered messages stay until you delete, replay or bulk-purge them, or the queue is deleted.

DELETE /api/v1/dlq

Access level admin. Bulk-purges DLQ snapshots for one exact queue, optionally narrowed to one exact consumer group:

curl -sS -X DELETE \
  'http://localhost:6632/api/v1/dlq?queue=orders&consumerGroup=invoicer'

queue is required and must be non-empty, so an omitted filter can never become a tenant-wide delete. An omitted or empty consumerGroup purges every group for that queue. Both filters and the delete itself remain tenant-scoped in SQL.

{
  "success": true,
  "deleted": 17,
  "queue": "orders",
  "consumerGroup": "invoicer"
}

No matches is a successful, idempotent delete with deleted: 0. A missing queue parameter returns 400 with error: "queue is required"; a database failure returns 500.

Ownership gating

GET, DELETE and POST .../retry are addressed by a raw partition UUID, and the resolver queries carry no tenant. With QUEEN_TENANCY_HEADER enabled the broker therefore checks partition ownership before reading any payload, and a partition belonging to another tenant gets exactly the same 404 as a partition that does not exist. A distinct 403 would confirm that the partition exists under some other tenant, so it is deliberately not used. Confirmed ownership is cached in memory, so repeated reads of your own partition skip the check round-trip. With tenancy off the gate is a no-op and costs no queries.

POST /api/v1/dlq/:id/replay is addressed by a row id rather than a partition, so it is gated in SQL instead: the row lookup joins through log_partitions and queues under the request tenant, and the move repeats the identical predicate under the row lock. Another tenant’s row is gone, the same 404 as a row that never existed, for the same reason.

GET /api/v1/messages and GET /api/v1/dlq are name-addressed: the tenant travels into the stored procedure inside the filter JSON.

Access levels

Broker-direct, those are the levels auth::route_access_level applies. Through the multi-tenant proxy both replay routes are queue admin, which an API key gets from scopes.admin and a session from the Admin role only, and they answer the storage and monthly push blocks the way a push does. See Endpoints and route classes.

Related: consumer groups for the cursors these statuses are computed against, and traces for per-message event history.