A queue can read other queues. A consumer group can read only some of a queue. A transaction can forward the message it holds into another queue. In each case the message is written once, at its source; the queue that reads it holds a link, about 20 bytes, written in the same commit as the message itself.
Three rules are all there is to learn:
- A group or a queue can read only what matches a
where. - The source keeps the messages; the queues that read it consume them.
- It is the same message, with the same
idandtransactionId, everywhere it shows.
The where language
One language, in every place that filters: filtered groups, reading queues and the tenant feed.
data.kind in ('order.paid', 'order.refunded') and data.total > 100 and headers.type prefix 'order.'| Part | What it takes |
|---|---|
| Fields | queue, partition, transactionId, data or data.a.b (a path into the JSON payload, up to 16 deep), headers.<name> |
| Operators | = != < <= > >= in exists prefix !prefix |
| Combining | and only; in (...) for alternatives |
| Values | 'text', numbers, true, false, null; in takes up to 64 values |
| Size | up to 16 conditions |
The same conditions can be sent as a list, [{"field":"data.kind","op":"=","value":"order.paid"}], which is what the SDKs’ .where(field, op, value) builds. A field the message does not have holds for no operator, != included, as in SQL.
Filtered consumer groups
await queen.queue('order-events')
.group('billing')
.where('data.kind', 'in', ['order.paid', 'order.refunded'])
.consume(handler)A group’s where is set on its first pop, like its start (subscriptionMode), and fixed from then on. The group then gets only the messages it takes; its lag counts only those, and a waiting pop wakes only for one of them. Leases, acks and dead letters work as for any group; retries, the dead-letter queue and priority are the queue’s.
| Case | Answer |
|---|---|
Another worker of the group sends a different where |
refused, where_mismatch, with the group’s current where |
A where with no group |
refused: name a group |
A where on a group that already reads the queue without one |
refused, where_mismatch: a filter starts as a new group |
subscriptionMode('all') or subscriptionFrom |
the group starts with every match the queue still keeps |
Change the where |
POST /api/v1/consumer-groups/:group/subscription with {"queue":"order-events","where":"..."}; it applies from the next push |
| Delete the group | as any group: DELETE /api/v1/consumer-groups/:group/queues/:queue |
| Every pop’s answer | carries "filtered": true, an empty one too (a 200 with that body, not a 204). An older broker ignores where and would hand the group the whole queue, so the SDKs stop a consumer whose answer lacks it. |
| The group listings | GET /api/v1/consumer-groups and the lag listings show the group under its queue, with "filtered": true and its where |
| A pop by namespace or task | reads each queue once: a filtered group’s messages are not handed out a second time |
Reading queues
A reading queue is a queue whose messages come from other queues: give it from and, if you want, where and partitionBy.
await queen.queue('orders-it').config({
from: 'orders',
where: "data.country = 'IT'",
partitionBy: 'same',
leaseTime: 30,
retryLimit: 5,
retryDelay: 1000,
deadLetterQueue: true,
dlqAfterMaxRetries: true,
priority: 0,
delayedProcessing: 0,
windowBuffer: 0,
}).create()| Option | Default | What it does |
|---|---|---|
from |
(none) | The source: a queue name, or a list of names and {"queue": "...", "where": "..."} objects. null stops the reading; what the queue holds stays readable. |
where |
every message | The condition a message of a source named by name only must meet. |
partitionBy |
same |
same: the source’s partition names, in the source’s order. data.<path> or headers.<name>: the field’s value names the partition (a message without it is not routed). |
start |
now |
now: the messages pushed from now on. all: every match the sources still keep, then the new ones, in order. |
A reading queue has its own leases, retries, dead-letter queue, priority, delays and consumer groups, and its own token can read it without reading its sources. It does not have keep settings: retentionEnabled, retentionSeconds, completedRetentionSeconds, maxWaitTimeSeconds, dedupWindowSeconds, encryptionEnabled, maxSize and the sink hold are refused on it (reading_queue), because the source decides how long a message is kept.
partitionBy), several sources, or a token of its own.| Case | Answer |
|---|---|
| A push into a reading queue | refused, reading_queue: push to its source |
An existing ordinary queue given from |
it becomes the reading queue if no message was ever pushed to it (a consumer that popped it before it was configured created it); refused once it holds messages |
| A reading queue that reads itself, at any depth | refused, route_loop |
| A source that does not exist, or is a filtered group’s | refused, source_not_found or bad_source |
| More than 16 sources, or a 65th reader of one source | refused, too_many_sources or too_many_readers (QUEEN_MAX_SOURCES_PER_QUEUE, QUEEN_MAX_READERS_PER_QUEUE) |
A where change |
applies from the next push; to rebuild, delete and create |
| Deleting a source that reading queues read | refused, has_readers (409), unless ?force=true: their links then read as gone |
A message that two sources’ partitions both route into one partition of a reading queue, under the same transactionId, is kept once: a reading queue deduplicates by transactionId within its partitions, as every queue does.
A queue’s page, GET /api/v1/resources/queues/:queue, shows its part in routing under route: what it reads (from, each with its where), who reads it (readers, each with its where and the age of its oldest unconsumed message, oldestUnconsumedAt), the queues that hold links to it (holders, with holdsRetention), and while it catches up, how far (catchUp: messages read of the total). The queue list shows the same route without the ages.
What a reading queue keeps
The source keeps the message. For the source’s retention, every reading queue and filtered group counts as one more consumer group of it:
| Source setting | What a reader gets |
|---|---|
retentionSeconds, maxWaitTimeSeconds |
the same limit: past it, a link reads as gone, as a message is gone for a group that is behind |
completedRetentionSeconds |
the source waits for the readers too: it keeps a message until they have consumed their link |
| retention off | everything is kept |
How far back a source’s readers hold its completed retention is the gauge queen_holder_floor_age_seconds{queue}: the age of the oldest message they still keep unconsumed. A link whose message is gone is skipped when it is read (the queen_link_expired_at_source_total counter). A dead letter is a copy: filed from a reading queue, it holds the message itself and its headers, because a dead letter outlives its source.
Lag, backlog and unread
A filtered group is a group of its queue, so its figures are its queue’s, counted from only the messages its where took.
| Figure | Where it shows | What it counts |
|---|---|---|
| A filtered group’s lag | GET /api/v1/consumer-groups, GET /api/v1/consumer-groups/lagging, and queen_consumer_group_lag_messages, queen_consumer_group_lag_seconds and queen_consumer_group_partitions_behind under its queue’s name |
the messages its where took that it has not consumed, on the partitions it took them from |
| A filtered group’s depth | GET /api/v1/resources/queues/:queue/depth?group=:group |
the same, partition by partition, with "filtered": true and its where |
A queue’s pending |
the queue list, the overview, GET /api/v1/resources/queues/:queue, /depth without a group, the backlog chart |
partition by partition, the most that one of its groups, its own or filtered, has not consumed, plus every unread message; a message counts once, however many groups took it |
A queue’s unread |
messages.unread on the queue list and the overview, totals.unread and each partition’s unread on the queue’s page, unread on /depth |
the messages on partitions that no group of the queue reads and no reader’s where can take |
| A reading queue’s backlog | the reading queue’s own figures | what its groups have not consumed; its source does not count it again |
An example: a queue of 100 partitions, read by three filtered groups that take 30 partitions each with partition in (...). Each group’s lag counts its 30 partitions only, never the other 70. The 10 partitions that no group takes are unread, and the console marks the queue “No reader” for them.
Which partitions a reader can take is decided by its conditions on queue and partition. A where on the payload or the headers may take a message of any partition, so it leaves no partition unread, and a message it does not take waits for nobody: it counts as completed. A queue that nothing reads at all is unread in full, as its pending always said.
link: forward a message without its payload
A transaction can put the message it holds into another queue, with the ack, the KV writes and the timers, all or nothing:
await queen.transaction()
.ack(message)
.queue('manual-review').partition(message.partition)
.link([message], { headers: { 'review-reason': 'amount' } })
.commit()On the wire it is one more operation of POST /api/v1/transaction:
{"type": "link", "queue": "manual-review", "partition": "order-9137",
"from": {"partitionId": "123", "offset": 42, "transactionId": "order-9137-paid", "consumerGroup": "billing"},
"headers": {"review-reason": "amount"}}from names the message as a pop returned it, with the group it was popped with (consumerGroup, queue mode when absent: a scoped credential must be allowed to consume the source with it); a feed entry names it by queue, partition, offset and transactionId instead. The target gets the same message, the same id and transaction id, with the source’s headers and then the link’s own. A second link of the same message (the same transactionId) into the same partition is a no-op: the transaction commits and the link’s result says "duplicate": true, so a forward is safe to retry and a message forwarded twice is held once. A link to a message that is gone rolls the transaction back (source_gone), and so does one whose transaction id is not the one at that offset (source_mismatch). A reading queue takes no link but its sources’ (reading_queue).
A queue that holds links this way is a holder of the source, which keeps its completed retention for as long as the holder keeps anything: set retention on a queue you forward into.
link, a hop costs about 20 bytes, whatever the payload weighs.In streams and the feed
A stream’s .to(queue) writes a link when its result is its input message unchanged: nothing but where, filter, keyBy or routing before it, on a queue source and on the feed. The target gets the same message, with the same id and transaction id. .to(queue, { link: false }) writes copies, and a result a step changed is always a copy. A second link of one message into the same partition is a no-op, so a retried commit is safe.
On the wire, a stream cycle’s push item names its input message instead of a payload: {"queue": "...", "partition": "...", "link": {"offset": 42, "transactionId": "..."}} (the cycle’s own partition unless it names partitionId). The SDKs send the payload beside it, which a broker that reads links ignores and an older one stores.
The tenant feed shows every queue’s messages, a reading queue’s and a link target’s read through their links: the same id, transaction id and payload as at the source, with the link’s headers over the source’s, and "link": {"queue", "partition", "offset"} saying where the message is at its source, so a reader of every queue can tell a message it already has. A filtered group has no queue of its own there.
What it costs
| Copy (a router) | Link | |
|---|---|---|
| Bytes through the log, per message and reader | the whole message | about 20 bytes, in the push’s own commit |
| When the reader has it | after the router reads and commits | in the push’s commit |
| Planning | the router’s commits | in the push’s own plan, with the readers’ partitions it writes (pushes to other partitions plan at the same time) |
| Reading | the copy | the source’s record, read and decoded once per record |
A where on the payload is evaluated once, by the leader, when the push is planned (an encrypted payload is decrypted for it); the entry holds the answer, so followers and replays never evaluate it.
Limits
| Limit | Default | Setting |
|---|---|---|
Conditions in one where |
16 | QUEEN_WHERE_MAX_CONDITIONS |
Values in one in |
64 | QUEEN_WHERE_MAX_IN |
| Readers of one source (reading queues and filtered groups) | 64 | QUEEN_MAX_READERS_PER_QUEUE |
| Sources of one reading queue | 16 | QUEEN_MAX_SOURCES_PER_QUEUE |
| Readers of readers (a chain) | 8 deep |
Every reader costs an evaluation of its where on each push to its source, which is why the readers of one source are limited by default; raise it knowing that.
All of this needs every node of the cluster on a broker that reads catalogue version 8; before that, a where, from, headers or link is refused with cluster_version.