Fan-out with consumer groups

Fan-out in Queen is not a routing feature. It is a consequence of where consumption state lives: the messages are one immutable log, and a consumer group is a cursor over it. Two groups reading the same queue is two integers advancing independently over one copy of the data.

Two groups over one queue

Name a group on the consumer. That is the whole declaration: there is no create-group call, no binding, no exchange.

// Illustrative, not extracted from a test.
// Process A
await client
  .queue('orders')
  .group('billing')
  .batch(50)
  .consume(async messages => { await bill(messages) })

// Process B, independently
await client
  .queue('orders')
  .group('analytics')
  .batch(500)
  .consume(async messages => { await index(messages) })

Both groups receive every message pushed to orders. Their batch sizes, their failure handling and their progress are unrelated: billing stalling does not slow analytics down, and analytics restarting does not replay anything for billing.

Why this costs one write, not N

The write path and the read path are different tables, and only one of them is per-group.

A push packs its messages into frames, length-prefixes them, zstd-compresses the batch and stores it as a single row in queen.log_segments, covering the inclusive offset range [base_offset, end_offset]. That row is written once, before any group exists and regardless of how many will later appear. A message’s identity is a single monotone BIGINT offset inside its partition.

Consumption state is one row in queen.log_consumers, keyed (partition_id, consumer_group). Its load-bearing column is committed, the last acknowledged offset. “Has this group consumed message N” is the comparison committed >= N. There is no per-message, per-group row anywhere in the schema.

So the cost of adding a group is:

The honest counterpart: reads do scale with groups. Each group’s pop fetches and decompresses the segments it needs, so N groups reading the same backlog means N decompressions of the same bytes and N sets of pop transactions. Fan-out is free on the write side and linear on the read side.

Where a group starts

The cursor row does not exist until the group first pops a partition. What it is seeded with is decided then, and never revisited: an existing cursor is never re-seeded, so changing the subscription options later has no effect on a group that has already run.

The choice is durable: the first pop of a (queue, group) pair writes a queen.consumer_groups_metadata row recording the mode and timestamp, and a partition created after that registration seeds from the stored timestamp rather than from its own end, so a new partition does not silently skip its backlog for an existing group.

To move an existing group, use a seek instead. See Replay history.

Watching the groups diverge

GET /api/v1/consumer-groups

lists every group with its per-queue lag. For one group:

GET /api/v1/consumer-groups/billing

And to find only the groups that have fallen behind (the threshold defaults to 3600 seconds):

GET /api/v1/consumer-groups/lagging?minLagSeconds=300

A single message read (GET /api/v1/messages/{partitionId}/{transactionId}) reports per-group progress directly, as a consumerGroups array of name, consumed and leaseExpiresAt. Its summary status field is completed only when every named group’s cursor has passed the message, which is exactly the fan-out question, answered per message.

Retention is the coupling you do have

Groups are independent in delivery. They are not independent in data lifetime, and this is the one place a slow group changes another group’s outcome.

Retention is opt-in (retentionEnabled plus a positive window) and deletes whole segments. Two rules run:

• retentionSeconds deletes by age, regardless of consumption. A segment older than the window goes even if a group never read it. A group that is down for longer than the window loses data, permanently, and with no error at the moment it happens.
• completedRetentionSeconds deletes only consumed data. Its time boundary is capped at MIN(committed) + 1 across the partition’s cursor rows: the slowest group’s next wanted offset. So the slowest group holds the floor for everyone, and a partition with no cursor rows at all has nothing consumed and keeps everything.

Two more retention facts worth carrying: retention never purges queen.log_dlq, and a partition is deleted only once it is empty and has been idle for PARTITION_CLEANUP_DAYS (30), which drops every group’s cursor for that partition along with it. Nothing is left to re-read, so the cost is the total_consumed counter, not a replay.

Removing a group

DELETE /api/v1/consumer-groups/billing

drops the group’s cursor rows for every partition, plus its empty-scan watermarks. Clearing the watermark is what allows a group of the same name to re-consume from the start afterwards. A stale watermark would otherwise fence off every partition. There is also a per-queue variant, DELETE /api/v1/consumer-groups/{group}/queues/{queue}, which scopes the deletion to one queue.

Both answer HTTP 200 with the merged result JSON, including deletedPartitions. Recreating the group is, again, just naming it on a consumer.

What to expect

Next