The model in one page

Queen is a queue with two halves, and knowing which half you are looking at explains most of its behaviour.

The Kafka half is the log. A queue is split into partitions; each message gets a monotone integer offset inside its partition; a consumer group owns one cursor per partition, so groups read the same data independently and a group can be moved backwards to re-read history.

The RabbitMQ half is the work queue. A consumer claims a batch under a lease with an expiry and then reports each message as completed, failed, retried or dead-lettered; a message that exhausts its retry budget is copied into a real dead-letter table; and consumers that name no group compete for the same messages.

Where the two halves would contradict each other, the log wins: progress is recorded as a cursor, not as per-message state, so “this one message is still in flight” is not a thing the system can represent.

Neither half involves the client in coordination. The broker hands out no partition assignments, there is no rebalancing protocol, and a consumer that dies is recovered by its lease expiring rather than by a group membership change.

flowchart LR
subgraph Q["queue: orders"]
  direction TB
  P1["partition customer-1<br/>offsets 0,1,2,3"]
  P2["partition customer-2<br/>offsets 0,1"]
  P3["partition customer-3<br/>offsets 0,1,2"]
end

PR["producers"] --> Q

Q --> G1
Q --> G2

subgraph G1["consumer group: billing"]
  direction TB
  C1["cursor per partition<br/>1 / 0 / 2"]
end

subgraph G2["consumer group: audit"]
  direction TB
  C2["cursor per partition<br/>3 / 1 / 0"]
end

One queue, one ordered lane per entity, and one independent cursor per lane per group. The two groups above have both read the same messages and are simply at different positions. Nothing is copied for the second group, and neither group can hold the other up.

The vocabulary

There is no per-message row anywhere in that list. Consumption state for a whole partition is one row per group holding one integer, which is why ten thousand partitions cost ten thousand rows rather than ten thousand files or processes.

How a message moves

push  ->  queue "orders"
            partition "cust-42"    offsets:   0   1   2   3   4   5
                                   segments: [ 0 .. 2 ][ 3 .. 5 ]

          group "billing"   committed = 2   ->  next wanted = 3
          group "audit"     committed = -1  ->  next wanted = 0

A push to orders / cust-42 locks that partition’s row, allocates the next offsets, writes one segment row, and commits. created_at is stamped while the lock is held, so it is monotone per partition in commit order. Time-based retention and timestamp subscriptions depend on that.

A pop by group billing claims from committed + 1 forward, up to the batch budget, and takes a lease on what it delivered: batch_end = 5, span (2, 5]. Group audit is unaffected; its cursor is a different row.

An ack for the message at offset 5 with status completed sets committed = 5. Offsets 3 and 4 are now also done, whether or not you acked them, because the cursor is the only record of progress. If the consumer dies instead, the lease expires and the next pop for billing on that partition re-delivers from offset 3.

The five invariants

1. Order is per partition, in commit order. log_partitions.last_offset is bumped under that row’s FOR UPDATE lock, and at most one write per partition is in flight at a time, so offset order equals commit order. There is no global ordering across partitions.
2. One in-flight leased batch per (partition, group). queen.log_consumers has one row per pair, with one batch_end and one worker_id. A second consumer in the same group asking for the same partition gets nothing until the lease is acked or expires, so parallelism inside a group is bounded by the number of partitions, not by the number of workers.
3. An ack is an offset commit. Acking message N completes every earlier unacked message in that partition for that group. You cannot complete offset 7 while leaving 5 outstanding for the same group; when one call carries mixed outcomes the cursor clamps at the lowest signal, and at equal offset dlq beats failed beats retry.
4. Deduplication is exact, per partition, and windowed. A repeated transactionId inside dedupWindowSeconds (default 3600, enabled) is rejected before an offset is allocated, so a duplicate writes nothing and comes back as status: "duplicate". The same key on a different partition is not a duplicate.
5. Delivery is at-least-once. A leased pop plus an explicit ack can redeliver but cannot lose. Setting autoAck=true on a pop commits the cursor inside the pop transaction, which is at-most-once: a consumer that crashes after the response has already been marked done.

What the model does not have

• No global ordering. If your correctness argument needs a total order over more than one partition, Queen will not give it to you.
• No priority queueing of any kind. A partition is drained from its cursor forward.
• No per-message delivery state. There is nothing to clean up, and nothing to inspect that would tell you “message 4 is in flight” independently of its neighbours.
• No client-side assignment or rebalancing. Which is why a restart is cheap, and also why you cannot pin a worker to a partition and rely on the broker to keep it there.
• No bulk DLQ requeue. The dead-letter queue is readable (GET /api/v1/dlq), and a single address can be deleted or replayed: POST /api/v1/messages/:partitionId/:transactionId/retry re-pushes the stored payload snapshot under a fresh transactionId and then drops the DLQ row. Nothing drains a whole DLQ in one call.

Where to go next