How Kafka, RabbitMQ, SQS and pgmq get per-entity ordering, what that costs in structure, and what the four of them did on one machine with matched resources.

One question separates these systems more sharply than any other: how do you get strict ordering per entity, when there are many entities, the set keeps changing, and one slow entity must not delay the others?

Queen’s answer: a partition is a row, created by the first push that names it, and ordering comes from taking that row’s lock before allocating an offset. Consumption state is one cursor per partition and consumer group. Nothing is preallocated, nothing is assigned, nothing rebalances.

flowchart LR
subgraph K["A fixed set of partitions, keys hash-modded onto it"]
  direction LR
  K1["cust-1"] --> KP0["partition 0"]
  K3["cust-3<br/>slow to process"] --> KP0
  K2["cust-2"] --> KP1["partition 1"]
  K4["cust-4"] --> KP1
  KP0 --> KX["cust-1 waits behind cust-3<br/>for sharing a partition with it"]
end
subgraph Q["One lane per entity, created by the push that names it"]
  direction LR
  Q1["cust-1"] --> QL1["lane cust-1"]
  Q3["cust-3<br/>slow to process"] --> QL3["lane cust-3"]
  Q2["cust-2"] --> QL2["lane cust-2"]
  Q4["cust-4"] --> QL4["lane cust-4"]
  QL3 --> QX["cust-3 waits for itself<br/>and for nobody else"]
end

The collision on the left is not a misconfiguration: it is what hashing a changing set of entities onto a fixed set of shards does. Adding partitions moves existing keys and does not remove it.

Kafka

A topic is split into a fixed number of physical partitions, each an append-only log with its own files and replicas. Producers hash the record key onto that fixed set, and a consumer group negotiates assignments through a coordinator.

Entities are hash-modded onto shards. Many entities share a partition, so a consumer stuck on one entity stalls every entity that collides with it.
The partition count is a capacity decision made in advance, and changing it changes the key-to-partition mapping for existing keys.
Kafka replicates itself; Queen does not. Queen’s durability and failover are PostgreSQL’s.
Ecosystem. Kafka has connectors, stream processing and a large operational literature. Queen has six SDKs, a CLI and an HTTP API.

RabbitMQ

Producers publish to exchanges, bindings route into queues, consumers take messages with per-message ack, nack and dead-letter exchanges. Queues are live server-side objects.

Per-entity ordering means a queue per entity, so broker-side objects scale with the entity count. Queen’s per-entity lane is a row.
No replay. An acknowledged message is gone. Queen’s consumption state is a cursor, so a second group can read the same history from the start.
Routing is RabbitMQ’s real strength. Queen has no exchanges, no bindings, no header matching: the producer names the queue and the partition.
Leases, retry budgets and a dead-letter table came from this side of the family, with one difference: Queen’s ack is an offset commit, so acking a message completes every earlier unacked message in that partition for that group.

Amazon SQS

SQS is two things at once here, and they are worth separating. It is a fully managed service you could choose instead, and it is an interface Queen speaks: queen-sqs is a facade that answers the SQS and SNS wire protocols, so an unmodified AWS SDK, and every driver built on one, reaches Queen by changing endpoint_url. The comparison below is about the service. The facade is the protocol reference, it ships in the broker image, and it changes what the first bullet means: the choice stops being SQS or Queen and becomes which of them is behind the SDK your code already uses.

As a service, then. FIFO queues order messages within a message group id and consume a group serially, which is structurally the same idea as a Queen partition.

There is no server, and that is the whole point. Queen is a binary plus a PostgreSQL you operate.
No consumer groups and no replay. Fan-out is a topology you assemble, each subscriber storing its own copy; a deleted message is gone.
The deduplication window is the service’s, not yours. Queen’s is a per-queue setting, keyed on the transactionId you supply, and the facade exposes it as an attribute that widens AWS’s fixed five minutes to anything up to a year.
You cannot query the backlog. Queen’s queues, partitions, cursors, dead letters and lag are tables.

pgmq

The closest comparison, because it makes the same bet: PostgreSQL is the right place to keep messages. A queue is a table, sending inserts a row, reading updates it under a visibility timeout. There is no broker process.

No consumer groups, so one stream to several consumers means a queue per consumer and one stored copy each.
Concurrent readers under visibility timeouts do not preserve per-key order, so per-entity FIFO again means a queue per entity.
Per-message row state versus a cursor. pgmq updates and deletes per message; Queen advances one committed offset per partition and group.
Extension versus schema. pgmq installs as an extension, which some managed offerings refuse. Queen needs CREATE SCHEMA and no extensions.
A broker tier, for better and worse. It buys HTTP clients in any language, long-polling and shared commits; it is also one more process to run.

What the measurements say

All four on one machine, 32 vCPU and 62 GB, with the same resource limits, the same PostgreSQL durability tier for the two that use PostgreSQL, and the same twelve-stage ordered pipeline. Each of these systems would go faster run by someone who operates it daily, and for three of the four that is not us.

Sparse, 1 000 ordered lanes at 2 messages per lane per second. Everyone served the full rate.

	p50	cores	RAM	physical queues
pgmq	55 ms	6.5	2.2 GB	12
Queen	72 ms	7.4	1.8 GB	4
RabbitMQ	93 ms	7.0	2.6 GB	12 000
Kafka	143 ms	2.2	11.6 GB	4

Dense, the same 1 000 lanes at 12 messages per lane per second. Two systems kept up.

	p50	% of rate served	shed
Queen	680 ms (median of 8 runs)	96%	0
Kafka	2 287 ms	89%	0
pgmq	11 863 ms	63%	104 780
RabbitMQ	19 952 ms	27%	363 942

The reversal between those two tables is the whole story. pgmq’s grouped read scans the queue table on every call, so its cost tracks standing rows rather than lane count. Queen amortises over the partition visit, so more messages per lane make it cheaper per message. RabbitMQ needs a queue per key, and 12 000 live queues is where that stops working.

Cardinality. pgmq is flat: 12 000 and 240 000 lanes give identical percentiles. Kafka refuses topic creation somewhere between 2 000 and 5 000 partitions per topic on default settings. Queen has held 1 000 000 ordered partitions, created during the run at a thousand a second.

Sustained throughput. Queen holds 1 000 000 msg/s per side for 24 hours with leases, explicit acks, deduplication and retention all active. pgmq’s ordered-consume path tops out between 20 000 and 40 000 msg/s on the same machine. Full conditions and caveats are in Cross-broker comparison.

What comes with each bet

	Queen	Kafka	RabbitMQ	SQS	pgmq
Ordering per entity at high cardinality	✅	⚠️ hash-modded	⚠️ queue per key	⚠️ per group id	⚠️ queue per key
Lanes created on demand	✅	❌ fixed count	❌ declared	❌	❌
Consumer groups over one stored copy	✅	✅	❌	❌	❌
Replay / seek to offset or timestamp	✅	✅	❌	❌	❌
Deduplication on your own key	✅ per queue	⚠️ producer-session	❌ plugin	⚠️ FIFO, fixed window	❌
Dead-letter queue with a retry budget	✅	❌ app-level	✅	✅ redrive	❌
Per-group lag as data you can query	✅	⚠️ via tooling	❌	❌	❌
Per-tenant isolation, quotas and metering in the box	✅ proxy	⚠️ quotas, ACLs	⚠️ vhosts	⚠️ per AWS account	❌
Ack and push in one transaction	✅	❌	❌	❌	✅ same database
Windowed aggregation with the state in the same commit	✅	⚠️ separate runtime	❌	❌	❌
Disk spool when the store is unreachable	✅	n/a	n/a	n/a	❌
Content-based routing	❌	❌	✅	❌	❌
Broker replicates itself	❌ PostgreSQL’s job	✅	✅	✅ managed	❌
Runs in-process as a library	✅ Rust, beta	❌	❌	❌	❌
No server to operate	❌	❌	❌	✅	⚠️ no broker, still a database
Runs without extensions on managed PostgreSQL	✅	n/a	n/a	n/a	❌ needs the extension

Two rows settle the question on their own. Replication is PostgreSQL’s, so one PostgreSQL is one failure domain. And routing is out of scope: no exchanges, no topic patterns, no header matching. If either is a requirement, it decides the choice regardless of any latency above.

Choosing something else

Choose	When
Kafka	Your ordering domains are few and stable rather than one per entity; you want the broker to replicate itself independently of a database; you need the connector and stream-processing ecosystem
RabbitMQ	You need real routing or protocol breadth beyond HTTP; ordering per queue is enough and you do not need replay
Amazon SQS	You want no server and no database to operate at all; your ordering domains fit FIFO’s quotas; you never need to re-read history
pgmq	You want PostgreSQL-backed queuing with no broker process; your consumers all speak SQL; a work queue with visibility timeouts is enough; you can install extensions
Queen	You need all three at once: strict FIFO per entity at cardinality you do not control, throughput that holds when the same lanes get busy, and consumer groups with replay, deduplication and dead-lettering over one stored copy

The boundaries of the model, stated directly, are in Limits and non-goals.

Comparison