Overview

High-performance transactional messaging for applications that need an ordered stream per entity.

Queen is a message broker written in Rust that keeps every byte of its state in PostgreSQL. Its defining abstraction is one logical ordered partition per application entity, a customer, an account, a conversation, a device, a workflow, a session or a job, created by the first push that names it and never provisioned in advance.

The problem

Most brokers give you ordering per shard. Your requirement is ordering per entity: this customer’s events processed in order, this conversation’s messages not overtaking each other, this account’s transactions settling in sequence.

Bridging the two is where the pain lives. Hash your entities onto a fixed partition count and the ones that collide block each other, so a slow customer stalls every customer sharing its shard. Give each entity its own queue instead and broker-side objects grow with your customer list.

Queen removes the bridge: the entity is the partition.

One entity, one ordered partition

You choose the partition key. It is your ordering boundary, not an infrastructure sizing decision: customer_id, account_id, conversation_id, device_id, workflow_id.

flowchart LR
A["customer A"] --> A1["A1 → A2 → A3"]
B["customer B"] --> B1["B1 → B2"]
C["customer C"] --> C1["C1 → C2 → C3"]
A1 --> AX["strict FIFO within the lane"]
B1 --> BX["B is not held up by A"]
C1 --> CX["C is not held up by A or B"]

The push creates the queue, the lane and the message. Nothing is preallocated, nothing is assigned, nothing rebalances when a consumer restarts.

const res = await client
  .queue('orders')
  .partition('customer-42')
  .push([{ data: { orderId: 9137, amount: 99.5 } }])

The transactionId is optional and is the idempotency key: give it your own and a retry of the same push writes nothing the second time.

Consumers choose how they read: which partition, how many at once, batch size, consumer group, lease time.

await client
  .queue('orders')
  .group('billing')
  .subscriptionMode('all')
  .limit(1)
  .each()
  .consume(async (message) => {
    console.log(message.data)
  })

Two things this does not mean. A single hot partition stays sequential, by design, because one leased batch exists per partition and consumer group: parallelism comes from many active partitions, not from splitting one, so add lanes rather than workers. And do not pick a key because it has high cardinality: pick the boundary your application genuinely requires. A partition per message is not a supported shape.

The whole of it, in one page: the model.

Transactional processing

The second reason Queen exists, and the reason PostgreSQL is not an implementation detail. One POST /api/v1/transaction bundles acknowledgements, pushes, key/value writes and timer operations into a single PostgreSQL transaction.

await client
  .queue('orders')
  .group('invoicing')
  .subscriptionMode('all')
  .each()
  .autoAck(false) // the acknowledgement belongs to the transaction, not to the loop
  .limit(1)
  .idleMillis(5000)
  .consume(async (message) => {
    // commit() throws when the broker rejects the bundle, so reaching the
    // line after it means the ack and the push are both durable.
    await client
      .transaction()
      .queue('invoices')
      .push([{ data: { orderId: message.data.orderId, invoiced: true } }])
      .ack(message, 'completed', { consumerGroup: 'invoicing' })
      .commit()
  })

The bundle is N to M: one call may acknowledge batches leased from any number of partitions, queues and consumer groups, and push to any number of queues and partitions, which is what makes a fan-in stage possible. This is what replaces transactional outbox tables, a separate store for idempotency markers, and the reconciliation code that exists only because the broker’s commit and the database’s commit were two different commits.

The mental model

Application entity        the thing whose order matters
    ▼  Partition key      your ordering boundary, chosen by you
    ▼  Ordered stream     one row, created on demand, strict FIFO
    ▼  Transaction        ack + state + output in one commit
    ▼  PostgreSQL         the single source of truth
    ▼  Stateless brokers  hold nothing durable; restart freely
    ▼  Cell               one deployment, one failure domain
    ▼  Region             where a cell physically lives

The top half is your application’s shape. The bottom half is infrastructure. Queen’s central bet is that these two halves should scale independently of each other.

What has been measured

Every figure on this site names the run that produced it, and a figure without an archived artifact recording its configuration does not get published. The two headline runs are Queen 1.0.0, one broker against one PostgreSQL 18 on a 32 vCPU / 62 GiB machine, with synchronous_commit left on.

• 86,369,975,300 messages in 24 hours, about 1,000,000 a second in each direction, pushed, popped and acknowledged, with explicit acks and deduplication on. Zero restarts, broker memory flat near 4.1 GB.
• 1,000,000 ordered partitions in one queue, none preallocated, created during the run at a thousand a second while serving 200,000 messages a second. Zero push, pop or ack errors over 722 million messages.
• 0 order violations across four stages and 1,000 partitions, over 88,503,408 verified messages, with zero duplicates and zero gaps.
• 0 cross-tenant deliveries over an hour with twelve tenants sharing one queue name and one consumer group name.

These are single-shape runs. They say nothing about your throughput, latency, PostgreSQL sizing, disk, retention capacity or partition distribution, which follow from your workload, payloads and hardware: read method and rig before quoting a number.

Application scale is not infrastructure scale

This is the principle the rest of the architecture follows from.

In most brokers the two are welded together: a per-entity ordering guarantee means a per-entity infrastructure object, either a topic partition with its own files and replicas or a live server-side queue. Doubling your customers means doubling something an operator has to think about.

In Queen a partition is a row. A million of them measured 315 MB in total, and the serve path does not care how many exist. Millions of logical entity streams do not require millions of infrastructure objects.

Three kinds of scale

Three axes, frequently confused, not interchangeable. Confusing them is the most common way to mis-size a deployment.

Inside a cell

A cell is PostgreSQL plus one or more stateless brokers, optionally fronted by queen_proxy. It is at once the scaling boundary, the failure boundary and the unit of upgrade and operational ownership. Cells do not coordinate with each other to process messages.

flowchart LR
C["clients<br/>six SDKs, queenctl, curl"] -->|HTTP| P["Queen Proxy<br/>optional: tenancy, quotas"]
P -->|HTTP| B["Queen brokers<br/>stateless, hold nothing durable"]
C -.->|HTTP, with no proxy| B
B -->|stored procedures| PG[("PostgreSQL<br/>the only durable state")]

PostgreSQL is the durable source of truth, and the brokers hold nothing authoritative. Messages, offsets, leases, deduplication state, queue configuration and dead letters are all rows. That is why a broker can be added, removed, restarted or rolled without a rebalance, and why deduplication stays exact across replicas with no coordination protocol at all. Brokers do exchange hints over a mesh port to shorten latency, but nothing on that wire is authoritative and a dropped hint costs nothing but time.

The failure domain is PostgreSQL. Queen does not replicate itself, so keeping the database alive is PostgreSQL’s own tooling. While it is unreachable, pushes spool to a node-local disk buffer and are replayed later, and reads fail safely because an unacknowledged lease redelivers (high availability).

Why PostgreSQL

Not because the bytes needed somewhere to live. PostgreSQL is chosen for what it lets the application do.

• Messaging state and application state share a transaction. This is the whole reason for the design, and no other storage choice offers it.
• Durability, ACID, replication, PITR, backup and recovery you already know how to operate.
• SQL introspection: your messages are rows, queryable with the tools you already have.
• No extensions and no migration step. The broker carries its own schema and applies it at boot. PostgreSQL 15 or newer.

The trade, plainly: the database is the throughput ceiling and the single failure domain.

Many tenants, one cell

queen_proxy is a second Rust binary and the tenant-facing boundary. Queen core stays focused on messaging, and everything a shared broker has no business holding lives in the proxy: per-cluster API keys and human logins, plan limits on rate, size and count, and usage metering.

Isolation is split across both processes on purpose. The broker scopes queue identity natively as (tenant, name) in SQL on every read and write, so two tenants both owning a queue called orders own different queues. The proxy is what makes the tenant identity driving that scoping trustworthy. Neither half is sufficient alone (isolation).

A cluster is the tenant-visible Queen: one hostname, one plan, one namespace. A cell is the physical stack it runs on. A cluster lives on exactly one cell and never spans two, which is why the proxy’s quota accounting is exact in-process state with nothing to coordinate.

What is in the box

• Messaging: ordered FIFO partitions created on demand, consumer groups with one cursor per partition, leases with explicit ack, nack, retry and dlq, replay and seek, retry budgets and a real dead-letter queue, long-poll consumption, retention by age and by completion.
• Exactly-once building blocks: deduplication at push keyed on your transaction id, exact rather than probabilistic, evaluated inside PostgreSQL so a duplicate writes nothing; transactional ack + kv + push + timers in one commit.
• State and time: queen.kv with optimistic locking and an expiry on every write, timers that stay cancellable until they fire, delayed delivery, window-buffer debounce, and conflation for last-value delivery.
• Stream processing: an operator chain that runs in your process with state in the same PostgreSQL. Four window types, event time with watermarks, per-message gating, and one cycle that commits state, sink pushes and the ack together.
• Ephemeral queues: an in-memory class with no database in the path, for request/reply, signalling and cache invalidation.
• Operations: one stateless binary serving HTTP on port 6632, six SDKs plus queenctl, a dashboard in the same binary on the same port, Prometheus metrics, JWT auth and a disk spool for database outages.
• Other brokers’ wire protocols, in the same image as the broker and off until you switch them on. queen-kafka advertises 32 Kafka API keys, so an unmodified Kafka producer or consumer reaches Queen by changing bootstrap.servers and nothing else; queen-sqs answers the SQS and SNS protocols, so an unmodified AWS SDK reaches it by changing endpoint_url. The contracts are reference/kafka and reference/sqs, running them is deploy/kafka and deploy/sqs.
• The log lands in your data lake. queen-s3, in the same image and off until you switch it on, reads a queue through the broker’s own API and writes JSONL or Parquet into any S3-compatible bucket under a Hive layout, exactly once, at any partition cardinality, with the per-entity order kept as two columns. DuckDB, Spark, ClickHouse and the rest read it without Queen. What the objects contain is reference/s3, running it is deploy/s3.

When something else is the better tool

These systems made different tradeoffs, and some of those tradeoffs are better than Queen’s for workloads that need them.

• The Kafka ecosystem, past the wire protocol. The protocol itself is spoken by the facade, but there is no log compaction and a transaction lives in one facade process, so Kafka Streams, Connect’s exactly-once source, the Schema Registry’s compacted topic, and Flink and Spark’s two-phase writers all stay out, and the literature about brokers, replicas and log directories does not apply.
• A single ordered stream that must itself scale horizontally. One lane is sequential. If your ordering boundary is everything, Queen’s core idea does nothing for you.
• Storage that scales independently of the database, or a system that replicates itself across regions. Retention lives in PostgreSQL, which is also the one failure domain.
• Rich content-based routing. No exchanges, no bindings, no header matching. That is RabbitMQ’s strength.
• No server to operate at all. That is Amazon SQS.

In one sentence: distributed logs make infrastructure partitions the fundamental scaling abstraction, and Queen makes application entities the fundamental ordering abstraction and cells the infrastructure scaling boundary. The side-by-side is the comparison.