High availability

Brokers hold no authoritative state, so a second broker is another copy of the same binary pointed at the same PostgreSQL. Messages, offsets, leases, deduplication, queue configuration and the dead-letter queue are all rows: nothing to replicate between brokers, no quorum, no serving leader, no split-brain.

That also says what replicas cover. A broker process dying, a rolling restart, one node’s network: yes. Losing PostgreSQL: no. The database is the failure domain, and keeping it alive is PostgreSQL’s own tooling.

Timers add one bounded cost to a rolling restart and no risk. A broker that is stopped while it holds claimed timer rows does not hand them back: in-flight HTTP requests drain, the sweeper task is aborted with the runtime, and those rows become claimable again only when their lease expires, up to QUEEN_SWEEPER_LEASE_MS (30 s) later. Nothing is lost and nothing is delivered twice, because the fire was never committed. A 30 second worst case on a deploy is cheaper than a second shutdown path to maintain.

Two brokers, condensed from test/compose/docker-compose.ha.yml:

services:
  queen-a:
    image: ghcr.io/queen-mq/queen:latest
    environment:
      PG_HOST: pg                       # the same database for every broker
      QUEEN_SERVER_ID: queen-a          # distinct per broker
      QUEEN_MESH_PORT: "6633"
      QUEEN_MESH_PEERS: "queen-b:6633"  # the other broker
      QUEEN_SYNC_SECRET: "change-me"    # byte-identical on every broker
      FILE_BUFFER_DIR: /var/lib/queen/buffers
    volumes: ["spool-a:/var/lib/queen/buffers"]   # never shared

queen-b is the mirror image: its own server id, its own volume, its peer list pointing back. The rest of the environment is what a single broker already takes (configuration).

Three brokers is the designed ceiling. Full-mesh connections grow quadratically, replicas × DB_POOL_SIZE has to fit PostgreSQL’s max_connections, and past three the bottleneck is the database rather than the number of brokers.

Put a load balancer in front and stop thinking about it. Every replica serves every route, with no session affinity and no partition ownership, so a pop taken from one broker is acked to another.

The mesh

Brokers talk to each other over framed TCP on QUEEN_MESH_PORT, and everything on that wire is a hint:

• pop wakes, so a push committed on A releases a long poll parked on B immediately
• hot-list marks, telling a peer that a partition went from empty to pending
• maintenance-mode flips
• queue-config invalidations, the instruction to forget a cached value and never the value itself

Messages, offsets, acks, leases and deduplication state never cross it. They belong to PostgreSQL, which is why deduplication stays exact across replicas with no coordination at all, and why leader election is an advisory lock rather than a protocol: the retention sweep and the statistics reconciler each run on exactly one replica per cycle.

The sweeper is the one background component that elects nothing. Every replica fires timers and prunes expired KV rows in parallel, sharing the work with FOR UPDATE ... SKIP LOCKED instead of a lock id, because any ownership scheme orphans the shards of a broker that dies and an orphaned timer never fires. Two brokers doing a little redundant scanning is a cost; a promised message that silently never arrives is a defect.

There is deliberately no mesh frame for a due timer either. The wake that a schedule rings is in-process only, so a timer scheduled on broker A and fired by broker B waits out B’s sleep ceiling, QUEEN_SWEEPER_MAX_SLEEP_MS, one second by default. That is correct rather than merely tolerable, because deliverAt is a floor and never an appointment, and it is the reason nobody may make the correctness of timers depend on the mesh being up.

Frames are dropped, not queued, when a peer is slow or down, and a dropped frame costs latency and nothing else, because every accelerated path has a periodic floor that re-reads the database. A broker with no peers configured binds nothing and keeps its in-process waker. The frame set and the cadences are in the mesh.

The mesh port is a control surface, not a data port, and it must be firewalled to the brokers themselves: Security.

While PostgreSQL is unreachable

A push does not have to fail. An item whose transaction cannot commit is appended to an on-disk spool under FILE_BUFFER_DIR, and the answer is still 201 with status: "buffered" on that item, which is why the per-item status is the contract and the status line is not. When the database returns, a drain replays each event under its original transactionId, so anything that had already committed comes back duplicate and writes nothing.

Reads need the database. Pops and acks error, safely, since an unacked lease redelivers, and POST /api/v1/transaction is all-or-nothing so it fails with them. GET /api/v1/status/buffers and the queen_file_buffer_pending gauge watch the backlog, and maintenance mode is the same diversion on purpose, for a planned failover (system routes).

How the drain behaves

The drain is not a loop that retries until something works. It classifies every failure and the classification decides whether the file is kept, retried or given up on.

One cycle runs every FILE_BUFFER_FLUSH_MS (100 ms by default). It first finalizes the file currently being written so its events become drainable, then replays finalized files oldest-first, which is FIFO.

Transient against permanent is a SQLSTATE decision, not a heuristic. A failure carrying no SQLSTATE at all is the connection itself, so it is transient. So are 40001 serialization failure, 40P01 deadlock detected, and the whole of classes 08, 53, 57 and 58: connection exception, insufficient resources, operator intervention, system error. Any other server-side error, a constraint violation or a RAISE inside a procedure, is permanent, because the same bytes will fail every retry. A spool file the broker cannot read is permanent for the same reason.

Two counters bound how long either failure can hold up everything behind it.

• Five attempts, then quarantine. A head file that fails permanently five consecutive times is moved to <FILE_BUFFER_DIR>/failed and the cycle carries on with the files behind it, so one poison file cannot block replay of every newer buffered message. If the move itself fails the file is deleted, on the reasoning that its data is a permanent SQL failure that can never be stored. The counter is per file: it resets when the head changes and on the first successful drain.
• Ten consecutive transient failures, then a 5 s cooldown. The circuit breaker parks the whole drain loop for five seconds and resets its counter, so a broker does not spin against a database that is still down.

Pacing is asymmetric on purpose. A cycle drains up to ten files while the database is marked healthy and exactly one while it is not, so a recovering broker refills PostgreSQL at ten files per cycle, while a broker that is still failing probes with a single file rather than a burst.

Replay is admitted, not privileged. Each file’s replay takes a slot in the Maint lane of the same admission arbiter that governs live traffic, and Maint is last in the wake order with a guaranteed share of 0.05, so a recovering spool cannot outrank the pushes, pops and acks arriving right now (flow control). Within a file, events are replayed in batches of FILE_BUFFER_MAX_BATCH (100 by default) rather than as one large transaction, and a batch that already committed replays idempotently on the next attempt at the file because the original transactionId is preserved.

The tenant is preserved on disk. Each spooled event carries the tenant resolved at push time, and the drain groups events by (tenant, queue, partition) and re-applies it on replay. A buffered push therefore cannot come back as the default tenant, which would be a quota and scope evasion. A push made by the default tenant omits the field, so a spool file written by a single-tenant broker is byte-identical to what it was before tenancy existed, and an older file without the field replays as the default tenant.

The spool is not replication

It is node-local and unreplicated, and a broker drains only the files it wrote itself. Behind a load balancer, buffered messages scatter across whichever brokers accepted them, and losing a node loses its spool: give every broker storage that outlives its container, and never point two brokers at one directory.

It has no size cap and no maximum age either, so a long outage grows it until the filesystem fills and pushes start reporting failed. Replay is idempotent exactly as far as the deduplication window reaches, and no further.

More brokers buy availability. Throughput comes from partitions, and durability comes from PostgreSQL.