High availability

Queen’s high-availability model is deliberately small: run one to three identical broker processes against one PostgreSQL, and put a load balancer in front of them. The brokers hold no authoritative state, so there is nothing to replicate between them, no quorum, no leader for serving traffic, and no split-brain to reason about. PostgreSQL is the only source of truth: messages, offsets, the deduplication index, queue configuration, leases and the DLQ all live there.

Availability of the database is therefore a separate problem, solved with PostgreSQL’s own tooling. Running three brokers protects you against a broker process dying, a rolling restart, and a single node’s network. It does not protect you against losing PostgreSQL.

The shape

flowchart TB
CL["clients"] --> LB["load balancer"]
LB --> A["broker-a"]
LB --> B["broker-b"]
LB --> C["broker-c"]
A -. "mesh hints" .- B
B -.- C
A -.- C
A --> PG["PostgreSQL<br/>messages, offsets, leases, dedup, queue config"]
B --> PG
C --> PG

Everything a client sends travels the solid path and ends in the single database at the bottom. The dotted links are the mesh, and a frame dropped on one costs a parked pop some latency and nothing else.

Every replica serves every route. A client can be routed to any of them on any request; there is no session affinity to maintain and no notion of a partition being “owned” by a broker. Leases are rows in queen.log_consumers, so a consumer that pops from broker-a can acknowledge to broker-c.

Configuration for a pair

The minimum: distinct QUEEN_SERVER_ID, matching QUEEN_SYNC_SECRET, peers pointing at each other, and a separate FILE_BUFFER_DIR per broker.

test/compose/docker-compose.ha.yml in the repository is exactly this, with a runner that asserts the mesh is up.

QUEEN_SYNC_ENABLED defaults to true, but the mesh only starts when sync is enabled and at least one peer is configured. A single stock broker binds nothing and sends nothing: the in-process long-poll waker, which needs no cluster, is the only thing live. If the mesh port is already in use, the broker logs a TCP mesh bind failed warning and keeps serving with the local waker only.

Three replicas is the designed ceiling. The mesh is a full mesh of static peers, so connection count grows with the square of the replica count, and every replica multiplies your PostgreSQL connection demand: replicas × DB_POOL_SIZE has to fit inside max_connections. Past three brokers, the bottleneck you are fighting is almost always PostgreSQL, and another broker will not help.

What the mesh is

Framed TCP. Each frame is u32 big-endian length, then one type byte, then a JSON payload; the length counts the type byte plus the payload. There are no sequence numbers and no per-frame signature, because TCP already provides ordering and integrity and the mesh tolerates loss. A length prefix above 16 MiB is rejected before any allocation, so a corrupt or hostile peer cannot drive an out-of-memory.

The topology is asymmetric on purpose, which sidesteps connection tie-breaking entirely: every node listens on the mesh port and dials every configured peer, a node sends only on its outbound (dialled) connections and receives only on inbound (accepted) ones. Between any two nodes there are therefore two connections, each used one way.

QUEEN_UDP_PEERS and QUEEN_UDP_NOTIFY_PORT still read the same settings as QUEEN_MESH_PEERS and QUEEN_MESH_PORT. Only the names survive from the retired UDP transport; the wire is TCP.

What the mesh carries

Four kinds of hint, all best-effort:

Plus HELLO at connection setup and a HEARTBEAT every QUEEN_SYNC_HEARTBEAT_MS (default 1000), which is the periodic write that detects a dead peer when no hints are flowing.

What the mesh does not carry

This is the more useful list. The mesh never carries:

• Messages. No payload ever crosses it.
• Offsets, acknowledgements or leases. All of that is queen.log_consumers.
• Deduplication state. Dedup is enforced in SQL under the partition row lock, which is why it is exact across every replica without any coordination.
• Queue configuration values. Only invalidations. A replica re-reads the configuration from PostgreSQL lazily.
• Cluster membership as authority. Peer liveness is used for logging and the stats surface. Nothing routes on it, and a replica that believes every peer is dead serves normally.
• Any form of leader election. That is done with PostgreSQL advisory locks.

Loss is designed in, not tolerated grudgingly

Each peer has a bounded send queue of 1024 frames. When it is full (the peer is slow) or closed (the peer is down), the frame is dropped and a counter is bumped. The send path never blocks a caller and never grows without bound. On reconnect, anything that queued while the peer was down is discarded before the peer is marked connected, because replaying stale wakes is pointless and bursting an out-of-date maintenance state onto a peer is actively wrong.

Reconnection backs off from 250 ms to a 5-second ceiling, re-resolving the hostname on every attempt, so a peer whose address moved is picked up without a separate DNS refresh.

Three independent mechanisms make a lost frame cost latency rather than correctness:

The reconcile loop runs unconditionally, on a single broker too, because reconverging on the database is useful even without peers.

Leader-gated background jobs

Three background services write on a cadence. Two are gated so exactly one replica does the work per cycle, and the gate is a PostgreSQL advisory lock: a replica that cannot take it skips that cycle and tries again on the next one. There is no persistent leader and no election protocol.

Because the retention lock is session-scoped and held across a whole cycle of autocommitting steps, it is one of the two reasons the broker needs a direct PostgreSQL connection. See PostgreSQL.

The mesh port must be firewalled

Not advice. A requirement.

The HELLO handshake is {server_id, nonce, mac} with mac = HMAC-SHA256(secret, nonce), verified in constant time before any further frame is accepted. That is the entire authentication story, and it has three properties an operator has to plan around:

1. The handshake is replayable. The nonce is generated by the dialer and the listener never tracks nonces it has already seen. A captured HELLO can be replayed verbatim to open a new accepted connection, without knowing QUEEN_SYNC_SECRET.
2. Post-handshake frames are unauthenticated JSON. There is no per-frame MAC. Once a connection is accepted, every frame in the set is honoured.
3. The frame set includes MAINTENANCE_MODE_SET. Maintenance mode diverts pushes to the disk spool; pop maintenance mode makes every pop answer empty. Anyone who can complete a handshake (by replay) can flip both on a replica.

Inbound connections are receive-only: the broker never writes back on an accepted connection. An attacker therefore needs no reply channel at all, only the ability to connect and send.

Setting QUEEN_SYNC_SECRET is still worth doing: an empty secret is open mode, where any well-formed HELLO is accepted from anyone, so the secret raises the bar from “connect” to “capture one handshake”. It does not replace the firewall. The boot config: sync block reports auth as hmac or off, and the mesh start line does the same.

What HA does not buy you

Two limits are worth stating plainly, because both are properties of the consumption model rather than of the deployment.

Replicas do not parallelise a single partition. Exactly one in-flight leased batch exists per (partition, consumer group). A queue with one partition cannot be consumed in parallel by one consumer group, however many brokers you run. Scaling consumption means more partitions.

Replicas do not increase PostgreSQL’s capacity. Every push is a commit and every pop and ack is a row update, in the one database all replicas share. Adding a broker adds HTTP capacity and connection demand at the same time. When throughput stops improving, the answer is in PostgreSQL, not in another broker.

Related