---
title: "High availability"
description: "One to three identical brokers against one PostgreSQL: what the framed-TCP mesh carries, what it does not, which jobs are leader-gated, and why the mesh port must be firewalled."
---

> Documentation Index
> Fetch the complete documentation index at: https://queenmq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# High availability

Queen's high-availability model is deliberately small: run one to three identical
broker processes against **one** PostgreSQL, and put a load balancer in front of
them. The brokers hold no authoritative state, so there is nothing to replicate
between them, no quorum, no leader for serving traffic, and no split-brain to
reason about. PostgreSQL is the only source of truth: messages, offsets, the
deduplication index, queue configuration, leases and the DLQ all live there.

Availability of the *database* is therefore a separate problem, solved with
PostgreSQL's own tooling. Running three brokers protects you against a broker
process dying, a rolling restart, and a single node's network. It does not protect
you against losing PostgreSQL.

## The shape

<pre class="mermaid">{`
flowchart TB
  CL["clients"] --> LB["load balancer"]
  LB --> A["broker-a"]
  LB --> B["broker-b"]
  LB --> C["broker-c"]
  A -. "mesh hints" .- B
  B -.- C
  A -.- C
  A --> PG["PostgreSQL<br/>messages, offsets, leases, dedup, queue config"]
  B --> PG
  C --> PG
`}</pre>

Everything a client sends travels the solid path and ends in the single database
at the bottom. The dotted links are the mesh, and a frame dropped on one costs a
parked pop some latency and nothing else.

Every replica serves every route. A client can be routed to any of them on any
request; there is no session affinity to maintain and no notion of a partition
being "owned" by a broker. Leases are rows in `queen.log_consumers`, so a consumer
that pops from `broker-a` can acknowledge to `broker-c`.

### Configuration for a pair

The minimum: distinct `QUEEN_SERVER_ID`, matching `QUEEN_SYNC_SECRET`, peers
pointing at each other, and a **separate** `FILE_BUFFER_DIR` per broker.

| Variable | broker-a | broker-b |
| --- | --- | --- |
| `PG_HOST` | same | same |
| `QUEEN_SERVER_ID` | `queen-a` | `queen-b` |
| `QUEEN_MESH_PORT` | `6633` | `6633` |
| `QUEEN_MESH_PEERS` | `queen-b:6633` | `queen-a:6633` |
| `QUEEN_SYNC_SECRET` | identical to b | identical to a |
| `FILE_BUFFER_DIR` | node-local | node-local, **different volume** |

`test/compose/docker-compose.ha.yml` in the repository is exactly this, with a
runner that asserts the mesh is up.

> **Caution**
>
> The disk spool is node-local and must never be shared between brokers. It holds
> pushes accepted while PostgreSQL was unreachable, in append-only files each broker
> finalises and drains on its own. Two brokers pointed at one volume will fight over
> the same files.

`QUEEN_SYNC_ENABLED` defaults to `true`, but the mesh only starts when sync is
enabled **and** at least one peer is configured. A single stock broker binds
nothing and sends nothing: the in-process long-poll waker, which needs no cluster,
is the only thing live. If the mesh port is already in use, the broker logs a
`TCP mesh bind failed` warning and keeps serving with the local waker only.

Three replicas is the designed ceiling. The mesh is a full mesh of static peers,
so connection count grows with the square of the replica count, and every replica
multiplies your PostgreSQL connection demand: `replicas × DB_POOL_SIZE` has to
fit inside `max_connections`. Past three brokers, the bottleneck you are fighting
is almost always PostgreSQL, and another broker will not help.

## What the mesh is

Framed TCP. Each frame is `u32` big-endian length, then one type byte, then a JSON
payload; the length counts the type byte plus the payload. There are no sequence
numbers and no per-frame signature, because TCP already provides ordering and
integrity and the mesh tolerates loss. A length prefix above 16 MiB is rejected
before any allocation, so a corrupt or hostile peer cannot drive an
out-of-memory.

The topology is asymmetric on purpose, which sidesteps connection tie-breaking
entirely: every node listens on the mesh port and dials every configured peer, a
node **sends** only on its outbound (dialled) connections and **receives** only on
inbound (accepted) ones. Between any two nodes there are therefore two
connections, each used one way.

`QUEEN_UDP_PEERS` and `QUEEN_UDP_NOTIFY_PORT` still read the same settings as
`QUEEN_MESH_PEERS` and `QUEEN_MESH_PORT`. Only the names survive from the retired
UDP transport; the wire is TCP.

## What the mesh carries

Four kinds of hint, all best-effort:

| Hint | Effect on the receiving replica |
| --- | --- |
| **Pop wakes** (`MESSAGE_AVAILABLE`, and a batched form) | A push that committed on A wakes a long-poll pop parked on B immediately, instead of B waiting out its backoff. Also records the partition so the woken pop can target it. |
| **Hot-list dirty hints** (batched) | Marks the peer's in-memory wildcard-pop candidate set for a (queue, partition) that went from empty to pending. Coalesced: sent on the transition, not per message. |
| **Maintenance flips** | `MAINTENANCE_MODE_SET` and `POP_MAINTENANCE_MODE_SET` flip the receiving replica's flags, and keep its spool drain lifecycle in step. |
| **Queue-config invalidation** | `QUEUE_CONFIG_SET` / `QUEUE_CONFIG_DELETE` drop the peer's cached lease time and encryption flag for that queue. The **value** is not sent, only the instruction to forget. |

Plus `HELLO` at connection setup and a `HEARTBEAT` every
`QUEEN_SYNC_HEARTBEAT_MS` (default 1000), which is the periodic write that
detects a dead peer when no hints are flowing.

## What the mesh does not carry

This is the more useful list. The mesh never carries:

- **Messages.** No payload ever crosses it.
- **Offsets, acknowledgements or leases.** All of that is `queen.log_consumers`.
- **Deduplication state.** Dedup is enforced in SQL under the partition row lock,
  which is why it is exact across every replica without any coordination.
- **Queue configuration values.** Only invalidations. A replica re-reads the
  configuration from PostgreSQL lazily.
- **Cluster membership as authority.** Peer liveness is used for logging and the
  stats surface. Nothing routes on it, and a replica that believes every peer is
  dead serves normally.
- **Any form of leader election.** That is done with PostgreSQL advisory locks.

### Loss is designed in, not tolerated grudgingly

Each peer has a bounded send queue of 1024 frames. When it is full (the peer is
slow) or closed (the peer is down), the frame is **dropped** and a counter is
bumped. The send path never blocks a caller and never grows without bound. On
reconnect, anything that queued while the peer was down is discarded *before* the
peer is marked connected, because replaying stale wakes is pointless and bursting
an out-of-date maintenance state onto a peer is actively wrong.

Reconnection backs off from 250 ms to a 5-second ceiling, re-resolving the
hostname on every attempt, so a peer whose address moved is picked up without a
separate DNS refresh.

Three independent mechanisms make a lost frame cost latency rather than
correctness:

| Mechanism | Cadence | Heals |
| --- | --- | --- |
| Periodic reconcile | `QUEEN_CACHE_REFRESH_INTERVAL_MS`, default 60000 ms | Re-reads both maintenance flags from `queen.system_state` (the database is the truth) and clears the per-queue lease and encryption caches, so a lost config invalidation heals within one interval. |
| Pop long-poll backoff | Up to `POP_WAIT_MAX_INTERVAL_MS`, default 1000 ms | A parked pop re-queries on its own schedule regardless of wakes, so a dropped wake costs at most one interval of latency. |
| Hot-list reseed floor | `QUEEN_HOTLIST_RESEED_MS`, default 30000 ms | Rebuilds a candidate ring from PostgreSQL even when it is non-empty, recovering a partition dropped by a missed mark or a dropped hint. |

The reconcile loop runs unconditionally, on a single broker too, because
reconverging on the database is useful even without peers.

## Leader-gated background jobs

Three background services write on a cadence. Two are gated so exactly one
replica does the work per cycle, and the gate is a PostgreSQL advisory lock: a
replica that cannot take it skips that cycle and tries again on the next one.
There is no persistent leader and no election protocol.

| Job | Cadence | Gate |
| --- | --- | --- |
| Retention and eviction sweep | `RETENTION_INTERVAL`, default 5000 ms | Session lock `737001`, taken with `pg_try_advisory_lock` and released on every exit path. |
| Statistics reconciler | `STATS_INTERVAL_MS`, default 10000 ms | Transaction lock `737002`, so it releases with the cycle's commit. Distinct from the retention lock so the two never block each other. |
| System and worker metrics collector | `METRICS_FLUSH_MS`, default 60000 ms | **Not gated.** Every replica records its own rows, keyed by host and worker; the dashboard aggregates across replicas. |

Because the retention lock is session-scoped and held across a whole cycle of
autocommitting steps, it is one of the two reasons the broker needs a direct
PostgreSQL connection. See [PostgreSQL](/selfhost/postgres).

## The mesh port must be firewalled

Not advice. A requirement.

The `HELLO` handshake is `{server_id, nonce, mac}` with
`mac = HMAC-SHA256(secret, nonce)`, verified in constant time before any further
frame is accepted. That is the entire authentication story, and it has three
properties an operator has to plan around:

1. **The handshake is replayable.** The nonce is generated by the dialer and the
   listener never tracks nonces it has already seen. A captured `HELLO` can be
   replayed verbatim to open a new accepted connection, without knowing
   `QUEEN_SYNC_SECRET`.
2. **Post-handshake frames are unauthenticated JSON.** There is no per-frame MAC.
   Once a connection is accepted, every frame in the set is honoured.
3. **The frame set includes `MAINTENANCE_MODE_SET`.** Maintenance mode diverts
   pushes to the disk spool; pop maintenance mode makes every pop answer empty.
   Anyone who can complete a handshake (by replay) can flip both on a replica.

Inbound connections are receive-only: the broker never writes back on an accepted
connection. An attacker therefore needs no reply channel at all, only the ability
to connect and send.

> **Danger**
>
> Bind the mesh to a private network and allow the mesh port **only** from the other
> brokers' addresses. Concretely: a `NetworkPolicy` selecting the broker pods, a
> security-group rule limited to the broker instances, or a Docker network the
> brokers share with nothing else. Never publish the mesh port on a host interface
> that is reachable from a client network, and never expose it to the internet.

Setting `QUEEN_SYNC_SECRET` is still worth doing: an empty secret is *open mode*,
where any well-formed `HELLO` is accepted from anyone, so the secret raises the
bar from "connect" to "capture one handshake". It does not replace the firewall.
The boot `config: sync` block reports `auth` as `hmac` or `off`, and the mesh
start line does the same.

## What HA does not buy you

Two limits are worth stating plainly, because both are properties of the
consumption model rather than of the deployment.

**Replicas do not parallelise a single partition.** Exactly one in-flight leased
batch exists per (partition, consumer group). A queue with one partition cannot be
consumed in parallel by one consumer group, however many brokers you run. Scaling
consumption means more partitions.

**Replicas do not increase PostgreSQL's capacity.** Every push is a commit and
every pop and ack is a row update, in the one database all replicas share. Adding
a broker adds HTTP capacity and connection demand at the same time. When
throughput stops improving, the answer is in PostgreSQL, not in another broker.

## Related

- [Upgrades](/selfhost/upgrade) — Why an upgrade is a fresh database plus a cutover, and what a restart does to in-flight leases.
- [Kubernetes](/selfhost/kubernetes) — A reference manifest with the mesh port as TCP, peers for replicas, and a Secret for the mesh secret.
- [PostgreSQL](/selfhost/postgres) — Pool sizing across replicas, and why the connection must be direct.

Source: https://queenmq.com/selfhost/ha/index.mdx