---
title: "The mesh"
description: "Wakeups and coordination between replicas over framed TCP: the HMAC HELLO handshake, the frame set, and why firewalling the mesh port is a requirement rather than advice."
---

> Documentation Index
> Fetch the complete documentation index at: https://queenmq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# The mesh

Queen runs one to three identical broker replicas over one PostgreSQL database. PostgreSQL is the
sole source of truth, so the replicas do not need to agree about anything: there is no consensus
protocol here, no leader for the data path, and no replicated state. The mesh exists only to make
three things prompt that would otherwise be merely eventual:

1. **Waking a parked pop.** A push commits on replica A; a long-poll parked on replica B should
   re-poll now rather than at the end of its backoff interval.
2. **Maintenance-mode flips**, so all replicas divert or pause together.
3. **Queue-config invalidation**, so a peer drops its cached lease time and encryption flag after a
   `/configure`.

Everything the mesh carries is therefore a **best-effort hint**. The reconcile loop re-reads
PostgreSQL every `QUEEN_CACHE_REFRESH_INTERVAL_MS` (60000 ms) and a parked pop re-polls on its own
backoff, capped at `POP_WAIT_MAX_INTERVAL_MS`. A dropped frame costs latency and nothing else. That
invariant is why the send path **drops rather than buffers** when a peer is slow or down: it never
blocks a caller and the queue never grows without bound.

## Enabling it

Two things must both be true: `QUEEN_SYNC_ENABLED` (default true) and at least one configured peer.
A single stock broker binds nothing and sends nothing. Only the in-process waker runs.

| Variable | Default | Legacy alias |
| --- | --- | --- |
| `QUEEN_MESH_PEERS` | empty | `QUEEN_UDP_PEERS` |
| `QUEEN_MESH_PORT` | 6633 | `QUEEN_UDP_NOTIFY_PORT` |
| `QUEEN_SYNC_SECRET` | empty | none |
| `QUEEN_SYNC_HEARTBEAT_MS` | 1000 | none |
| `QUEEN_SYNC_DEAD_THRESHOLD_MS` | 5000 | none |

The `QUEEN_UDP_*` names survive as aliases and nothing else about the old transport does. The
transport is TCP; the names are historical.

`QUEEN_MESH_PEERS` is a comma-separated list of `host` or `host:port` entries; a bare host uses the
mesh port. An `http://` or `https://` prefix is tolerated and stripped, whitespace is trimmed, and a
non-numeric port tail falls back to the default port rather than failing.

## Topology

A full mesh of static peers. Every node listens on the mesh port and dials every configured peer in
a reconnect loop.

The two directions are deliberately **asymmetric**: a node *sends* only on its outbound (dialled)
connections and *receives* only on inbound (accepted) ones. Between any two nodes there are
therefore two connections, each used one way. This sidesteps connection tie-breaking entirely:
there is no protocol for deciding which of two simultaneous connections wins, because both are kept
and each has one job.

The dial loop re-resolves the hostname on every attempt, so a peer whose IP moved is picked up
without a separate DNS refresh loop. Reconnect backoff runs from 250 ms to 5 s. Each peer has a
bounded send queue of 1024 frames; when it is full the frame is dropped and a counter is bumped.

Liveness is inbound-based: the last time any frame was received from a peer, keyed by the server id
it announced in its handshake. A peer whose last inbound frame is older than
`QUEEN_SYNC_DEAD_THRESHOLD_MS` is reported dead in the status output. Heartbeat frames every
`QUEEN_SYNC_HEARTBEAT_MS` keep that signal alive on an idle cluster.

## Framing

```text
u32 BE length | u8 type | JSON payload
```

`length` counts the type byte plus the payload. No fixed header, no per-frame signature, no sequence
numbers. TCP already provides ordering and integrity, and the mesh tolerates loss.

A length prefix larger than 16 MiB is rejected *before* allocating, so a corrupt or hostile peer
cannot drive an out-of-memory condition with a bogus four-gigabyte length. Batched wakes are the
only non-tiny frame and fit comfortably below that.

Framing is what made batching possible at all: the retired UDP transport had a 1308-byte datagram
cap, so a multi-partition push meant one datagram per partition.

## The frame set

| Tag | Frame | Payload | Effect on receipt |
| --- | --- | --- | --- |
| 1 | `MESSAGE_AVAILABLE` | `{queue, partition, tenant}` | wake local pops parked on that queue, record the partition hint; no re-broadcast |
| 2 | `MESSAGE_AVAILABLE_BATCH` | `{items:[{queue, partition, tenant}]}` | same, once per item (one send for a whole multi-partition push) |
| 3 | `HEARTBEAT` | none | liveness only |
| 4 | `HELLO` | `{server_id, nonce, mac}` | connection handshake; see below |
| 5 | `HOTLIST_DIRTY_BATCH` | `{items:[{queue, partition, tenant}]}` | mark the peer's hot-list rings pending (idempotent, commutative); no re-broadcast |
| 10 | `QUEUE_CONFIG_SET` | `{queue, tenant}` | drop the cached lease time and encryption flag for that queue |
| 11 | `QUEUE_CONFIG_DELETE` | `{queue, tenant}` | same invalidation |
| 40 | `MAINTENANCE_MODE_SET` | `{enabled}` | flip the maintenance atomic |
| 41 | `POP_MAINTENANCE_MODE_SET` | `{enabled}` | flip the pop-maintenance atomic |

Two of these are worth separating because they look redundant and are not.
`MESSAGE_AVAILABLE` is a parked-pop wake, sent on every push. `HOTLIST_DIRTY_BATCH` marks a peer's
candidate ring, and is sent only on a coalesced dirty transition: a partition pushed a thousand
times in one flush window produces one hint. A dropped hint is healed by the hot-list's periodic
reseed floor; a dropped wake is healed by the parked pop's own backoff.

Queue-carrying frames include the tenant explicitly, and an **absent** tenant is not the same as the
default tenant. An absent tenant means a peer that predates tenant scoping (a rolling upgrade), and
the receiver must then fan the mark out to every tenant holding that queue name. Attributing it to
the default tenant would silently drop the wake for everyone else. With tenancy off no non-default
tenant can exist, so the bare name provably names the default tenant and the receive path is one hash
lookup.

## The handshake

A per-connection handshake replaces per-packet authentication. On connect the dialer sends `HELLO`
with `{server_id, nonce, mac}` where `mac = HMAC-SHA256(secret, nonce)` over the nonce's hex bytes.
The listener verifies it in constant time before accepting any further frame. An empty
`QUEEN_SYNC_SECRET` means open mode: the `HELLO` is still exchanged, to learn the peer's server id,
but not verified.

## The security reality

State this plainly, because the mechanism above reads as stronger than it is.

- **The handshake is replayable.** The nonce is generated by the **dialer** and the listener never
  tracks nonces it has seen. A captured `HELLO` can be replayed by anyone who can reach the mesh
  port, and it will authenticate.
- **Post-handshake frames are unauthenticated JSON.** There is no per-frame MAC and no sequence
  number. Once a connection is accepted, everything on it is trusted.
- **The frame set includes `MAINTENANCE_MODE_SET`.** So an attacker who can reach the port and replay
  one captured handshake can put every replica into maintenance mode, which diverts all pushes to the
  disk spool. `POP_MAINTENANCE_MODE_SET` will likewise pause consumption cluster-wide.
- **With no secret configured, not even the replay is needed.**

> **Danger**
>
> **Firewalling the mesh port is a requirement, not advice.** `QUEEN_MESH_PORT` must be reachable only
> from the other brokers in the cluster: a private network, a security group, or a Kubernetes network
> policy. Do not treat `QUEEN_SYNC_SECRET` as a substitute for that. Set it as well, but the network
> boundary is what actually protects the cluster.

Nothing on the mesh can corrupt or misdeliver messages, because it carries no data-plane writes and
PostgreSQL rejects nothing it says. The exposure is availability and operational control, not
integrity.

## What it cannot do

- **It is not a replication channel.** No offset, cursor, lease or segment ever travels on it.
- **It is not required for correctness.** Turn it off (`QUEEN_SYNC_ENABLED=0`, or configure no peers)
  and a multi-replica cluster still works: cross-replica discovery falls back to the parked pop's
  backoff and the reconcile loop's 60-second re-read. You lose latency, not correctness.
- **It is not ordered across peers.** A frame is a hint that something changed; it never carries a
  version, and receipt is idempotent by construction.
- **It has no C++ or cross-version wire compatibility requirement.** Rust-only clusters, which is
  why the wire is this simple.

## Observing it

Two internal routes return the transport's own statistics:
`GET /internal/api/shared-state/stats` and `GET /internal/api/inter-instance/stats`. Both emit the
same body: the local server id, the transport name (`tcp-mesh`), the port, and per peer the host,
port, whether an outbound connection is established and its resolved remote IP. Alongside that, how
many peers are alive versus dead by the inbound-liveness threshold, frames sent, received and
**dropped**, handshake failures, and the two maintenance flags. With no peers configured the body is
`{"enabled": false, "reason": "no_peers", ...}`.

`GET /api/v1/system/shared-state` is a different, deliberately minimal route: it always reports
`enabled: false` with the reason `single_node_segments_broker` plus the two maintenance flags. It is
not the mesh view.

Two of those are the ones to watch. A rising `messages_dropped` means a peer's bounded send queue is
full: the peer is slow or down, and discovery on it has fallen back to backoff. A rising handshake
failure count means something is dialling the port that does not share the secret, which is worth
investigating before it is anything else.

Source: https://queenmq.com/internals/mesh/index.mdx