The mesh

Queen runs one to three identical broker replicas over one PostgreSQL database. PostgreSQL is the sole source of truth, so the replicas do not need to agree about anything: there is no consensus protocol here, no leader for the data path, and no replicated state. The mesh exists only to make three things prompt that would otherwise be merely eventual:

1. Waking a parked pop. A push commits on replica A; a long-poll parked on replica B should re-poll now rather than at the end of its backoff interval.
2. Maintenance-mode flips, so all replicas divert or pause together.
3. Queue-config invalidation, so a peer drops its cached lease time and encryption flag after a /configure.

Everything the mesh carries is therefore a best-effort hint. The reconcile loop re-reads PostgreSQL every QUEEN_CACHE_REFRESH_INTERVAL_MS (60000 ms) and a parked pop re-polls on its own backoff, capped at POP_WAIT_MAX_INTERVAL_MS. A dropped frame costs latency and nothing else. That invariant is why the send path drops rather than buffers when a peer is slow or down: it never blocks a caller and the queue never grows without bound.

Enabling it

Two things must both be true: QUEEN_SYNC_ENABLED (default true) and at least one configured peer. A single stock broker binds nothing and sends nothing. Only the in-process waker runs.

The QUEEN_UDP_* names survive as aliases and nothing else about the old transport does. The transport is TCP; the names are historical.

QUEEN_MESH_PEERS is a comma-separated list of host or host:port entries; a bare host uses the mesh port. An http:// or https:// prefix is tolerated and stripped, whitespace is trimmed, and a non-numeric port tail falls back to the default port rather than failing.

Topology

A full mesh of static peers. Every node listens on the mesh port and dials every configured peer in a reconnect loop.

The two directions are deliberately asymmetric: a node sends only on its outbound (dialled) connections and receives only on inbound (accepted) ones. Between any two nodes there are therefore two connections, each used one way. This sidesteps connection tie-breaking entirely: there is no protocol for deciding which of two simultaneous connections wins, because both are kept and each has one job.

The dial loop re-resolves the hostname on every attempt, so a peer whose IP moved is picked up without a separate DNS refresh loop. Reconnect backoff runs from 250 ms to 5 s. Each peer has a bounded send queue of 1024 frames; when it is full the frame is dropped and a counter is bumped.

Liveness is inbound-based: the last time any frame was received from a peer, keyed by the server id it announced in its handshake. A peer whose last inbound frame is older than QUEEN_SYNC_DEAD_THRESHOLD_MS is reported dead in the status output. Heartbeat frames every QUEEN_SYNC_HEARTBEAT_MS keep that signal alive on an idle cluster.

Framing

u32 BE length | u8 type | JSON payload

length counts the type byte plus the payload. No fixed header, no per-frame signature, no sequence numbers. TCP already provides ordering and integrity, and the mesh tolerates loss.

A length prefix larger than 16 MiB is rejected before allocating, so a corrupt or hostile peer cannot drive an out-of-memory condition with a bogus four-gigabyte length. Batched wakes are the only non-tiny frame and fit comfortably below that.

Framing is what made batching possible at all: the retired UDP transport had a 1308-byte datagram cap, so a multi-partition push meant one datagram per partition.

The frame set

Two of these are worth separating because they look redundant and are not. MESSAGE_AVAILABLE is a parked-pop wake, sent on every push. HOTLIST_DIRTY_BATCH marks a peer’s candidate ring, and is sent only on a coalesced dirty transition: a partition pushed a thousand times in one flush window produces one hint. A dropped hint is healed by the hot-list’s periodic reseed floor; a dropped wake is healed by the parked pop’s own backoff.

Queue-carrying frames include the tenant explicitly, and an absent tenant is not the same as the default tenant. An absent tenant means a peer that predates tenant scoping (a rolling upgrade), and the receiver must then fan the mark out to every tenant holding that queue name. Attributing it to the default tenant would silently drop the wake for everyone else. With tenancy off no non-default tenant can exist, so the bare name provably names the default tenant and the receive path is one hash lookup.

The handshake

A per-connection handshake replaces per-packet authentication. On connect the dialer sends HELLO with {server_id, nonce, mac} where mac = HMAC-SHA256(secret, nonce) over the nonce’s hex bytes. The listener verifies it in constant time before accepting any further frame. An empty QUEEN_SYNC_SECRET means open mode: the HELLO is still exchanged, to learn the peer’s server id, but not verified.

The security reality

State this plainly, because the mechanism above reads as stronger than it is.

• The handshake is replayable. The nonce is generated by the dialer and the listener never tracks nonces it has seen. A captured HELLO can be replayed by anyone who can reach the mesh port, and it will authenticate.
• Post-handshake frames are unauthenticated JSON. There is no per-frame MAC and no sequence number. Once a connection is accepted, everything on it is trusted.
• The frame set includes MAINTENANCE_MODE_SET. So an attacker who can reach the port and replay one captured handshake can put every replica into maintenance mode, which diverts all pushes to the disk spool. POP_MAINTENANCE_MODE_SET will likewise pause consumption cluster-wide.
• With no secret configured, not even the replay is needed.

Nothing on the mesh can corrupt or misdeliver messages, because it carries no data-plane writes and PostgreSQL rejects nothing it says. The exposure is availability and operational control, not integrity.

What it cannot do

• It is not a replication channel. No offset, cursor, lease or segment ever travels on it.
• It is not required for correctness. Turn it off (QUEEN_SYNC_ENABLED=0, or configure no peers) and a multi-replica cluster still works: cross-replica discovery falls back to the parked pop’s backoff and the reconcile loop’s 60-second re-read. You lose latency, not correctness.
• It is not ordered across peers. A frame is a hint that something changed; it never carries a version, and receipt is idempotent by construction.
• It has no C++ or cross-version wire compatibility requirement. Rust-only clusters, which is why the wire is this simple.

Observing it

Two internal routes return the transport’s own statistics: GET /internal/api/shared-state/stats and GET /internal/api/inter-instance/stats. Both emit the same body: the local server id, the transport name (tcp-mesh), the port, and per peer the host, port, whether an outbound connection is established and its resolved remote IP. Alongside that, how many peers are alive versus dead by the inbound-liveness threshold, frames sent, received and dropped, handshake failures, and the two maintenance flags. With no peers configured the body is {"enabled": false, "reason": "no_peers", ...}.

GET /api/v1/system/shared-state is a different, deliberately minimal route: it always reports enabled: false with the reason single_node_segments_broker plus the two maintenance flags. It is not the mesh view.

Two of those are the ones to watch. A rising messages_dropped means a peer’s bounded send queue is full: the peer is slow or down, and discovery on it has fallen back to backoff. A rising handshake failure count means something is dialling the port that does not share the secret, which is worth investigating before it is anything else.