The mesh

Queen runs one to three identical broker replicas over one PostgreSQL database. PostgreSQL is the sole source of truth, so the replicas do not need to agree about anything: there is no consensus protocol here, no leader for the data path, and no replicated state. The mesh exists only to make three things prompt that would otherwise be merely eventual:

1. Waking a parked pop. A write lands frames on replica A; a long-poll parked on replica B should re-poll now rather than at the end of its backoff interval.
2. Maintenance-mode flips, so all replicas divert or pause together.
3. Queue-config invalidation, so a peer drops its cached lease time and encryption flag after a /configure.

Everything the mesh carries is therefore a best-effort hint. The reconcile loop re-reads PostgreSQL every QUEEN_CACHE_REFRESH_INTERVAL_MS (60000 ms) and a parked pop re-polls on its own backoff, capped at POP_WAIT_MAX_INTERVAL_MS. A dropped frame costs latency and nothing else. That invariant is why the send path drops rather than buffers when a peer is slow or down: it never blocks a caller and the queue never grows without bound.

Enabling it

Two things must both be true: QUEEN_SYNC_ENABLED (default true) and at least one configured peer. A single stock broker binds nothing and sends nothing. Only the in-process waker runs.

QUEEN_MESH_BIND_ADDR narrows where the mesh listener binds without moving the HTTP listener. QUEEN_MESH_ADVERTISE_HOST is the host this broker tells its peers to reach its HTTP surface on (the port is always the HTTP listener’s own PORT); it feeds the http_addr the handshake below advertises, and defaults to the OS hostname.

The QUEEN_UDP_* names survive as aliases and nothing else about the old transport does. The transport is TCP; the names are historical.

QUEEN_MESH_PEERS is a comma-separated list of host or host:port entries; a bare host uses the mesh port. An http:// or https:// prefix is tolerated and stripped, whitespace is trimmed, and a non-numeric port tail falls back to the default port rather than failing.

Topology

A full mesh of static peers. Every node listens on the mesh port and dials every configured peer in a reconnect loop.

The two directions are deliberately asymmetric: a node sends only on its outbound (dialled) connections and receives only on inbound (accepted) ones. Between any two nodes there are therefore two connections, each used one way. This sidesteps connection tie-breaking entirely: there is no protocol for deciding which of two simultaneous connections wins, because both are kept and each has one job.

The dial loop re-resolves the hostname on every attempt, so a peer whose IP moved is picked up without a separate DNS refresh loop. Reconnect backoff runs from 250 ms to 5 s. Each peer has a bounded send queue of 1024 frames; when it is full the frame is dropped and a counter is bumped.

Liveness is inbound-based: the last time any frame was received from a peer, keyed by the server id it announced in its handshake. A peer whose last inbound frame is older than QUEEN_SYNC_DEAD_THRESHOLD_MS is reported dead in the status output. Heartbeat frames every QUEEN_SYNC_HEARTBEAT_MS keep that signal alive on an idle cluster.

Framing

u32 BE length | u8 type | JSON payload

length counts the type byte plus the payload. No fixed header, no per-frame signature, no sequence numbers. TCP already provides ordering and integrity, and the mesh tolerates loss.

A length prefix larger than 16 MiB is rejected before allocating, so a corrupt or hostile peer cannot drive an out-of-memory condition with a bogus four-gigabyte length. Batched wakes are the only non-tiny frame and fit comfortably below that.

Framing is what made batching possible at all: the retired UDP transport had a 1308-byte datagram cap, so a multi-partition push meant one datagram per partition.

The frame set

Two of these are worth separating because they look redundant and are not. MESSAGE_AVAILABLE is a parked-pop wake, sent by every commit that lands frames in the log: a push, a streams sink emit, a fired timer, a spool replay after an outage. HOTLIST_DIRTY_BATCH marks a peer’s candidate ring, and is sent only on a coalesced dirty transition: a partition pushed a thousand times in one flush window produces one hint. A dropped hint for a landing write is healed by the hot-list’s periodic reseed floor; a dropped wake is healed by the parked pop’s own backoff. A dropped hint for a cursor move is healed by neither, because the reseed floor is bounded to recently written partitions and a seek writes nothing: that one has a durable marker table under it, read on the reconcile pass.

group was added the same way tenant was, as an optional field on an existing frame rather than a new tag: a peer mid rolling-upgrade drops a tag it does not know, but ignores a field it does not know and does the queue-wide mark it has always done, which is a superset of the scoped one.

Queue-carrying frames include the tenant explicitly, and an absent tenant is not the same as the default tenant. An absent tenant means a peer that predates tenant scoping (a rolling upgrade), and the receiver must then fan the mark out to every tenant holding that queue name. Attributing it to the default tenant would silently drop the wake for everyone else. With tenancy off no non-default tenant can exist, so the bare name provably names the default tenant and the receive path is one hash lookup.

The handshake

A per-connection handshake replaces per-packet authentication. On connect the dialer sends HELLO with {server_id, nonce, mac, http_addr, eph_epoch} where mac = HMAC-SHA256(secret, nonce) over the nonce’s hex bytes. The listener verifies it in constant time before accepting any further frame. An empty QUEEN_SYNC_SECRET means open mode: the HELLO is still exchanged, to learn the peer’s server id, but not verified.

The last two fields are the v2 additions and ride the existing frame rather than a new tag: the broker’s advertised HTTP address (QUEEN_MESH_ADVERTISE_HOST plus the HTTP PORT) and its ephemeral-store epoch, which together admit the peer to the ephemeral queues’ rendezvous ring. A HELLO without the pair, an older build or half a pair, still authenticates exactly as before; the peer is only marked not ephemeral-capable, so during a rolling upgrade the stats’ eph_members count runs below servers_alive and that is the expected reading, not an error.

The security reality

State this plainly, because the mechanism above reads as stronger than it is.

• The handshake is replayable. The nonce is generated by the dialer and the listener never tracks nonces it has seen. A captured HELLO can be replayed by anyone who can reach the mesh port, and it will authenticate.
• Post-handshake frames are unauthenticated JSON. There is no per-frame MAC and no sequence number. Once a connection is accepted, everything on it is trusted.
• The frame set includes MAINTENANCE_MODE_SET. So an attacker who can reach the port and replay one captured handshake can put every replica into maintenance mode, which diverts all pushes to the disk spool. POP_MAINTENANCE_MODE_SET will likewise pause consumption cluster-wide.
• The frame set includes EPH_ADMIN. Ephemeral queues live in broker RAM with no PostgreSQL underneath, so for them the same attacker is not limited to availability: reset and delete destroy their contents on receipt.
• With no secret configured, not even the replay is needed.

Nothing on the mesh can corrupt or misdeliver a durable queue’s messages, because it carries no data-plane writes and PostgreSQL rejects nothing it says. For the durable engine the exposure is availability and operational control, not integrity; the RAM-class exception above is the one place the frame set reaches state PostgreSQL does not hold.

What it cannot do

• It is not a replication channel. No offset, cursor, lease or segment ever travels on it.
• It is not required for correctness. Turn it off (QUEEN_SYNC_ENABLED=0, or configure no peers) and a multi-replica cluster still works: cross-replica discovery falls back to the parked pop’s backoff and the reconcile loop’s 60-second re-read. You lose latency, not correctness.
• It is not ordered across peers. A frame is a hint that something changed; it never carries a version, and receipt is idempotent by construction.
• It has no C++ wire compatibility requirement. Rust-only clusters, which is why the wire is this simple. The one compatibility rule it does keep is cross-version tolerance during a rolling deploy: an unknown tag is skipped without closing the connection, and a new capability rides new fields on an existing frame (the tenant, the group, the HELLO v2 pair) whenever it can.

Observing it

Two internal routes return the transport’s own statistics: GET /internal/api/shared-state/stats and GET /internal/api/inter-instance/stats. Both emit the same body: the local server id, the transport name (tcp-mesh), the port, and per peer the host, port, whether an outbound connection is established and its resolved remote IP. Alongside that, how many peers are alive versus dead by the inbound-liveness threshold, how many of the live peers are in the ephemeral rendezvous ring (eph_members), frames sent, received and dropped, handshake failures, and the two maintenance flags. With no peers configured the body is {"enabled": false, "reason": "no_peers", ...}.

GET /api/v1/system/shared-state is a different, deliberately minimal route: it always reports enabled: false with the reason single_node_segments_broker plus the two maintenance flags. It is not the mesh view.

Two of those are the ones to watch. A rising messages_dropped means a peer’s bounded send queue is full: the peer is slow or down, and discovery on it has fallen back to backoff. A rising handshake failure count means something is dialling the port that does not share the secret, which is worth investigating before it is anything else.