The hot-list

A wildcard pop has to answer “which of this queue’s partitions might have work for my group?” before it can claim anything. Answering that in SQL means joining log_partitions against log_consumers and ordering randomly on every poll, a cost that scales with the partition count and is paid even when the queue is idle. The hot-list answers it from memory instead, and calls PostgreSQL only about the handful of partitions it names.

It is on by default (QUEEN_HOTLIST=1). Setting QUEEN_HOTLIST=0 reverts to the SQL candidate scan described in Life of a pop; every hook in the pop, push and ack paths becomes a single branch on the flag.

The correctness contract

PostgreSQL remains the sole authority on claiming, leases, cursors and visibility. The hot-list is allowed to be wrong in exactly one direction: stale in excess. A partition in the ring that has nothing costs one empty FOR UPDATE SKIP LOCKED claim, a single indexed row-lock attempt that returns no rows. A partition missing from the ring would be a real defect, and the design closes that hole with a periodic reseed floor rather than with a claim that the ring is always right.

Everything below is either a way to keep the ring cheap, or a way to make sure the floor catches what the ring lost.

Keying

Every ring is keyed on the composite "<tenant>\x1f<queue>", never a bare queue name. On a shared cell the names orders and workers collide across tenants, and a shared ring would mean one tenant’s pop checking out (and then clearing or wheeling) another tenant’s candidate. The same composite key is used by the long-poll wake gates, and it must be byte-identical between the two: a mismatch is a silently lost wake.

Three structures

Interning. Per queue, a partition name maps to a dense u32 index, append-only. The strings live only here; everything else is flat arrays indexed by that number. A second map remembers partition id ↔ index, learned from pop responses and reseeds, which is what lets the partition-id-keyed ack hook find its way back to a ring entry.

The ring. Per (queue, group), an intrusive doubly-linked “ready” list over flat arrays indexed by the partition index. Insertion is O(1) and only happens on an empty-to-pending transition; a push to an already-pending partition is an epoch increment with no ring operation at all. Each ring is sub-sharded by index % QUEEN_HOTLIST_SHARDS (default 8, inherited from QUEEN_V2_FUSION_SHARDS) so one hot mega-queue does not convoy on a single mutex, and each sub-ring’s arrays are indexed by the shard-local position index / S, so sub-sharding costs no wasted slots.

The wheel. Deferred visibility and lease revisits are held in a per-sub-ring binary min-heap keyed on revisit_at, with stale entries discarded lazily on drain.

Each partition slot in a sub-ring carries: a state byte, an epoch counter, a revisit timestamp, an accumulated mark count, a drained flag, and the two ring links. The state byte is one of:

Marks: how a partition becomes a candidate

A mark says “this partition just received data”. It comes from four places: a local push (quiet; see the wake tick below), a transaction commit or DLQ move (which wakes immediately), a peer’s mesh dirty hint (wakes immediately, never re-broadcast), and a reseed row (local discovery, no broadcast).

A mark is applied to every existing group ring of the queue, because a push makes the partition pending for every group already polling it. A group nobody polls has no ring and therefore no allocation; it will discover the partition through the reseed floor on first contact.

Every mark increments the slot’s epoch and adds the pushed frame count to batch_count. Then the state decides:

• IDLE with delayedProcessing > 0: schedule in the wheel at now + D + 300 ms. The padding acknowledges that the broker clock only schedules the retry. PostgreSQL still enforces the exact cut.
• IDLE with windowBuffer > 0: hold until first mark + W, unless batch_count has already reached QUEEN_HOTLIST_WINDOW_BATCH (default 100), in which case promote immediately. The effective rule is min(window elapsed, batch full).
• IDLE with no deferral: push to the ready tail.
• WHEEL on a windowBuffer queue: promote early if the batch has now fattened past the threshold. A delayedProcessing deadline is never shortened.
• READY: nothing to do.
• INFLIGHT: nothing but the epoch bump, which is exactly what makes the in-flight pop’s check-in re-append the entry instead of clearing it.

A queue with no group ring at all still counts as a transition for wake purposes. A partition-targeted long-poll parks on the queue gate without registering a ring, and gating the wake purely on a ring transition would strand a tenant whose only consumer is partition-targeted.

Serving: check out, ask SQL, check in

take_batch drains due wheel entries first, then takes up to k entries from the ready lists round-robin across sub-rings, marking each INFLIGHT and snapshotting its epoch. The pop path uses k = clamp(partitions × 8, 16, 256).

Those names go to queen.log_pop_list_v1(queue, group, names[], ...), which runs the same claim core as the wildcard pop (one queen.log_pop_v1 call per candidate) and returns, alongside the segments and blobs, a tri-state verdict per evaluated candidate:

The tri-state is what removes a second probe. The lease expiry is already on the consumer row the claim read, so SQL reports it in the same round trip rather than making the broker ask again.

Two details in the took handling are non-obvious:

A leased took goes to the wheel, not back to ready. The pop that just took frames holds a lease, so re-appending the partition would only make the next consumer probe it, get leased, and wheel it for the whole lease anyway. Parking it directly, and pulling it back early on our own ack, saves that round trip.

An auto-ack took on a windowBuffer queue is debounced rather than made ready. A fresh delivery means the partition just fired, so it is held for another window. Going straight to ready would re-probe immediately and deliver each subsequent message un-batched under continuous writes, diverging from the SQL quiet-period debounce the option promises.

Epoch-CAS: why a race cannot lose work

The empty verdict is the only one that removes a partition from tracking, and it is guarded:

if slot.epoch == verdict.epoch_snapshot { clear } else { re-append }

The pop snapshotted the epoch when it checked the candidate out. If a push committed and marked the partition in between, the epoch moved, the compare-and-swap fails, and the entry stays, because that push’s data is real and nobody has claimed it. Every ordering of the race degrades to a false positive.

On a deferral queue (delayedProcessing or windowBuffer set) an empty verdict never hard-clears at all: it schedules a bounded revisit instead, between 50 ms and 1000 ms.

Promote on ack

When a full-batch ack releases a lease, the ack path calls into the hot-list with the partition id and group. If the entry is still pending (pushes arrived during the lease) it is promoted to ready and the queue is woken, so the next batch is claimable immediately rather than at lease expiry.

This is also where the drained flag from the claim matters. A took verdict carries whether the claim exhausted the partition’s visible backlog, computed by comparing the served batch_end with the partition’s allocator watermark read at claim time. Only a drained claim may be cleared on ack. Using the mark count alone was measurably wrong: a thousand-message backlog consumed in hundred-message batches has no in-lease marks, yet nine hundred messages remain, and clearing the entry stalled every subsequent batch until the reseed floor fired.

The promote can still lose a race: a pop whose SQL snapshot saw the lease live may re-park the entry after the releasing ack already fired. That is why a lease park is capped at 1000 ms rather than parked for the full lease. Without the cap, a lost promote left a partition dark until the original lease expiry, up to leaseTime (a multi-minute single-partition stall, observed on the bench VM). With the cap the worst case is one empty re-probe per second per still-leased partition. Fast consumers rarely hit it: their ack’s promote wins the race.

Background ticks

The wake tick is why the push path marks quietly. Waking parked pops per push costs one notify_waiters (itself proportional to the number of parked consumers) for every single push. Marking sets one atomic flag instead, and the tick issues one wake per marked queue, so a hot queue goes from one wake per push to at most one per tick. It is purely a latency optimisation: a parked pop re-polls on its own backoff regardless, so a late or stopped tick adds bounded latency and can never strand a consumer.

The reseed floor

queen.log_hotlist_reseed_v1 enumerates a (queue, group)’s probably-pending partitions the same way the SQL candidate scan does (last_offset > committed or no consumer row), but keyset paginated in id order rather than ORDER BY random(), so the broker can walk a large set in bounded chunks. The handler walks pages of 10,000 with a cap of 200 pages, interning each name and remembering each id.

Three things about it are deliberate:

• It over-includes. Lease-held partitions are returned, because a lease-held partition is still pending; the ring must hold it and the tri-state will mark it leased.
• It reclaims stranded entries. An INFLIGHT entry whose pop was dropped between check-out and check-in (a client disconnect) cannot be re-linked by a mark or a promote, which only bump the epoch. The reseed is its last resort. (The serve path also arms a drop guard that re-appends checked-out candidates if the future is dropped, so this is the second line of defence.)
• It is jittered. Each (queue, group) gets a fixed random phase offset in the first 15 seconds, so groups that registered together at cold start never reseed as one synchronised keyset scan storm.

The periodic floor runs even when the ring is non-empty. That is the part that matters: it is what recovers a partition erroneously dropped from a busy ring (a cross-broker false clear, a stale-config hard clear, a missed mark, a dropped mesh hint) within one interval. The pop path also reseeds opportunistically when it finds the ring cold.

Memory, and the idle sweep

The rings are the one broker structure an untrusted client can grow by naming things. A tenant looping GET /api/v1/pop/queue/<random-name> would otherwise pin one interning table and one set of rings per distinct name for the life of the process.

Ring registration happens after the group-seeded check, which reads the committed consumer_groups_metadata marker, so the first pop of a name allocates no ring in memory. But that first pop is routed through the SQL wildcard procedure, and that procedure provisions: a pop on a queue that does not exist creates its queen.queues config row and the seed marker (a subscription may legitimately precede the first push, so the marker’s foreign key to the queue row forces the upsert; see Life of a pop). From the second pop on, the queue exists and the ring is registered. What actually bounds ring memory is therefore the idle sweep, which drops a queue’s entire state when three conditions hold simultaneously, all checked while holding the map lock:

1. A second-chance flag that every access sets and each sweep clears is false: nothing touched this queue for a full sweep interval.
2. The map is the only holder of the Arc, an exact test that nobody is mid-operation, since any concurrent operation cloned the handle out first.
3. Every ring entry is IDLE: no ready candidate, no deferral, no lease park, no in-flight claim.

A queue is therefore dropped somewhere between one and two sweep intervals after its last use. Evicting is never a correctness event even if all three checks were somehow wrong, because the ring is a cache and the reseed floor rebuilds it on first contact. QUEEN_HOTLIST_IDLE_SWEEP_MS=0 disables the sweep, which means unbounded growth, only sensible for a single-tenant deployment that wants the pre-eviction behaviour.

Consumer-group deletion drops rings explicitly rather than waiting for the sweep: deleting a group for one queue drops that queue’s ring for the group, and deleting it across all queues drops it everywhere for that tenant only. On a shared cell workers is a universal group name, and one tenant’s delete must not cold-start every other tenant’s ring. Over-forgetting is a harmless cold start; a surviving stale ring that hides the re-consume until the next floor is the real danger.

Observing it

The hotlist log target emits a reseed floor line about every 30 seconds with the reseed delta and rate, the number of live rings, total ready and wheel entries, and the local and remote mark counters. It then emits one ring line per busiest non-empty ring, ranked by ready + wheel and limited to QUEEN_LOG_TOPN_QUEUES (default 10). Ranking matters: an earlier version walked a hash map and showed an arbitrary, unstable subset.

For latency debugging, QUEEN_HOTLIST_TRACE=<queue-prefix> emits bounded mark, wake and serve breadcrumbs for matching queues. It is off by default and costs one Option check when off.

Interaction with the mesh

With peers configured, a local mark also enqueues a coalesced (queue, partition) dirty hint, flushed to peers as a batched frame. The enqueue happens on every local push, not only on a local ring transition, because the pushing broker usually has no consumer for the peers’ groups (so no local ring and no transition), and the peers are exactly who need the hint. The coalescing set deduplicates, so a partition pushed a thousand times in one flush window costs one hint. The set is bounded at 200,000 entries and drops beyond that; the loss is healed by the reseed floor. Received marks never re-broadcast. With no peers configured the whole mechanism is skipped, so a single broker pays neither the lock traffic nor the allocation.