The hot-list

A wildcard pop has to answer “which of this queue’s partitions might have work for my group?” before it can claim anything. Answering that in SQL means joining log_partitions against log_consumers and ordering randomly on every poll, a cost that scales with the partition count and is paid even when the queue is idle. The hot-list answers it from memory instead, and calls PostgreSQL only about the handful of partitions it names.

It is on by default (QUEEN_HOTLIST=1). Setting QUEEN_HOTLIST=0 reverts to the SQL candidate scan described in Life of a pop; every hook in the pop, push and ack paths becomes a single branch on the flag.

The correctness contract

PostgreSQL remains the sole authority on claiming, leases, cursors and visibility. The hot-list is allowed to be wrong in exactly one direction: stale in excess. A partition in the ring that has nothing adds a row to the batched FOR UPDATE SKIP LOCKED claim and a row to the fresh re-check that follows it, indexed row-lock work that returns no segments. A partition missing from the ring would be a real defect, and the design closes that hole with a periodic reseed floor rather than with a claim that the ring is always right.

Everything below is either a way to keep the ring cheap, or a way to make sure the floor catches what the ring lost.

Keying

Every ring is keyed on the composite "<tenant>\x1f<queue>", never a bare queue name. On a shared cell the names orders and workers collide across tenants, and a shared ring would mean one tenant’s pop checking out (and then clearing or wheeling) another tenant’s candidate. The same composite key is used by the long-poll wake gates, and it must be byte-identical between the two: a mismatch is a silently lost wake.

Three structures

Interning. Per queue, a partition name maps to a dense u32 index, append-only. The strings live only here; everything else is flat arrays indexed by that number. A second map remembers partition id ↔ index, learned from pop responses and reseeds, which is what lets the partition-id-keyed ack hook find its way back to a ring entry.

The ring. Per (queue, group), an intrusive doubly-linked “ready” list over flat arrays indexed by the partition index. Insertion is O(1) and only happens on an empty-to-pending transition; a push to an already-pending partition is an epoch increment with no ring operation at all. Each ring is sub-sharded by index % QUEEN_HOTLIST_SHARDS (default 8, inherited from QUEEN_V2_FUSION_SHARDS) so one hot mega-queue does not convoy on a single mutex, and each sub-ring’s arrays are indexed by the shard-local position index / S, so sub-sharding costs no wasted slots.

The wheel. Deferred visibility and lease revisits are held in a per-sub-ring binary min-heap keyed on revisit_at, with stale entries discarded lazily on drain.

Each partition slot in a sub-ring carries: a state byte, an epoch counter, a revisit timestamp, an accumulated mark count, a drained flag, and the two ring links. The state byte is one of:

Marks: how a partition becomes a candidate

A mark says “this partition just received data”. It comes from four places: a local write that landed frames (quiet; see the wake tick below), a transaction commit or DLQ move (which wakes immediately), a peer’s mesh dirty hint (wakes immediately, never re-broadcast), and a reseed row (local discovery, no broadcast).

The first of those is one function, shared by every path that lands frames in the log: the push handler, the streams cycle’s sink emit, the spool drain’s replay after a database outage, and the sweeper’s timer fire, for every segment the fire procedure reports as fired. A fire marks exactly like a push: the partition goes pending on every group ring of its queue, the local wake is coalesced into the tick, and peers get the batched message-available frame. That matters because these routes consult the ring, not the log: a partition no ring was told about is invisible until the reseed floor, which is how a fired timer reached its consumer about 30 s late from 1.0.3 through 1.5.1.

A mark is applied to every existing group ring of the queue, because a push makes the partition pending for every group already polling it. A group nobody polls has no ring and therefore no allocation; it will discover the partition through the reseed floor on first contact.

One shape of mark is narrower than that. A backward seek or a consumer-group delete makes partitions pending for one group, so those paths mark that group’s ring alone and the dirty hint they enqueue carries the group name with it. Marking queue-wide would arm every other group’s ring for nothing, which on the production queue measured below is 9563 entries per uninvolved group. The group is an optional field on the existing frame, so a peer running an older build ignores it and does the queue-wide mark it has always done: a superset, and therefore safe during a rolling upgrade.

Every mark increments the slot’s epoch and adds the pushed frame count to batch_count. Then the state decides:

• IDLE with delayedProcessing > 0: schedule in the wheel at now + D + 300 ms. The padding acknowledges that the broker clock only schedules the retry. PostgreSQL still enforces the exact cut.
• IDLE with windowBuffer > 0: hold until first mark + W, unless batch_count has already reached QUEEN_HOTLIST_WINDOW_BATCH (default 100), in which case promote immediately. The effective rule is min(window elapsed, batch full).
• IDLE with no deferral: push to the ready tail.
• WHEEL on a windowBuffer queue: promote early if the batch has now fattened past the threshold. A delayedProcessing deadline is never shortened.
• READY: nothing to do.
• INFLIGHT: nothing but the epoch bump, which is exactly what makes the in-flight pop’s check-in re-append the entry instead of clearing it.

A queue with no group ring at all still counts as a transition for wake purposes. A partition-targeted long-poll parks on the queue gate without registering a ring, and gating the wake purely on a ring transition would strand a tenant whose only consumer is partition-targeted.

Serving: check out, ask SQL, check in

take_batch drains due wheel entries first, then takes up to k entries from the ready lists round-robin across sub-rings, marking each INFLIGHT and snapshotting its epoch. On the pop path k = clamp(partitions × 2, 2, 64), and the take stops earlier as soon as the marks on the entries it has already claimed cover the caller’s batch. (The fused claim path takes the SQL serve cap 1:1 instead, which on the default partitions=1 is the same 2.) A pop that opts in with autopilot=true has partitions and batch chosen server-side before this formula, steered on two ring facts read in one pass: how many partitions are ready, and how long the oldest of them has waited. Checkout width is what bounds how many pops can serve one ring at once: every claimed entry is held INFLIGHT for the whole round trip, and that hold spans the client’s work and its ack, not just the SQL. A deep partition is therefore claimed alone, while a sparse ring still claims up to k.

The fixed over-claim this replaced (partitions × 8, floor 16) held 16 to 80 partitions per serve and capped a 100-partition ring at about six concurrent serves. Raising the ceiling back to 512 after the claim went batched measured 4x to 13x worse on a sparse ring of 1000 partitions taking about one message per second each on an 8-core cell: p50 21.8 s at k 512 with batch 512, against about 1.5 s at the default.

Those names go to queen.log_pop_list_v1(queue, group, names[], ...), which claims every candidate in one batched pass: one FOR UPDATE ... SKIP LOCKED claim over all of them, one no-lock probe of whatever the claim did not take, one segment-metadata read that never selects the blob column, one lease UPDATE, and one blob fetch for the chosen ranges. About six statements in total, independent of how many candidates it was given. Only a partition on first contact, with no cursor row yet, still routes through queen.log_pop_v1, because that is where subscription seeding and the advisory-guarded row creation live.

The per-partition shape it replaced cost about six statements per candidate. On the sparse shape it was measured on, 1000 partitions holding about one message each, that was 3.42 ms per pop of which 0.17 ms was data work, and it pinned the broker at about 1.2k pops/s with the box idle.

The call returns, alongside the segments and blobs, a tri-state verdict per evaluated candidate:

The tri-state is what removes a second probe. The lease expiry is already on the consumer row the claim read, so SQL reports it in the same round trip rather than making the broker ask again.

Two details in the took handling are non-obvious:

A leased took goes to the wheel, not back to ready. The pop that just took frames holds a lease, so re-appending the partition would only make the next consumer probe it, get leased, and wheel it for the whole lease anyway. Parking it directly, and pulling it back early on our own ack, saves that round trip.

An auto-ack took on a windowBuffer queue is debounced rather than made ready. A fresh delivery means the partition just fired, so it is held for another window. Going straight to ready would re-probe immediately and deliver each subsequent message un-batched under continuous writes, diverging from the SQL quiet-period debounce the option promises.

Epoch-CAS: why a race cannot lose work

The empty verdict is the only one that removes a partition from tracking, and it is guarded:

if slot.epoch == verdict.epoch_snapshot { clear } else { re-append }

The pop snapshotted the epoch when it checked the candidate out. If a push committed and marked the partition in between, the epoch moved, the compare-and-swap fails, and the entry stays, because that push’s data is real and nobody has claimed it. Every ordering of the race degrades to a false positive.

The deferral deadline

On a deferral queue (delayedProcessing or windowBuffer set) an empty verdict is ambiguous in a way it is not anywhere else: the partition may hold nothing, or it may hold something the visibility cut still hides. So it schedules a bounded revisit, between 50 ms and 1000 ms, rather than clearing.

That ambiguity has a deadline, and the revisit is bounded by it. A partition asserted pending at T can hide data no later than T plus the deferral cut, so once that much time has passed with no new mark, an empty claim means what it means on a plain queue and the entry clears. Every mark and every reseed row slides the instant forward, so a partition still being written is never called empty. The grace added on top of the cut is two clock pads plus one full revisit ceiling, which guarantees at least one confirming probe on either queue shape, so the broker has to be about 1.6 seconds out from PostgreSQL before a single skewed probe could clear anything.

The deadline exists because an unconditional revisit made the deferral state a one-way door: nothing ever returned an armed entry to IDLE, so it cycled through the wheel forever and the idle sweep, which requires every entry idle, could never reclaim the queue. Clearing on the deadline is not a correctness event: a partition whose data a window is still hiding was written moments ago, so the windowed reseed re-adds it.

Promote on ack

When a full-batch ack releases a lease, the ack path calls into the hot-list with the partition id and group. If the entry is still pending (pushes arrived during the lease) it is promoted to ready and the queue is woken, so the next batch is claimable immediately rather than at lease expiry.

This is also where the drained flag from the claim matters. A took verdict carries whether the claim exhausted the partition’s visible backlog, computed by comparing the served batch_end with the partition’s allocator watermark read at claim time. Only a drained claim may be cleared on ack; the mark count alone cannot decide, because a thousand-message backlog consumed in hundred-message batches has no in-lease marks, yet nine hundred messages remain.

The promote can still lose a race: a pop whose SQL snapshot saw the lease live may re-park the entry after the releasing ack already fired. That is why a lease park is capped at 1000 ms rather than parked for the full lease: a lost promote costs one empty re-probe per second, not a partition dark until the original lease expiry. Fast consumers rarely hit it: their ack’s promote wins the race.

Background ticks

The reseed floor comes in two shapes because the two jobs it does have very different costs. Nearly all of its work is discovering data a push created, and a push always stamps last_write_at, so the scan can be bounded to the partitions written recently and still see everything a push can produce. That bound is what the floor runs on its ordinary interval. Measured on a production queue with 9563 partitions the difference is 49 ms against 0.375 ms, and 39700 buffers against 160, because the cost of the walk is not reading the partitions but probing log_consumers once per partition.

What the bounded scan cannot see is pendingness that no write explains: a ring entry cleared in error, a claim stranded by a dropped pop, a lease park left behind by a lost promote, a mesh hint dropped past its bound, a cursor moved backwards by a seek on another broker. Those are repair cases, and the full walk is what recovers them, which is why its interval is really an answer to “how long may a repair stay hidden”. Setting QUEEN_HOTLIST_RESEED_FULL_MS to 0 makes every reseed a full walk again, and reverts with it the repair machinery the windowing made necessary (below).

QUEEN_HOTLIST_RESEED_WINDOW_MS is clamped rather than taken literally, whether it was derived or set by hand. The floor is one reseed interval plus that ring’s widest de-phasing offset, which is the longest gap two consecutive passes of one ring can leave: a window narrower than the gap leaves a band of writes that neither pass covers, which is the exact invariant the window exists to preserve. The ceiling is one week, because a lookback wider than the partition set is a full walk run through the index built for a narrow one, and the way to ask for that is QUEEN_HOTLIST_RESEED_FULL_MS=0. A value the broker had to move is named in a boot warning with the value actually in force.

The wake tick is why the push path marks quietly. Waking parked pops per push costs one notify_waiters (itself proportional to the number of parked consumers) for every single push. Marking sets one atomic flag instead, and the tick issues one wake per marked queue, so a hot queue goes from one wake per push to at most one per tick. It is purely a latency optimisation: a parked pop re-polls on its own backoff regardless, so a late or stopped tick adds bounded latency and can never strand a consumer.

The reseed floor

queen.log_hotlist_reseed_window_v1 enumerates a (queue, group)’s probably-pending partitions the same way the SQL candidate scan does (last_offset > committed or no consumer row), but keyset paginated in (last_write_at, id) order rather than ORDER BY random(), so the broker can walk a large set in bounded chunks. It is the one statement both reseed shapes run: a windowed pass binds the lookback, and a full walk binds the cutoff to -infinity up front, so the window is never read. (The dedicated full-walk statement, log_hotlist_reseed_v1, survives in the SQL as the test suite’s oracle, but the broker no longer calls it: under the generic plan prepare_cached converges to, its id-ordered keyset walk read every partition in the cell per ring rather than the queue’s own.) The handler walks pages of 10,000 with a cap of 200 pages, interning each name and remembering each id.

Three things about it are deliberate:

• It over-includes. Lease-held partitions are returned, because a lease-held partition is still pending; the ring must hold it and the tri-state will mark it leased.
• It reclaims stranded entries. An INFLIGHT entry whose pop was dropped between check-out and check-in (a client disconnect) cannot be re-linked by a mark or a promote, which only bump the epoch. The reseed is its last resort. (The serve path also arms a drop guard that re-appends checked-out candidates if the future is dropped, so this is the second line of defence.)
• It is jittered in proportion to what it de-phases. Each (queue, group) gets a fixed random phase, spread over a fifth of the interval or 15 seconds, whichever is larger: 15 seconds for the 30-second cadence, 60 seconds for the 300-second full walk. Proportional rather than flat, so rings that cold-started together never come due as one synchronised keyset scan storm on either cadence.
• A walk holds a ticket on the ring it started on. The ticket carries the ring identity rather than the (queue, group) name, so a completion stamp whose ring was forgotten and recreated mid-walk is dropped and counted, and the recreated ring is not credited with a cold-start full walk it never had.
• The cutoff is pinned for the whole walk. Each page is a separate statement, so a per-page now() - window would let the lower bound creep forward while the keyset cursor climbs, skipping partitions written into the band between the two. The first windowed page derives the cutoff and returns it, later pages echo it back exactly as they echo the keyset, and the broker never holds a clock of the database’s. A full walk starts with the cutoff already pinned to -infinity.

The periodic floor runs even when the ring is non-empty. That is the part that matters: it is what recovers a partition erroneously dropped from a busy ring (a cross-broker false clear, a stale-config hard clear, a missed mark, a dropped mesh hint) within one interval. The pop path also reseeds opportunistically when it finds the ring cold.

When a cursor moves backwards

Two operations make old partitions pending without writing anything: a backward seek, and a consumer-group delete (which removes the group’s cursor rows, so every partition holding data reads as uncommitted again). Neither moves last_write_at, so the windowed pass is blind to both by construction. Three mechanisms cover them, in decreasing order of speed.

The broker that serves the operation repairs itself immediately. A queue-wide seek or a group delete runs a full walk on the spot for the affected (queue, group) rings. A seek scoped to one partition marks that one partition instead, rather than walking and broadcasting the whole queue because one cursor moved.

The peers are told over the mesh, from the same walk: its rows go out as ordinary dirty hints, carrying the group they belong to, so a peer re-adds exactly those partitions rather than waiting for its own next full walk.

A durable marker underneath, because the mesh drops frames by design when a peer is slow or down, and a dropped cursor-move hint would cost a replay a full-walk interval. So both operations also write a row into queen.hotlist_repairs inside their own transaction, keyed (tenant, queue, group) and naming a partition when the repair is that narrow. Every broker reads the table on the reconcile cadence (QUEEN_CACHE_REFRESH_INTERVAL_MS, 60 s), acts on what it has not acted on already, and marks the named partition or owes a full walk for a queue-wide row. Rows are pruned after an hour, which is longer than any broker can be absent and still hold a stamped ring: one that was offline longer cold-starts every ring it has.

The reader compares each row’s (timestamp, partition) against what it last applied and acts on any change, rather than tracking a high-water mark. A watermark would be wrong in a way that fails silently: now() is the transaction’s start time, so a long-running group delete can commit a row dated before one a later, faster seek already pushed the watermark past, and that repair would never be read at all.

This whole layer exists only because the windowed pass cannot see a cursor move, so QUEEN_HOTLIST_RESEED_FULL_MS=0 switches all of it off together with the windowing: every pass is a full walk at the 30-second cadence again, the fan-out and the delete’s repair walk do not run, and the marker poll is skipped. The markers are still written, so a mixed cluster keeps working. A single-partition seek still sends its one hint, which is a push-shaped event rather than a fan-out.

A repair walk that fails is reported rather than swallowed. The cursor move itself committed, so the seek still answers 200, but the body carries hotlistRepaired: false and a warning saying that redelivery resumes at the next full reseed, and the broker logs the failure with the queue and group, so an operator’s “replay from yesterday” is never an unqualified success that replays nothing.

Memory, and the idle sweep

The rings are the one broker structure an untrusted client can grow by naming things. A tenant looping GET /api/v1/pop/queue/<random-name> would otherwise pin one interning table and one set of rings per distinct name for the life of the process.

Ring registration happens after the group-seeded check, which reads the committed consumer_groups_metadata marker, so the first pop of a name allocates no ring in memory. But that first pop is routed through the SQL wildcard procedure, and that procedure provisions: a pop on a queue that does not exist creates its queen.queues config row and the seed marker (a subscription may legitimately precede the first push, so the marker’s foreign key to the queue row forces the upsert; see Life of a pop). From the second pop on, the queue exists and the ring is registered. What actually bounds ring memory is therefore the idle sweep, which drops a queue’s entire state when three conditions hold simultaneously, all checked while holding the map lock:

1. A second-chance flag that every access sets and each sweep clears is false: nothing touched this queue for a full sweep interval.
2. The map is the only holder of the Arc, an exact test that nobody is mid-operation, since any concurrent operation cloned the handle out first.
3. Every ring entry is IDLE: no ready candidate, no deferral, no lease park, no in-flight claim.

A queue is therefore dropped somewhere between one and two sweep intervals after its last use. Evicting is never a correctness event even if all three checks were somehow wrong, because the ring is a cache and the reseed floor rebuilds it on first contact. QUEEN_HOTLIST_IDLE_SWEEP_MS=0 disables the sweep, which means unbounded growth, only sensible for a single-tenant deployment that wants the pre-eviction behaviour.

The idle sweep cannot bound one shape: a broker whose rings fill but never drain. Marks arrive on paths independent of the broker’s own consumers, the reseed floor and peers’ mesh hints, so the passive half of an active/passive pair accumulates the union of every partition ever momentarily pending, and those entries are READY, which condition 3 never allows past. A second, faster loop covers it: every QUEEN_HOTLIST_UNSERVED_TRIM_MS (30 s; 0 disables) the broker drops any queue state that went one to two intervals without a single served pop, and on glibc hands the freed pages back to the operating system rather than only to the allocator. The discriminator is served pops, never entry age: the ready list is FIFO, so a healthy but deeply backlogged ring legitimately holds a very old head, and an age rule would discard real work exactly when the cell is furthest behind, while what separates a standby is that nothing ever leaves its ring. A claim, an ack’s promote and a parked pop’s ready probe (re-run every backoff interval) all count as service; an INFLIGHT entry or a concurrently held handle still blocks the drop; and a trimmed ring rebuilds through the reseed floor like any other cold start. A consumer that exists but has stopped popping is trimmed with its backlog still real, deliberately: reversing that costs one cold-start walk on its next pop.

Consumer-group deletion drops rings explicitly rather than waiting for the sweep: deleting a group for one queue drops that queue’s ring for the group, and deleting it across all queues drops it everywhere for that tenant only. On a shared cell workers is a universal group name, and one tenant’s delete must not cold-start every other tenant’s ring. Over-forgetting is a harmless cold start; a surviving stale ring that hides the re-consume until the next floor is the real danger.

Observing it

The hotlist log target emits a reseed floor line about every 30 seconds with the reseed delta and rate, the number of live rings, total ready and wheel entries, and the local and remote mark counters. It then emits one ring line per ring, ranked by ready + wheel and limited to QUEEN_LOG_TOPN_QUEUES (default 10).

The two reseed modes are counted separately, because they differ by about 130x in database cost and one summed number answers neither “what is the reseed costing” nor “is this ring windowed or full”. reseeds_delta and per_s keep their old names and their old meaning, the sum, so existing dashboards survive; full_delta, window_delta and full_per_s split it. failed_delta counts walks that ended on a database error and dropped_delta walks whose ring was replaced underneath them. full_overdue counts rings whose last full walk is older than twice the interval it should be running at, which includes rings that have never had one.

Each ring line carries next (the mode its next walk will run, the answer rather than the arithmetic), full_age_ms and reseed_age_ms, with -1 meaning never. A ring that is overdue for its full walk prints even when it holds nothing, which is deliberate: a ring pinned to full mode by a walk that keeps failing is empty, so a busiest-first filter would hide it exactly when it matters. That same failure also warns on its own, at three consecutive failures and then on each doubling, per ring: a database refusing this query refuses it for every ring, and one line per ring per pass buries the cause it is reporting.

For latency debugging, QUEEN_HOTLIST_TRACE=<queue-prefix> emits bounded mark, wake and serve breadcrumbs for matching queues. It is off by default and costs one Option check when off.

Interaction with the mesh

With peers configured, a local mark also enqueues a coalesced (queue, partition, group?) dirty hint, flushed to peers as a batched frame. The enqueue happens on every local push, not only on a local ring transition, because the pushing broker usually has no consumer for the peers’ groups (so no local ring and no transition), and the peers are exactly who need the hint. The coalescing set deduplicates, so a partition pushed a thousand times in one flush window costs one hint; the group is part of the identity, so an ungrouped hint and a grouped one for the same partition are both kept and both applied, and the ungrouped one is a superset of the other. The set is bounded at 200,000 entries and drops beyond that. Received marks never re-broadcast. With no peers configured the whole mechanism is skipped, so a single broker pays neither the lock traffic nor the allocation.

A dropped hint for a push is healed by the reseed floor, which is what the window covers by construction. A dropped hint for a cursor move is not, and is covered by the durable repair marker instead.