Flow control

The broker does not let requests reach PostgreSQL as fast as they arrive. Every write transaction on the hot paths holds a slot from one process-wide admission arbiter, and the number of slots moves based on what the broker observes about PostgreSQL’s commit pipeline.

The reason to have it at all is that a connection pool is a poor admission controller. A pool of 160 connections will happily put 160 statements into PostgreSQL, and if PostgreSQL can only usefully run twenty of them the other 140 are lock contention, SKIP LOCKED racing and fatter round trips: work that makes throughput worse while every gauge still looks unsaturated.

Why the previous design was replaced

Until 1.0.0-beta.2 the broker used two Vegas-style limiters, one for push and one for pop, each inferring queueing from per-operation round-trip time. That design is sound on a network link and wrong on this path, for a reason worth stating plainly because it generalises.

A delay-based controller estimates queueing as the ratio of current round-trip time to the smallest one it has seen. On a WAL-bound commit path most of the excess over that minimum is not queueing at all: it is the group-commit flush wait, an intrinsic cost that admission cannot remove. The estimator counted a fixed floor as congestion and backed off from it.

Two consequences followed. The controller’s dead band widened at low limits (the grow and shrink thresholds were absolute, so the band alpha < queue < beta covers a larger RTT range the smaller the limit is), which made low limits an attractor it could not climb out of. And because the limiter sat below what the workload needed, the broker left cores idle while the pool reported no waiters at all: the controller and the ground truth disagreed, and nothing exported the controller’s inputs, so the disagreement was invisible.

The replacement inverts the topology. One owner for admission instead of several self-balancing loops, and a sensor that measures the shared resource directly instead of inferring it from the latency of individual operations.

The sensor: commit trains

Group commit makes write completions cluster in time. One flush acknowledgement releases every transaction that boarded it, so the broker’s own commit completions arrive in bursts. Clustering them measures the flush pipeline with no PostgreSQL-side telemetry at all, which means it works on any managed PostgreSQL.

Completions closer together than QUEEN_ADMISSION_TRAIN_GAP_US (300 by default) belong to the same train. From the train sequence the arbiter derives:

Queueing, the thing admission can actually remove, shows up as in-flight slots that have waited several cycles. That is the control error.

The control law

One AIMD step per tick (QUEEN_ADMISSION_TICK_MS, 500 by default) over a total budget B of concurrently admitted write transactions:

grow    when B was saturated during the tick
shrink  when the oldest admitted live-lane slot has waited several flush cycles
hold    otherwise

B is clamped to [QUEEN_ADMISSION_MIN, QUEEN_ADMISSION_MAX] and never rises above DB_POOL_SIZE - QUEEN_ADMISSION_POOL_RESERVE. Admitted work must never starve on the connection pool, and the reserve keeps rare unmetered writes (admin operations) from deadlocking behind metered ones.

One thing the law deliberately does not do is shrink on a widening flush cycle. A bigger budget grows the trains, and every flush then carries more, so the cycle widens exactly when amortisation is working. An earlier build shrank on that signal and pinned the budget below the workload: the same category of mistake the Vegas estimator made with per-operation round-trip time.

Four lanes

Push, Pop, Ack and Maint each have their own slot accounting inside the shared budget. A grant is work conserving: any lane may use any free slot below B.

The guarantees act on the wake order when B is exhausted. Released slots go first to lanes sitting below their guaranteed share, in the order Ack, Pop, Push, Maint. Acks come first because they unblock lanes and shorten everyone’s lap; maintenance can always wait.

The share is a wake ordering, not a slot reservation and not a floor on a lane’s own concurrency cap. A lane’s optimum can sit well below its share, and pinning its cap at the share would force exactly the contention the arbiter exists to avoid.

Each lane also carries its own concurrency cap inside B, moved by a per-lane controller that probes one step up (10% of the cap, at least one slot) when the lane presses its cap. Some paths have negative returns to concurrency, and this is what finds each lane’s operating point without hardcoding it. What the controller judges the probe on is not the same in every lane.

Push, Ack and Maint judge a probe on marginal throughput: the raise survives only if the lane’s completion rate grew by at least 5%.

The Pop lane does not, because throughput is the wrong objective for a delivery path. Its objective is the measured age of the oldest servable but unvisited entry in the hot-list ready ring, sampled from each shard’s FIFO head so a lane that is never visited still reports. Age-at-visit sampling has survivorship bias: the starved lane is the one that never samples.

Because ready age is roughly ring depth divided by visit rate, judging it alone would blame the probe for an arrival surge, so the verdict reads all three terms.

The last row is the one to read twice: inside a struggling system the default is to take capacity out, never to keep it. A reverted probe also holds the lane still for six ticks, 3 seconds on the default cadence, before it may probe again.

The global budget deliberately watches a different variable, the oldest admitted slot’s wait. Two controllers chasing the same number is how the coupled loops of the previous design were built.

Slots are RAII

A slot releases when it drops. There is no code path that can leak in-flight accounting, which is a class of bug the previous limiter had at eight call sites: several paths took a permit and returned early without recording it, so its in-flight counter drifted upward until the controller’s own anti-ramp guard was permanently disabled.

Feeding the sensor is separate and explicit. Slot::commit_done(rtt) marks a real commit completion; a slot dropped without it (a pool failure, an empty take) releases cleanly and contributes no fake train sample.

How it interacts with the connection pool

The slot and the pooled connection are two separate resources acquired in a fixed order: slot first, then connection. The reverse order deadlocks, with slot holders parked in pool.get() while connection holders park waiting for a slot.

Three ordering rules exist because getting them wrong produced real regressions:

A parked pop holds neither. A long poll releases the slot and returns the connection before it parks, and re-acquires both on wake. Otherwise every parked consumer would pin a connection, and the pool would be exhausted by consumers doing nothing.

A cheap in-memory check comes before the slot. On the hot-list serve path, a pop whose ring has nothing ready returns empty without touching either resource. The reason is a starvation inversion: the rate of empty re-polls is proportional to the number of parked consumers, so taking the shared pop slot on every empty re-poll saturated admission, and a freshly woken real delivery then queued behind thousands of empty polls. The legacy path achieves the same thing with the cheap indexed log_has_pending_v1 probe.

The minimum-pop-wait hold happens before the slot too. Holding an under-full claim back is done in Rust, before either resource is taken. A pg_sleep inside the pop procedure would hold a connection, a slot, a PostgreSQL backend, and (after the first partition claim) row locks, for the whole window.

One knob moves the slot boundary itself. With QUEEN_POP_FUSION set (off by default), N pop claim legs share one transaction, and the Pop lane takes one slot per fused flush rather than one per claim. What the sensor then measures is the fused transaction, which is the thing that actually queues on PostgreSQL, instead of N copies of the same WAL wait. The claim leg it fuses is described in life of a pop.

Background jobs run in the Maint lane rather than bypassing admission, which is what lets the wake order keep them behind request traffic without starving them outright. The sweeper is one of them: a timer fire is a writing transaction holding partition locks, and it takes a Maint slot before its connection like every other job here.

The one path that has no lane

KV state is the exception, and it is deliberate on both halves. Neither KV reads nor KV writes take a lane. They have their own connection pool, QUEEN_KV_POOL_SIZE, derived as clamp(DB_POOL_SIZE / 10, 4, 32), and that pool is the semaphore.

Writes were the tempting half to put on the Push lane, and doing so would have been a defect. Thirty tenants each inside their own limit of a hundred writes a second are three thousand Push slots a second, on a small stack whose measured commit-bound ceiling is around 480 messages a second. No tenant would have violated anything, the message path would be starved, and the weigher could not tell the two kinds of work apart because they would be the same lane.

What the dedicated pool gives that a lane would not is the shape of its failure. At roughly 1 ms a read it is worth about 16,000 reads a second, so it never binds in normal operation; when the database slows to 100 ms a read it is worth 160 a second, and the excess takes a 503 instead of taking connections away from the log. queen_kv_pool exports its size, availability and waiters.

The exception to the exception: a KV write that rides inside a push or an ack inherits the slot the handler already took and does not take another. That is a bundle of messages, and it belongs in the message lane.

The second control loop, and why it is not a second arbiter

This page opens by replacing several self-balancing loops with one owner, so a second feedback loop in the same broker needs accounting for. Pop autopilot is one, and it does not compete with the arbiter because the two act on different quantities.

A wider sweep is not more slots. It is more partitions inside the same claim, which is the same one slot for the Pop lane either way, so the controller cannot spend the arbiter’s budget however it steers. What it can spend is the value of a slot, and that is what its two failure directions look like. Too wide and the claim holds its slot while it visits partitions that had nothing, which is the speculative scan the controller exists to prevent. Too narrow and ready partitions are left behind, their ready-age grows, and that growth is precisely the signal the width loop steers on.

The message budget is a constant rather than a loop for a related reason, and it is the one worth knowing before raising it. A claimed entry is held INFLIGHT for the whole round trip, and that hold spans the consumer’s own work and its ack, not just the SQL. So a deep claim on a backlogged partition stays inflight for as long as the client takes to drain it serially, and the tail a consumer sees grows with the depth it asked for. QUEEN_POP_AUTOPILOT_BATCH therefore ships at 100, below the 200 an absent batch resolves to in the handler, and a drain-aware successor that sizes it from the observed ack rate is the open item behind that constant.

Degraded signal

With synchronous_commit = off, or on a device faster than the clustering gap, commit waits collapse and the trains carry no information. The arbiter detects this (the tenth-percentile commit round trip falls under an observability floor, held for three consecutive ticks) and pins the budget to QUEEN_ADMISSION_NOSYNC_BUDGET instead of chasing noise. The mode is visible in telemetry as adm_mode.

What is exported

On /metrics/prometheus:

queen_admission_budget
queen_admission_inflight{lane="push|pop|ack|maint"}
queen_admission_waiting{lane="push|pop|ack|maint"}
queen_admission_trains_per_s
queen_admission_txn_per_train
queen_admission_cycle_ms

The 10 second broker rates log line carries the same picture in one row: adm_budget, adm_mode, adm_lanes as push:in/cap w<waiters> per lane, trains_s, cycle_ms, oldest_wait_ms and adm_last.

adm_last is why the budget last moved, and it is the field that turns a flat budget into a diagnosis. It reads init until the first change, then saturated (the budget grew because B was exhausted), oldest-age (it shrank on queueing), or nosync-detected and signal-restored for the two mode transitions. saturated and oldest-age are written only when the budget actually changes by at least one slot, so a steady budget leaves the previous reason standing rather than reporting the ticks that decided to hold.

QUEEN_ADMISSION_TRACE=1, off by default, adds a per-decision line on the admission target that none of the above carries: budget adjusted with the reason, the previous budget, the flush cycle, the oldest wait and transactions per train, plus lane probe reverted with the lane, its new cap and the verdict reason, and lane probe kept for the Pop lane. It logs on every tick that changes something, which on a busy broker is most of them, so it is a debugging knob rather than a setting to leave on.

Tuning it

The short version: read the telemetry before changing anything, and prefer changing nothing.

• The floor and the initial budget derive from the pool. Two thirds of DB_POOL_SIZE - QUEEN_ADMISSION_POOL_RESERVE, which is 96 on the defaults. They are derived rather than fixed so that a small deployment can never admit more concurrent transactions than it has connections for.
• waiting is the signal that admission is binding. If a lane shows waiters while the pool reports idle connections, the arbiter is the constraint, not PostgreSQL.
• Raising QUEEN_ADMISSION_MAX cannot help unless the budget is sitting at it. The maximum is a safety ceiling, not an operating point.
• The Vegas-era variables are gone. QUEEN_SEG_{PUSH,POP}_{INIT,MIN,MAX}, QUEEN_VEGAS_ALPHA and QUEEN_VEGAS_BETA are no longer read. The broker logs a warning at boot if it finds any of them set, rather than letting you tune a control loop that is not there.

The generated configuration reference lists every one of these variables with the default the code applies.