The storage model

A message’s address in Queen is one number: a BIGINT offset, monotone within its partition. Everything in the storage layer follows from that choice. There is no per-message row, no per-message state, and no compound cursor. A consumer group’s progress in a partition is one integer; the difference between two integers is a message count; whether one message precedes another is a comparison.

The data itself is not stored per message either. Messages are packed into segments: one queen.log_segments row holds many messages as length-prefixed frames, compressed together with zstd, covering an inclusive offset range [base_offset, end_offset]. A partition’s segments have disjoint ranges, allocated in ascending order.

flowchart TB
P["log_partitions<br/>last_offset 11, log_start 4"]
S1["log_segments base 4, end 6<br/>blob: 3 frames, zstd"]
S2["log_segments base 7, end 11<br/>blob: 5 frames, zstd"]
T1["log_txns base 4, end 6<br/>hashes: 48 bytes"]
T2["log_txns base 7, end 11<br/>hashes: 80 bytes"]
C["log_consumers, group billing<br/>committed 6"]
P --> S1
P --> S2
P --> C
S1 -.-> T1
S2 -.-> T2

Offsets 0 to 3 are gone, deleted by retention, which is why log_start sits at 4 rather than 0. Segment lengths differ because nothing chose them, and each dashed sidecar row mirrors the range of the one segment it hangs off.

The tables

Four log tables are created by 001_log_schema.sql, applied at every boot; queen.log_dlq is a fifth, created by the ack file (005_log_ack) and covered below; Ack internals has the path that writes it. All of them hang off queen.queues (created by schema.sql), which is both the queue’s identity and its configuration.

queen.queues

One row per queue, and queue identity is this row’s id. On the wire a queue is named by (tenant_id, name), established by the unique index queues_tenant_name_uk, so two tenants can hold the same queue name on one broker; internally, every partition-to-config join is by id, because queen.log_partitions.queue_id references queen.queues(id) with ON DELETE CASCADE. There is no separate engine-side queue table: the same row carries the full configuration (the retry limit, the DLQ flags, visibility delays, encryption, retention policy) plus the two engine-side options below. A queue that was never /configured exists anyway: the first push, or an early subscription, provisions the row with its defaults.

lease_time needs care. It defaults to 60 on a row created implicitly by a push, but POST /api/v1/configure computes leaseTime with a default of 300 and writes it into the same column, so the effective default depends on how the queue came into existence. The 60 is a truth fix, not a behaviour change: implicitly created queues always leased at 60 (the pop path read that default), while the old config display showed a 300 the pop path never used. One column, one truth, and what the configuration shows is now the lease actually granted.

There is no storage engine-selector column. One engine, nothing to select; the wire still echoes storage: "segments" as a hard-coded literal.

queen.log_partitions

One row per partition. This row is the write serializer for the partition, and its columns are three offset watermarks, two retention timestamps derived from them, and bookkeeping.

UNIQUE (queue_id, name) makes partition names unique inside a queue. last_write_at is indexed by idx_log_partitions_queue_write (queue_id, last_write_at), which is what lets a wildcard pop range-scan only recently written partitions instead of the whole set. The quantization exists precisely because that index is there: bumping an indexed column makes the allocator UPDATE non-HOT, so push only writes a new value when more than a second has elapsed. The candidate scan absorbs the staleness by allowing two minutes of slack.

The two oldest_* columns are the retention work-list, and they exist so retention’s cost scales with deletable work rather than with the partition count. created_at is monotone within a partition (invariant 2 below), so “this partition holds something deletable under cutoff C” is exactly “the row at its watermark is older than C”: one range scan over a partial index, (queue_id, oldest_live_at) or (queue_id, oldest_txn_at) with a NOT NULL predicate, instead of one probe per partition. Each is a fact, never a policy (cutoffs stay computed from queue config at query time), and each is maintained in the same transaction, under the same row lock, as the watermark it mirrors: the retention step moves oldest_live_at with log_start, the sidecar purge moves oldest_txn_at with txns_start, and push only fills them on the empty-to-non-empty transition (COALESCE), so the allocator UPDATE stays HOT for a partition that already holds data. These two indexes are also the ones the schema apply builds CONCURRENTLY, because they arrive on an already-populated hot table.

The table is also tuned against churn, and the reasoning is worth knowing because it explains a class of latency spike. Every push updates this row, so between autovacuum passes the dead tuple count dwarfs the live row count. The table therefore sets fillfactor = 70, autovacuum_vacuum_scale_factor = 0 with autovacuum_vacuum_threshold = 500 (threshold-based triggering, so vacuum re-fires under churn), and, the important one, vacuum_truncate = off. Heap truncation takes an ACCESS EXCLUSIVE lock, and on a table with a fixed row population it reclaims nothing while freezing every push and pop behind the lock for seconds. Turning it off removed the whole periodic-spike class.

queen.log_segments

The messages.

blob is SET STORAGE EXTERNAL, so TOAST does not spend CPU trying to re-compress bytes the broker already compressed. log_segments carries zero secondary indexes, and that is a property of this one table rather than of the model: its primary key is also the pop path, so every read is a range scan or a backward step on it. The other tables here do carry an index where a scan needs one: log_partitions on (queue_id, last_write_at) plus the two partial retention-work-list indexes on (queue_id, oldest_live_at) and (queue_id, oldest_txn_at), and log_dlq on (partition_id, failed_at DESC).

Autovacuum on this table is set by measurement, and the numbers are recorded with their dates in 001_log_schema.sql. The earlier factors, 0.02 for vacuum and 0.05 for insert, re-fired vacuum roughly every 13 seconds at 1M msg/s, and each pass over the growing heap and its TOAST table competed for I/O exactly when the system had no headroom left: an absorbed hiccup at 800k msg/s, a stall at 900k, and a contributor to collapse at 1M. The table now sets autovacuum_vacuum_scale_factor = 0.1 and autovacuum_vacuum_insert_scale_factor = 0.3, with autovacuum_vacuum_cost_limit = 4000 and autovacuum_vacuum_cost_delay = 0 so each pass finishes fast. The same four values are set again with the toast. prefix. The blob bytes live in the TOAST table, so tuning only the heap would leave the larger half of the write volume on the stock cadence. At 0.1 and 0.3 vacuum still keeps pace with retention churn: the heap plateaus through slot reuse, verified over 300 seconds at 1M msg/s with retention sweeping every 5 seconds. These are storage parameters only, so applying them at every boot is idempotent and rewrites nothing.

Operators meet the same reasoning from the other side, next to the server settings that matter alongside it, in PostgreSQL. Do not strip these parameters in a tuning pass.

queen.log_txns

The hash sidecar. One row per segment, mirroring its offset range, holding 16 bytes per frame.

The broker computes the hashes; SQL never hashes anything, it only stores and compares bytea. The 128-bit width is what makes collisions a non-issue. There is deliberately no foreign key on partition_id: the purge path must never pay foreign-key trigger cost, and the rows are only ever reached through a live log_partitions row. The missing key leaves one obligation: whatever deletes a partition must delete these rows itself, because the purge phase walks live partitions, so a row orphaned by a partition delete would be unreachable forever. Exactly two things delete partitions, log_partition_cleanup_step_v1 and delete_queue_v1, and both delete the sidecar explicitly first.

The sidecar’s size is O(rate × window), independent of retention or backlog, because it is purged on its own clock. The retention loop purges rows older than GREATEST(dedup_window_seconds, completed_retention_seconds, 900) seconds and advances txns_start. A hash that outlives even that window resolves as unknown on ack, which the ack path treats as not acked: redelivery rather than loss.

queen.log_consumers

Coordination state, one row per (partition, consumer_group). This is the entire consumption model.

A leased batch is the span (committed, batch_end]. One row can hold at most one lease, which is why exactly one in-flight leased batch exists per (partition, group) and a single-partition queue cannot be consumed in parallel by one group.

attempt_count is telemetry and never consumes budget: lease expiry increments it but does not spend a retry. The two counters are separate on purpose: a consumer crash must not eat the retry budget an explicit failure is entitled to.

Like log_partitions, this table sets fillfactor = 50, threshold-based autovacuum and vacuum_truncate = off, for the same churn and exclusive-lock reasons.

queen.log_dlq

The per-message escape hatch. Everything else in this model is a cursor; this is the one table with a row per message. The payload is a snapshot, not a pointer, and the table carries no foreign key on purpose: a dead letter survives the retention that deletes its source segment, and the cleanup phase refuses to reclaim a partition that still holds one.

One secondary index, idx_log_dlq_partition_failed_at on (partition_id, failed_at DESC), which is what the DLQ browse and depth queries scan.

queen.log_dlq_head_v1 files the row and moves the cursor in one transaction: it sets committed = GREATEST(committed, p_off), releases the lease, and resets batch_retry_count and the attempt counters. The GREATEST is a guard rather than arithmetic. The cursor already sits at the completed prefix, so the normal move is exactly the one disposed frame, and a stale caller can never walk it backward.

The missing foreign key leaves the same obligation log_txns carries: whatever deletes a partition must delete these rows itself, and delete_queue_v1 does exactly that before dropping the queue row. Partition cleanup takes the opposite route and declines to delete at all. The eligibility predicate log_partition_dead_v1 vetoes on EXISTS (SELECT 1 FROM queen.log_dlq ...), so an empty, long-inactive partition that still holds a dead letter is not empty and stays. Nothing removes a row on a timer: retention never touches this table, and the only removals are DELETE /api/v1/messages/:partitionId/:transactionId, which removes every consumer group’s row for that address, and the two replay routes, which move exactly one row back into the log and delete it in the same transaction as the push.

The helper

queen.log_unnest_hashes(p BYTEA) RETURNS TABLE (idx INT, h BYTEA)

Explodes a 16-byte-stride blob into (idx, hash) rows with idx zero-based. It is IMMUTABLE pure byte slicing, so PostgreSQL can inline it into the calling query. Both the push deduplication probe and ack-by-hash resolution go through it.

What a segment looks like inside

The broker packs frames, then compresses the whole buffer with zstd at QUEEN_V2_ZSTD_LEVEL (default 3). Each frame is little-endian:

u32 body_len
body:
  u8   flags                  1 = trace id present, 2 = producer sub present, 4 = encrypted
  u8[16] message_id
  [u8[16] trace_id]           present only when flag 1 is set
  u16  txn_len
  u8[txn_len] transaction_id
  [u16 psub_len | u8[psub_len] producer_sub]   present only when flag 2 is set
  u8[...] payload             the JSON bytes as the client sent them

flowchart LR
S["log_segments row<br/>base_offset 7"] --> F0["frame 0<br/>offset 7"]
S --> F1["frame 1<br/>offset 8"]
S --> F2["frame 2<br/>offset 9"]
H["log_txns.hashes"] --> H0["bytes 0 to 15"]
H --> H1["bytes 16 to 31"]
H --> H2["bytes 32 to 47"]
H0 -.-> F0
H1 -.-> F1
H2 -.-> F2

Position is the entire addressing scheme, and it is why an ack that arrives carrying only a transactionId can be turned into a number: find the matching 16-byte slot, and its index is the message.

Reading offset O from a segment based at B means decompressing the blob and skipping to frame O - B. That is why a pop returns whole segment rows plus slicing bounds and does the frame walk in Rust: PostgreSQL never looks inside a blob.

Segment size is emergent, not configured

There is no “segment size” setting, and this is a design decision rather than a missing knob. The push fusion layer dispatches a partition’s accumulating segment the moment that partition has no flush in flight, so a segment contains exactly whatever arrived during the previous flush’s round trip. At low load that is one message; under load it grows with offered load on its own. Life of a push explains the mechanism.

The practical consequence: you cannot tune throughput by setting a batch size on the server. A larger client-side push batch does produce larger segments, and that does reduce commits per message.

The invariants everything else depends on

Four properties hold, and every other page in this section leans on at least one of them.

1. Ranges are disjoint and allocated monotonically. The allocator UPDATE on log_partitions.last_offset happens under that row’s lock, so two concurrent pushers to one partition cannot interleave ranges. A rolled-back push rolls back the allocator bump, so steady state has no holes.
2. created_at is monotone per partition, in commit order. The timestamp is stamped after the row lock is held, so a later pusher can only stamp after this transaction commits and releases. Time-based retention and timestamp subscriptions are both a single forward walk on the primary key because of this; without it they would need a sort.
3. log_start is the first live segment’s base_offset, or last_offset + 1 when the partition is empty. Retention only ever deletes a contiguous prefix of whole segments, and advances the watermark to the last deleted end_offset + 1.
4. Offsets are not dense. Retention leaves gaps, and the pop scan tolerates them: if the segment covering the wanted offset is gone, the scan takes the next segment whose base_offset is greater. A gap is not an error and not a stall.

What this model does not have

• No global order. Order is per partition. Two messages in different partitions of the same queue have no defined relative order, and nothing in the storage layer could give them one.
• No per-message pending state. Progress is a cursor: acking offset N implicitly completes every earlier unacked offset in that partition for that group, and there is nowhere to record “message 5 still pending while message 7 is done”. Terminal failure is different. A message that exhausts its retries is disposed of individually into queen.log_dlq, and the cursor moves past exactly that offset, which is what keeps one poison message from pinning an ordered lane. Dead lettering is on by default; with it disabled the poison frame is dropped instead of filed, and the cursor still moves past it.
• No reordering mechanism of any kind. A partition is strictly first-in, first-out in commit order. Nothing can jump the queue.