The storage model

A message’s address in Queen is one number: a BIGINT offset, monotone within its partition. Everything in the storage layer follows from that choice. There is no per-message row, no per-message state, and no compound cursor. A consumer group’s progress in a partition is one integer; the difference between two integers is a message count; whether one message precedes another is a comparison.

The data itself is not stored per message either. Messages are packed into segments: one queen.log_segments row holds many messages as length-prefixed frames, compressed together with zstd, covering an inclusive offset range [base_offset, end_offset]. A partition’s segments have disjoint ranges, allocated in ascending order.

flowchart TB
P["log_partitions<br/>last_offset 11, log_start 4"]
S1["log_segments base 4, end 6<br/>blob: 3 frames, zstd"]
S2["log_segments base 7, end 11<br/>blob: 5 frames, zstd"]
T1["log_txns base 4, end 6<br/>hashes: 48 bytes"]
T2["log_txns base 7, end 11<br/>hashes: 80 bytes"]
C["log_consumers, group billing<br/>committed 6"]
P --> S1
P --> S2
P --> C
S1 -.-> T1
S2 -.-> T2

Offsets 0 to 3 are gone, deleted by retention, which is why log_start sits at 4 rather than 0. Segment lengths differ because nothing chose them, and each dashed sidecar row mirrors the range of the one segment it hangs off.

The tables

Four log tables are created by 001_log_schema.sql, applied at every boot; queen.log_dlq is a fifth, created by the ack file (005_log_ack). See Ack internals. All of them hang off queen.queues (created by schema.sql), which is both the queue’s identity and its configuration.

queen.queues

One row per queue, and queue identity is this row’s id. On the wire a queue is named by (tenant_id, name), established by the unique index queues_tenant_name_uk, so two tenants can hold the same queue name on one broker; internally, every partition-to-config join is by id, because queen.log_partitions.queue_id references queen.queues(id) with ON DELETE CASCADE. There is no separate engine-side queue table: the same row carries the full configuration (the retry limit, the DLQ flags, visibility delays, encryption, retention policy) plus the two engine-side options below. A queue that was never /configured exists anyway: the first push, or an early subscription, provisions the row with its defaults.

lease_time needs care. It defaults to 60 on a row created implicitly by a push, but POST /api/v1/configure computes leaseTime with a default of 300 and writes it into the same column, so the effective default depends on how the queue came into existence. The 60 is a truth fix, not a behaviour change: implicitly created queues always leased at 60 (the pop path read that default), while the old config display showed a 300 the pop path never used. One column, one truth, and what the configuration shows is now the lease actually granted.

There is no storage engine-selector column. One engine, nothing to select; the wire still echoes storage: "segments" as a hard-coded literal.

queen.log_partitions

One row per partition. This row is the write serializer for the partition, and its columns are three watermarks plus bookkeeping.

UNIQUE (queue_id, name) makes partition names unique inside a queue. last_write_at is indexed by idx_log_partitions_queue_write (queue_id, last_write_at), which is what lets a wildcard pop range-scan only recently written partitions instead of the whole set. The quantization exists precisely because that index is there: bumping an indexed column makes the allocator UPDATE non-HOT, so push only writes a new value when more than a second has elapsed. The candidate scan absorbs the staleness by allowing two minutes of slack.

The table is also tuned against churn, and the reasoning is worth knowing because it explains a class of latency spike. Every push updates this row, so between autovacuum passes the dead tuple count dwarfs the live row count. The table therefore sets fillfactor = 70, autovacuum_vacuum_scale_factor = 0 with autovacuum_vacuum_threshold = 500 (threshold-based triggering, so vacuum re-fires under churn), and, the important one, vacuum_truncate = off. Heap truncation takes an ACCESS EXCLUSIVE lock, and on a table with a fixed row population it reclaims nothing while freezing every push and pop behind the lock for seconds. Turning it off removed the whole periodic-spike class.

queen.log_segments

The messages.

blob is SET STORAGE EXTERNAL, so TOAST does not spend CPU trying to re-compress bytes the broker already compressed. The table has zero secondary indexes: the primary key is also the pop path, and every read is a range scan or a backward step on it.

queen.log_txns

The hash sidecar. One row per segment, mirroring its offset range, holding 16 bytes per frame.

The broker computes the hashes; SQL never hashes anything, it only stores and compares bytea. The 128-bit width is what makes collisions a non-issue. There is deliberately no foreign key on partition_id: the purge path must never pay foreign-key trigger cost, and the rows are only ever reached through a live log_partitions row.

The sidecar’s size is O(rate × window), independent of retention or backlog, because it is purged on its own clock. The retention loop purges rows older than GREATEST(dedup_window_seconds, completed_retention_seconds, 900) seconds and advances txns_start. A hash that outlives even that window resolves as unknown on ack, which the ack path treats as not acked: redelivery rather than loss.

queen.log_consumers

Coordination state, one row per (partition, consumer_group). This is the entire consumption model.

A leased batch is the span (committed, batch_end]. One row can hold at most one lease, which is why exactly one in-flight leased batch exists per (partition, group) and a single-partition queue cannot be consumed in parallel by one group.

attempt_count is telemetry and never consumes budget: lease expiry increments it but does not spend a retry. The two counters are separate on purpose: a consumer crash must not eat the retry budget an explicit failure is entitled to.

Like log_partitions, this table sets fillfactor = 50, threshold-based autovacuum and vacuum_truncate = off, for the same churn and exclusive-lock reasons.

The helper

queen.log_unnest_hashes(p BYTEA) RETURNS TABLE (idx INT, h BYTEA)

Explodes a 16-byte-stride blob into (idx, hash) rows with idx zero-based. It is IMMUTABLE pure byte slicing, so PostgreSQL can inline it into the calling query. Both the push deduplication probe and ack-by-hash resolution go through it.

What a segment looks like inside

The broker packs frames, then compresses the whole buffer with zstd at QUEEN_V2_ZSTD_LEVEL (default 3). Each frame is little-endian:

u32 body_len
body:
  u8   flags                  1 = trace id present, 2 = producer sub present, 4 = encrypted
  u8[16] message_id
  [u8[16] trace_id]           present only when flag 1 is set
  u16  txn_len
  u8[txn_len] transaction_id
  [u16 psub_len | u8[psub_len] producer_sub]   present only when flag 2 is set
  u8[...] payload             the JSON bytes as the client sent them

flowchart LR
S["log_segments row<br/>base_offset 7"] --> F0["frame 0<br/>offset 7"]
S --> F1["frame 1<br/>offset 8"]
S --> F2["frame 2<br/>offset 9"]
H["log_txns.hashes"] --> H0["bytes 0 to 15"]
H --> H1["bytes 16 to 31"]
H --> H2["bytes 32 to 47"]
H0 -.-> F0
H1 -.-> F1
H2 -.-> F2

Position is the entire addressing scheme, and it is why an ack that arrives carrying only a transactionId can be turned into a number: find the matching 16-byte slot, and its index is the message.

Reading offset O from a segment based at B means decompressing the blob and skipping to frame O - B. That is why a pop returns whole segment rows plus slicing bounds and does the frame walk in Rust: PostgreSQL never looks inside a blob.

Segment size is emergent, not configured

There is no “segment size” setting, and this is a design decision rather than a missing knob. The push fusion layer dispatches a partition’s accumulating segment the moment that partition has no flush in flight, so a segment contains exactly whatever arrived during the previous flush’s round trip. At low load that is one message; under load it grows with offered load on its own. Life of a push explains the mechanism.

The practical consequence: you cannot tune throughput by setting a batch size on the server. A larger client-side push batch does produce larger segments, and that does reduce commits per message.

The invariants everything else depends on

Four properties hold, and every other page in this section leans on at least one of them.

1. Ranges are disjoint and allocated monotonically. The allocator UPDATE on log_partitions.last_offset happens under that row’s lock, so two concurrent pushers to one partition cannot interleave ranges. A rolled-back push rolls back the allocator bump, so steady state has no holes.
2. created_at is monotone per partition, in commit order. The timestamp is stamped after the row lock is held, so a later pusher can only stamp after this transaction commits and releases. Time-based retention and timestamp subscriptions are both a single forward walk on the primary key because of this; without it they would need a sort.
3. log_start is the first live segment’s base_offset, or last_offset + 1 when the partition is empty. Retention only ever deletes a contiguous prefix of whole segments, and advances the watermark to the last deleted end_offset + 1.
4. Offsets are not dense. Retention leaves gaps, and the pop scan tolerates them: if the segment covering the wanted offset is gone, the scan takes the next segment whose base_offset is greater. A gap is not an error and not a stall.

What this model does not have

• No global order. Order is per partition. Two messages in different partitions of the same queue have no defined relative order, and nothing in the storage layer could give them one.
• No per-message consumption state. Acking offset N implicitly completes every earlier unacked offset in that partition for that group. There is no place to record “message 7 done, message 5 still pending”.
• No reordering mechanism of any kind. A partition is strictly first-in, first-out in commit order. Nothing can jump the queue.