24-hour soak

In 24 hours, one broker against one PostgreSQL moved 51,820,403,100 messages (about 600,000 per second in each direction) with leased pops and explicit acknowledgement, zero restarts, and resident memory flat at roughly 6.3 GB. The host was a 32 vCPU / 62 GiB machine and the broker was commit 615efdc. Artifacts are in benchmark-queen/2026-07-25-soak24/, and the loader’s own stdout is raw/soak24b.out.

This is the reference run for production semantics, and the only one long enough to distinguish “fast” from “stays fast”.

Configuration

The loader printed its own configuration, which is what makes this quotable:

goload -mode openloop -> queue=segbench partitions=200 consumers=600 manualAck=true ackAsync=true ackInflight=256
  offered: rate=600000 msg/s | push-batch=100 -> 6000.0 req/s across 64 pacer workers | max-inflight=20000 | payload=256B
[configure] queue=segbench completedRetentionSeconds=300

Result

Straight from the final line of raw/soak24b.out:

Average accepted throughput is 51,820,403,100 over 86,400 s, or 599,773 msg/s. Per interval, the median achieved rate across all 8,640 samples is 599,980 msg/s, and 600,000 msg/s across the 7,020 samples after the four-and-a-half hour mark.

Popped exceeds pushed by 220,101 messages, or 0.0004%. That is the signature of at-least-once delivery: a lease that expires or an ack that fails redelivers its span, and the consumer counts it again.

Resources

From the 1 Hz host sampler, 87,500 samples, medians over the run after the first fifth:

Broker memory is the number this run exists to produce. A leak at this rate would be obvious: at 600,000 messages per second, a per-message overhead of even tens of bytes is gigabytes per hour. Across the last 19.5 hours the samples span 6,330 MB to 6,875 MB, with a median of 6,336 MB: a 545 MB band, no trend. The watchdog, sampling every five minutes, recorded restarts=0 on every line.

The database plateau is retention working. Completed messages leave after 300 s, so steady-state size is set by the retention window and the rate rather than by the run length. After load stopped, the watchdog’s last lines show the database drained to 8,292 MB.

Errors, in time order

The single “error rate 0.00012%” figure is the sum of all three error counters (15,742 plus 26 plus 47,600, or 63,368) over 51,820,403,100 accepted messages. It is worth decomposing, because those counters are not on the same scale and they did not accrue uniformly.

Push and pop errors count failed requests; each push request carried 100 messages. Ack errors count messages. So the message-level statement is different and also worth having: 1,604,500 messages of 51,822,007,600 offered were never accepted, which is 0.0031%, with nothing shed.

Cumulative counters at checkpoints:

82% of the failed pushes happened in the first four and a half hours. Errors did not stop after that, though: 13,500 acks failed between the nine- and twelve-hour marks. The session note’s claim of “zero new errors in the last 14 hours” is not what the stdout shows; the last increment to any counter is at 21.3 hours.

Latency was not flat, and the excursions have a cause

Whole-run p99 is 272 ms, and the median of the per-interval p99 values is 236 ms. But 43 of the 8,640 ten-second intervals (one in 200) reported a p99 above one second, and they cluster:

The first cluster was diagnosed live and fixed during the run. The cause was autovacuum’s heap-truncation phase on queen.log_partitions and queen.log_consumers, two fixed-population tables. Truncation takes an ACCESS EXCLUSIVE lock and freezes every push and pop for seconds while recovering nothing, because those tables do not shrink. It was caught with pg_stat_progress_vacuum at about the 2.7-hour mark and disabled with ALTER TABLE … SET (vacuum_truncate = off); the waves that continue to 4.5 hours are a vacuum already in flight plus the checkpoint cycle settling.

The two later clusters are checkpoint boundaries: PostgreSQL was writing about 10 GB every 15 minutes, close to back to back, and each excursion recovered on its own.

Two lessons in that, and both are operational rather than about Queen. The vacuum_truncate = off setting on the log engine’s fixed-population tables is load-bearing at this rate. And max_wal_size wants to be large enough to space checkpoints out. The residual blips are a checkpoint tuning artifact, not a broker one.

What this run establishes

• No memory leak on the hot path at scale. 24 hours at 600,000 messages per second per side with flat resident memory. Short runs cannot show this; the ceiling run in this same benchmark family found three separate out-of-memory conditions that only appear after about 580 seconds of accumulation.
• Retention reaches a steady state. The database plateaued instead of growing, across roughly 17,000 sweep cycles, and drained when load stopped.
• Explicit acknowledgement is affordable at this rate. Leases plus an offset commit per batch sustained the same 600,000 per second per side, at about 14 cores of broker CPU and 6 cores of PostgreSQL.
• PostgreSQL is the ceiling, and it is a commit ceiling. About 3,160 commits per second and 736 WAL fsyncs per second on storage with roughly 95 µs fdatasync.

What it does not establish

• Not a per-message completeness proof. The open-loop loader counts aggregates: offered, accepted, popped, acked. It does not assign a sequence number per message and check the set on arrival, so “zero loss” is not one of this run’s findings. The run that does per-message accounting is the ordered pipeline, and the run that does per-tenant bitmaps is the multi-tenant cell.
• Not an ordering proof. No verifier ran. Ordering was verified separately.
• Nothing about high availability. One broker, no mesh, no peer.
• Nothing about failure. No PostgreSQL restart, no failover, no broker kill, no consumer churn. The disk spool was never exercised.
• Nothing about the proxy, tenancy, quotas or rate limits. The loader spoke to the broker directly.
• One workload shape. 256-byte payloads, push batch 100, 200 partitions, one queue, one consumer group. Batch 1, large payloads and wildcard consumption are all materially different measurements.
• Fast storage. At about 95 µs per fdatasync, this figure is not disk-bound. The session note records that the disk plateaus at roughly 600,000 to 650,000 per second, which is why this run was configured at 600,000 rather than higher.

Files

To run something of this shape yourself, see Reproducing these runs.