Reproducing these runs

The loader, its six modes and the flags that matter, how to build a capped cell rig, and the handful of knobs that move the result more than anything else.

Everything on these pages was produced by one Go program and a handful of shell scripts, all in the repository. This page is how to run them and, more usefully, which few settings actually move the answer.

The loader

benchmark-queen/2026-07-29-vm-campaign/goload/ is a load generator built on the official Go client, which matters: it exercises the same SDK a user would, not a bespoke HTTP path.

benchmark-queen/2026-07-29-vm-campaign/goload/build.sh

build.sh linux cross-compiles a static goload-linux-amd64 for a bench VM. The script exports GOWORK=off, and that is required rather than tidy: the module lives inside the repository tree, so the repository’s own go.work claims the directory and go build refuses. With the workspace off, the module’s replace directive resolves the in-tree client-go, so the loader is built against exactly the client being measured.

Modes

Select with -mode; each mode has its own flag set, and goload -mode <name> -h prints it.

Mode	Shape	Used for
`openloop`	Paced open-loop producers against one queue; closed-loop drainers as consumers	24-hour soak, Throughput ceiling
`cm`	Four-stage pipeline over N entities with per-entity work sleeps and a built-in order verifier	Ordered pipeline
`cloud`	Multi-tenant: every simulated tenant is a real tenant with its own cluster, API key and Host, with per-message delivery accounting	Multi-tenant cell
`provision`	Creates N real tenants through the proxy’s bootstrap function and caches their API keys to a file, so re-runs never re-provision	prerequisite for `cloud` against the proxy
`max`	Pure in-and-out throughput with server-side auto-ack; many producers, many consumers	quick “max pipe” checks
`app`	Closed-loop target rate with simulated processing, fan-out groups, skewed partitions, failure into retry and dead-letter	application-shaped exploration

The open-loop contract

Three properties of -mode openloop and -mode cloud are what make their numbers interpretable, and all three are worth understanding before changing anything.

The pacer never blocks. Producers are a fixed schedule, not a worker pool. Each push launches at its scheduled instant in its own goroutine. In-flight requests are capped by -max-inflight, and a request that arrives at the cap has its messages shed: counted as offered, never sent. Blocking at the cap instead would silently turn the pacer into a closed-loop worker pool and the “offered rate” would become meaningless. A run with a non-zero shed count did not offer what it claims.

Latency is coordinated-omission-corrected. It is measured from each request’s scheduled instant. A pacer that fell behind shows up as latency instead of vanishing from the histogram.

Producers do not retry. RetryAttempts is set to force exactly one attempt: a failed push is counted and dropped. Retrying would double-offer and corrupt the offered-rate accounting. Real producers retry; the loader does not, and the error counters are where the difference lands.

Flags that define the semantics under test

These are the ones that change what is being measured rather than how hard:

Flag	Default	Effect
`-manual-ack` (openloop) / `-auto-ack` (cloud)	leased pop with explicit ack in `cloud`; `openloop` defaults to server-side auto-ack	Server-side auto-ack removes a round trip and a transaction per batch and is at-most-once. This single flag is most of the difference between the 600,000 and 1,000,000 per second figures on this site
`-ack-async`	off	Dispatch each batch’s ack on its own task so a consumer does not hold a lease blocked on its own ack. Never sheds an ack: it blocks when `-ack-inflight` is full
`-dedup-window`	0, meaning off, in `openloop` and `cloud`; 300 in `cm`	Turns on the exact per-partition duplicate probe under the partition row lock. Not free, and the reason “deduplication off” is part of two headlines here
`-push-batch`	10 (openloop), 1 (cloud)	Amortises per-transaction cost. Batch 1 is the honest small-message shape and a much lower ceiling
`-pop-batch`, `-pop-partitions`	200 / 1 (openloop)	Consumption cost per message. Claiming several partitions per call uses the multi-partition wildcard path
`-pop-wait`	off (openloop), on (cloud)	Long polling. This is what holds a parked consumer slot
`-partitions`, `-consumers`	100 / 150	Consumption parallelism per group is bounded by partition count
`-verify` (cloud)	on	Per-message `(tenant, seq)` accounting: loss, duplication, cross-tenant
`-retry429-attempts` (cloud)	1	1 surfaces the 429 immediately. The campaign measures rate limiting rather than hiding it
`-fault` (cloud)	none	`dup-push=N`, `lose-msg=N`, `drop-ack=N`: deliberate discrepancies that prove the checker can fail

Reproducing a single-node throughput run

The rig is one machine for the broker and PostgreSQL and a separate machine for the loader. The separation is not optional: at 32 cores, goload itself was once the ceiling. Parsing pop responses pinned it at 93% CPU with 4% host idle, and moving to a 48-core loader was what turned a ~2 s p99 into 358 ms. Before believing any ceiling, check the loader’s own idle time.

Start PostgreSQL 18 with a memory configuration sized to the host. At a million messages per second the defaults will end the run: shared_buffers 16 GB, work_mem 12 MB and maintenance_work_mem 512 MB is the combination that survived, on a 62 GiB host, alongside a 20 GB broker deduplication cache cap.
Start the broker against it, and let it apply its own schema.
Run the loader from the other machine, in the 24-hour soak’s shape: explicit asynchronous acks, push batch 100, 200 partitions, 600 consumers.
```
./goload -mode openloop -url http://BROKER:6632 -rate 600000 -push-batch 100 -partitions 200 -consumers 600 -pop-batch 500 -manual-ack -ack-async -ack-inflight 256 -max-inflight 20000 -idle-conns 8192 -payload 256 -duration 86400 -report 10
```
The soak’s archived header records the rate, push batch, partition and consumer counts, both ack settings, -max-inflight and the payload size. -pop-batch 500 comes from that session’s note, and -idle-conns 8192 is the value the sessions settled on rather than one recorded for that run. See the knobs list below for why it matters.
Sample both hosts at 1 Hz alongside the run. The archived sessions used small shell samplers writing CSV: broker and PostgreSQL CPU and memory, pg_stat_database commit deltas, WAL records and fsyncs, database size, active backends and the top wait event on the broker side; loader CPU, memory and network on the other.

To find a ceiling rather than hold a rate, ladder the offered rate and stop where achieved stops tracking offered. s1-ceiling.sh is that pattern: reset the database and restart the broker before every point, 45 seconds of load and 45 seconds of drain, correctness verified inside the run. The long drain is deliberate: a partition can go quiet for around 30 seconds, and a short drain reports a stall as loss.

Reproducing the ordered pipeline

-mode cm builds the four-queue topology itself, configures all four queues at t=0, runs, and then verifies. The verifier can also be run on its own afterwards.

Run the pipeline. The warm-up is on by default and pushes sequence 0 for every property on both flows before the pacer starts, so 4000 partitions are not created cold under rated load.

./goload -mode cm -url http://BROKER:6632 -properties 1000 -rate-events 25000 -ramp-sec 15 -duration 600 -pop-batch 100 -pop-partitions 10 -pop-wait -dedup-window 300 -logdir /root/cmlogs

Re-verify the logs without re-running, which is useful when tuning the verifier or checking a run after the fact.
```
./goload -mode cm -verify-only -logdir /root/cmlogs
```

The verifier reads produced.meta for ground truth and one <queue>_<group>.log per stage, and prints the per-stage table reproduced on Ordered pipeline. A gap below the delivery frontier always fails; an ordering violation fails unless acknowledgements failed during the run.

Reproducing a capped cell

benchmark-queen/vm-cell.sh builds the whole cell (the proxy’s state database, the cell’s PostgreSQL, the broker and the proxy) and puts all four under one systemd slice, so a single number caps them together. Two shapes come out of the same script, which is the point.

Bring up a free-tier shape: two cores and 8 GiB covering everything.
```
vm-cell.sh up --cell-cpus 2 --cell-mem 8
```
For the full-machine ceiling instead, --cell-cpus 0 leaves it uncapped. --enforce 0 runs the proxy in shadow mode. PostgreSQL’s own tuning tracks the cell’s memory budget rather than the host’s (25% of the cap as shared_buffers), because a free cell measured with a large host’s settings is fiction.

Provision real tenants and cache their keys.

./goload -mode provision -tenants 12 -prefix soak -plan bench -cell bench -file tenants.json

Drive them through the proxy with per-message accounting on.

./goload -mode cloud -target proxy -tenants-file tenants.json -tenants 12 -shared-queue -queue orders -group workers -partitions 8 -per-tenant-rate 70 -push-batch 1 -consumers-per-tenant 3 -pop-batch 50 -pop-wait -lease-time 30 -duration 3600 -drain 30 -out .

For a comparable direct-to-broker control point, swap the target. The -broker-tenant-header default sends the same tenant UUID the proxy would inject, so both targets hit the same broker-side tenant rows.
```
./goload -mode cloud -target broker -tenants-file tenants.json -tenants 12 -shared-queue -rate 840 -duration 3600
```

Two scripts around that are worth copying rather than reinventing. reset-cell-db.sh drops and recreates the broker’s database and restarts the broker, so every measured point starts from cold in-memory state with no carried-over retained rows or committed offsets. And runpt.sh wraps one point with the accounting the cell numbers require: per-cgroup CPU for each cell component and for the load generator sampled every second, xact_commit deltas per second (which is where commits-per-delivered-message comes from), and pg_stat_activity wait events during load. It also records throttled_usec, which is the honest “did we hit the ceiling” signal: a capped cgroup can sit slightly under its quota while runnable work waits.

The knobs that move the result most

Roughly in order of effect, from the archived sessions:

Delivery semantics. Server-side auto-ack versus leased pop with an explicit ack. On the single-node rig this is the difference between 1,000,000 and 600,000 messages per second per side.
Push batch size. The per-transaction cost is amortised across the batch. Batch 100 and batch 1 are different measurements of different systems, and every headline here names which.
Idle connections. -idle-conns at or above the concurrent worker count was worth about +9.7% throughput and +18.5% messages per commit in a controlled A/B, and took TIME_WAIT sockets from 26,000 to zero. There is no reason to leave it low.
PostgreSQL memory configuration. Three separate out-of-memory failures at a million messages per second, all at about t=580 s, all invisible to 300-second runs. Size shared_buffers, work_mem, maintenance_work_mem and the broker’s deduplication cache cap against the host budget together.
Deduplication window. The exact probe happens under the partition row lock before offsets are allocated. Turning it on is a real cost and a real guarantee.
Autovacuum behaviour on the log engine’s two fixed-population tables. Heap truncation on queen.log_partitions and queen.log_consumers takes an ACCESS EXCLUSIVE lock, freezes every push and pop for seconds and recovers nothing. 041_log_schema.sql now sets vacuum_truncate = off and threshold-based autovacuum on both, but if you are running an older schema this is the single largest source of multi-second latency spikes.
Checkpoint spacing. max_wal_size too small means PostgreSQL writes near-continuously at checkpoint boundaries. Those are the residual latency blips visible in the 24-hour soak.
Warm-up. Creating thousands of partitions cold under rated load is a measured wedge. If your production entities pre-exist, pre-create them in the benchmark too.
Ramp. -ramp-sec avoids a cold-start storm where pool dial-up, first-contact seeding and cache hydration all land in the first second.
Where the loader runs. Same box means a shared CPU budget unless it is in a different cgroup, and a loader at 93% CPU measures the loader.

Reporting a run

The bar this site holds itself to is on Benchmarks, and it is worth adopting: keep the loader’s stdout or its JSON run record, the printed configuration line, the host shape, and the identity of the broker build. Without those four, a number is a memory. Two of the strongest results this project ever produced are unpublishable for exactly that reason.