---
title: "Reproducing these runs"
description: "The loader, its six modes and the flags that matter, how to build a capped cell rig, and the handful of knobs that move the result more than anything else."
---

> Documentation Index
> Fetch the complete documentation index at: https://queenmq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Reproducing these runs

Everything on these pages was produced by one Go program and a handful of shell scripts, all in
the repository. This page is how to run them and, more usefully, which few settings actually
move the answer.

## The loader

`benchmark-queen/2026-07-29-vm-campaign/goload/` is a load generator built on the official Go
client, which matters: it exercises the same SDK a user would, not a bespoke HTTP path.

```bash
benchmark-queen/2026-07-29-vm-campaign/goload/build.sh
```

`build.sh linux` cross-compiles a static `goload-linux-amd64` for a bench VM. The script exports
`GOWORK=off`, and that is required rather than tidy: the module lives inside the repository tree,
so the repository's own `go.work` claims the directory and `go build` refuses. With the workspace
off, the module's `replace` directive resolves the in-tree `client-go`, so the loader is built
against exactly the client being measured.

### Modes

Select with `-mode`; each mode has its own flag set, and `goload -mode <name> -h` prints it.

| Mode | Shape | Used for |
| --- | --- | --- |
| `openloop` | Paced open-loop producers against one queue; closed-loop drainers as consumers | [24-hour soak](/benchmarks/soak-24h), [Throughput ceiling](/benchmarks/peak) |
| `cm` | Four-stage pipeline over N entities with per-entity work sleeps and a built-in order verifier | [Ordered pipeline](/benchmarks/ordered-pipeline) |
| `cloud` | Multi-tenant: every simulated tenant is a real tenant with its own cluster, API key and Host, with per-message delivery accounting | [Multi-tenant cell](/benchmarks/multitenant-cell) |
| `provision` | Creates N real tenants through the proxy's bootstrap function and caches their API keys to a file, so re-runs never re-provision | prerequisite for `cloud` against the proxy |
| `max` | Pure in-and-out throughput with server-side auto-ack; many producers, many consumers | quick "max pipe" checks |
| `app` | Closed-loop target rate with simulated processing, fan-out groups, skewed partitions, failure into retry and dead-letter | application-shaped exploration |

### The open-loop contract

Three properties of `-mode openloop` and `-mode cloud` are what make their numbers interpretable,
and all three are worth understanding before changing anything.

**The pacer never blocks.** Producers are a fixed schedule, not a worker pool. Each push launches
at its scheduled instant in its own goroutine. In-flight requests are capped by `-max-inflight`,
and a request that arrives at the cap has its messages **shed**: counted as offered, never sent.
Blocking at the cap instead would silently turn the pacer into a closed-loop worker pool and the
"offered rate" would become meaningless. A run with a non-zero shed count did not offer what it
claims.

**Latency is coordinated-omission-corrected.** It is measured from each request's *scheduled*
instant. A pacer that fell behind shows up as latency instead of vanishing from the histogram.

**Producers do not retry.** `RetryAttempts` is set to force exactly one attempt: a failed push is
counted and dropped. Retrying would double-offer and corrupt the offered-rate accounting. Real
producers retry; the loader does not, and the error counters are where the difference lands.

### Flags that define the semantics under test

These are the ones that change *what* is being measured rather than how hard:

| Flag | Default | Effect |
| --- | --- | --- |
| `-manual-ack` (openloop) / `-auto-ack` (cloud) | leased pop with explicit ack in `cloud`; `openloop` defaults to server-side auto-ack | Server-side auto-ack removes a round trip and a transaction per batch and is at-most-once. This single flag is most of the difference between the 600,000 and 1,000,000 per second figures on this site |
| `-ack-async` | off | Dispatch each batch's ack on its own task so a consumer does not hold a lease blocked on its own ack. Never sheds an ack: it blocks when `-ack-inflight` is full |
| `-dedup-window` | 0, meaning **off**, in `openloop` and `cloud`; 300 in `cm` | Turns on the exact per-partition duplicate probe under the partition row lock. Not free, and the reason "deduplication off" is part of two headlines here |
| `-push-batch` | 10 (openloop), 1 (cloud) | Amortises per-transaction cost. Batch 1 is the honest small-message shape and a much lower ceiling |
| `-pop-batch`, `-pop-partitions` | 200 / 1 (openloop) | Consumption cost per message. Claiming several partitions per call uses the multi-partition wildcard path |
| `-pop-wait` | off (openloop), on (cloud) | Long polling. This is what holds a parked consumer slot |
| `-partitions`, `-consumers` | 100 / 150 | Consumption parallelism per group is bounded by partition count |
| `-verify` (cloud) | on | Per-message `(tenant, seq)` accounting: loss, duplication, cross-tenant |
| `-retry429-attempts` (cloud) | 1 | 1 surfaces the 429 immediately. The campaign measures rate limiting rather than hiding it |
| `-fault` (cloud) | none | `dup-push=N`, `lose-msg=N`, `drop-ack=N`: deliberate discrepancies that prove the checker can fail |

> **Caution**
>
> The loader in the tree is the 2026-07-29 build, and its defaults are not the defaults the earlier
> sessions ran with. The clearest example: `-mode cm` now defaults `-final-targeted` and
> `-intermediate-targeted` to true, which uses static partition ownership instead of wildcard pops,
> because the wildcard candidate scan was later measured at about 12 ms per pop at 1000 partitions.
> The archived [ordered pipeline](/benchmarks/ordered-pipeline) run predates those flags and was
> **wildcard throughout**. Pass `-final-targeted=false -intermediate-targeted=false` to reproduce
> its shape; leave them on to measure the faster one. Always read a run's printed header rather
> than assuming today's defaults.

## Reproducing a single-node throughput run

The rig is one machine for the broker and PostgreSQL and a separate machine for the loader. The
separation is not optional: at 32 cores, `goload` itself was once the ceiling. Parsing pop
responses pinned it at 93% CPU with 4% host idle, and moving to a 48-core loader was what turned
a ~2 s p99 into 358 ms. Before believing any ceiling, check the loader's own idle time.

1. Start PostgreSQL 18 with a memory configuration sized to the host. At a million messages per
   second the defaults will end the run: `shared_buffers` 16 GB, `work_mem` 12 MB and
   `maintenance_work_mem` 512 MB is the combination that survived, on a 62 GiB host, alongside a
   20 GB broker deduplication cache cap.

2. Start the broker against it, and let it apply its own schema.

3. Run the loader from the other machine, in the 24-hour soak's shape: explicit asynchronous
   acks, push batch 100, 200 partitions, 600 consumers.

```bash
   ./goload -mode openloop -url http://BROKER:6632 -rate 600000 -push-batch 100 -partitions 200 -consumers 600 -pop-batch 500 -manual-ack -ack-async -ack-inflight 256 -max-inflight 20000 -idle-conns 8192 -payload 256 -duration 86400 -report 10
```

   The soak's archived header records the rate, push batch, partition and consumer counts, both
   ack settings, `-max-inflight` and the payload size. `-pop-batch 500` comes from that session's
   note, and `-idle-conns 8192` is the value the sessions settled on rather than one recorded for
   that run. See the knobs list below for why it matters.

4. Sample both hosts at 1 Hz alongside the run. The archived sessions used small shell samplers
   writing CSV: broker and PostgreSQL CPU and memory, `pg_stat_database` commit deltas, WAL
   records and fsyncs, database size, active backends and the top wait event on the broker side;
   loader CPU, memory and network on the other.

To find a ceiling rather than hold a rate, ladder the offered rate and stop where achieved stops
tracking offered. `s1-ceiling.sh` is that pattern: reset the database and restart the broker
before *every* point, 45 seconds of load and 45 seconds of drain, correctness verified inside the
run. The long drain is deliberate: a partition can go quiet for around 30 seconds, and a short
drain reports a stall as loss.

## Reproducing the ordered pipeline

`-mode cm` builds the four-queue topology itself, configures all four queues at t=0, runs, and
then verifies. The verifier can also be run on its own afterwards.

1. Run the pipeline. The warm-up is on by default and pushes sequence 0 for every property on both
   flows before the pacer starts, so 4000 partitions are not created cold under rated load.

```bash
   ./goload -mode cm -url http://BROKER:6632 -properties 1000 -rate-events 25000 -ramp-sec 15 -duration 600 -pop-batch 100 -pop-partitions 10 -pop-wait -dedup-window 300 -logdir /root/cmlogs
```

2. Re-verify the logs without re-running, which is useful when tuning the verifier or checking a
   run after the fact.

```bash
   ./goload -mode cm -verify-only -logdir /root/cmlogs
```

The verifier reads `produced.meta` for ground truth and one `<queue>_<group>.log` per stage, and
prints the per-stage table reproduced on
[Ordered pipeline](/benchmarks/ordered-pipeline). A gap below the delivery frontier always fails;
an ordering violation fails unless acknowledgements failed during the run.

## Reproducing a capped cell

`benchmark-queen/vm-cell.sh` builds the whole cell (the proxy's state database, the cell's
PostgreSQL, the broker and the proxy) and puts all four under **one systemd slice**, so a single
number caps them together. Two shapes come out of the same script, which is the point.

1. Bring up a free-tier shape: two cores and 8 GiB covering everything.

```bash
   vm-cell.sh up --cell-cpus 2 --cell-mem 8
```

   For the full-machine ceiling instead, `--cell-cpus 0` leaves it uncapped. `--enforce 0` runs
   the proxy in shadow mode. PostgreSQL's own tuning tracks the *cell's* memory budget rather
   than the host's (25% of the cap as `shared_buffers`), because a free cell measured with a
   large host's settings is fiction.

2. Provision real tenants and cache their keys.

```bash
   ./goload -mode provision -tenants 12 -prefix soak -plan bench -cell bench -file tenants.json
```

3. Drive them through the proxy with per-message accounting on.

```bash
   ./goload -mode cloud -target proxy -tenants-file tenants.json -tenants 12 -shared-queue -queue orders -group workers -partitions 8 -per-tenant-rate 70 -push-batch 1 -consumers-per-tenant 3 -pop-batch 50 -pop-wait -lease-time 30 -duration 3600 -drain 30 -out .
```

4. For a comparable direct-to-broker control point, swap the target. The
   `-broker-tenant-header` default sends the same tenant UUID the proxy would inject, so both
   targets hit the same broker-side tenant rows.

```bash
   ./goload -mode cloud -target broker -tenants-file tenants.json -tenants 12 -shared-queue -rate 840 -duration 3600
```

Two scripts around that are worth copying rather than reinventing. `reset-cell-db.sh` drops and
recreates the broker's database and restarts the broker, so every measured point starts from cold
in-memory state with no carried-over retained rows or committed offsets. And `runpt.sh` wraps one
point with the accounting the cell numbers require: per-cgroup CPU for each cell component *and*
for the load generator sampled every second, `xact_commit` deltas per second (which is where
commits-per-delivered-message comes from), and `pg_stat_activity` wait events during load. It also
records `throttled_usec`, which is the honest "did we hit the ceiling" signal: a capped cgroup can
sit slightly under its quota while runnable work waits.

> **Note**
>
> `runpt.sh`'s samplers reach PostgreSQL over the bridge from the host with `psql`, never through
> `docker exec`. A `docker exec` would fork the sampler's own `psql` *inside* the cell's cgroup and
> bill its CPU to Postgres, which quietly inflates the thing being measured.

## The knobs that move the result most

Roughly in order of effect, from the archived sessions:

1. **Delivery semantics.** Server-side auto-ack versus leased pop with an explicit ack. On the
   single-node rig this is the difference between 1,000,000 and 600,000 messages per second per
   side.
2. **Push batch size.** The per-transaction cost is amortised across the batch. Batch 100 and
   batch 1 are different measurements of different systems, and every headline here names which.
3. **Idle connections.** `-idle-conns` at or above the concurrent worker count was worth about
   +9.7% throughput and +18.5% messages per commit in a controlled A/B, and took `TIME_WAIT`
   sockets from 26,000 to zero. There is no reason to leave it low.
4. **PostgreSQL memory configuration.** Three separate out-of-memory failures at a million
   messages per second, all at about t=580 s, all invisible to 300-second runs. Size
   `shared_buffers`, `work_mem`, `maintenance_work_mem` and the broker's deduplication cache cap
   against the host budget *together*.
5. **Deduplication window.** The exact probe happens under the partition row lock before offsets
   are allocated. Turning it on is a real cost and a real guarantee.
6. **Autovacuum behaviour on the log engine's two fixed-population tables.** Heap truncation on
   `queen.log_partitions` and `queen.log_consumers` takes an `ACCESS EXCLUSIVE` lock, freezes
   every push and pop for seconds and recovers nothing. `041_log_schema.sql` now sets
   `vacuum_truncate = off` and threshold-based autovacuum on both, but if you are running an
   older schema this is the single largest source of multi-second latency spikes.
7. **Checkpoint spacing.** `max_wal_size` too small means PostgreSQL writes near-continuously at
   checkpoint boundaries. Those are the residual latency blips visible in the 24-hour soak.
8. **Warm-up.** Creating thousands of partitions cold under rated load is a measured wedge. If
   your production entities pre-exist, pre-create them in the benchmark too.
9. **Ramp.** `-ramp-sec` avoids a cold-start storm where pool dial-up, first-contact seeding and
   cache hydration all land in the first second.
10. **Where the loader runs.** Same box means a shared CPU budget unless it is in a different
    cgroup, and a loader at 93% CPU measures the loader.

## Reporting a run

The bar this site holds itself to is on [Benchmarks](/benchmarks), and it is worth adopting: keep
the loader's stdout or its JSON run record, the printed configuration line, the host shape, and
the identity of the broker build. Without those four, a number is a memory. Two of the strongest
results this project ever produced are unpublishable for exactly that reason.

Source: https://queenmq.com/benchmarks/reproduce/index.mdx