Skip to content

Benchmark methodology: machines, settings, commands

The machines, the load generators and what they count, each system's settings and durability, QueenMQ's non-default settings, the commands that ran every benchmark in this section, and the archive the runs live in.

Updated View as Markdown

Every number in this section comes from a folder of logs, and most can be produced again with one command. This page is how: the machines, the load generators and what they count, the settings each system ran with, and the scripts.

Three sets of machines

The section has three generations of runs, and no page compares a number across two of them.

  • GCP, 2026-10-14 to 16. QueenMQ 2.4 against Kafka 4.3.1, Pulsar 4.2.4 and Redpanda 26.2.3: the comparison and partition count.
  • DigitalOcean, 2026-10-12 and 13. Pre-release builds of 2.3.0: the transactions against 2.2.0 and five Kafka-client points.
  • DigitalOcean, September and 2026-10-01 and 02. QueenMQ 2.0.0-beta builds with Kafka, Redpanda and Pulsar in transactions, the soak, the first Kafka-client runs, and Laravel on one node.

The two DigitalOcean sets are six droplets of the same size, not the same droplets.

The GCP runs

Role Machine type CPU Memory Disk for the data
3 brokers c4-standard-24-lssd 24 vCPU, Intel Xeon 6985P-C 88 GB four local NVMe disks (4 × 375 GB) in one RAID0, XFS noatime
3 load generators, 6 for Pulsar’s largest points c4-highcpu-16 16 vCPU, Intel Xeon Platinum 8581C 32 GB boot disk only

One zone (europe-west1-b), one VPC, Ubuntu 24.04, the same kernel settings for every system, clocks synced with chrony. The VMs were made three times, on 2026-10-14, 15 and 16, and every broker’s disks were measured before use: a 32 KB write and fdatasync took 0.05 to 0.06 ms at the median on every broker of every set, 16,000 to 17,700 a second. A local SSD that has been written for hours without TRIM slows down (on 10-14 one disk of one broker went to 30 ms a write), so the RAID gets an fstrim before a series. On 10-16 QueenMQ and Kafka ran their 1,000-partition point again and read what they had read on the earlier sets.

QueenMQ ran as 2.4.0-beta.1 (the image ghcr.io/queen-mq/queen:2.4.0-beta.1; the runs of 10-15 are the same code built before the version commit), three voters, one tenant and one raft group, every broker setting at its default except QUEEN_CONSUME_IDLE_UNLOAD_S=30 at ten million partitions. Its load generator turns on the wire forms that are off by default in every client (-push-wire bin -push-zstd -ack-counts -pop-frames: the binary push, a client-compressed group, acks answered with counts, framed pops). Each queue had deduplication off and kept completed messages 1,800 seconds, so nothing was deleted during a point.

A point is a fresh cluster with its partitions created and warmed, 70 seconds of load with a 10-second ramp in them (90 for QueenMQ’s runs of 10-16), nine load processes. A latency is the median of the three 10-second windows in seconds 30 to 60, each window taking the slowest process, with the worst window in brackets; CPU is the three brokers’ cores over the loaded window. A point is carried when at least 99.5% of the offered rate is pushed, nothing is shed or errs, and 99% of what was pushed is consumed. Each system’s settings are on the comparison.

The 2.3.0 runs on DigitalOcean

The transactions against 2.2.0 and the Kafka-client points of 2026-10-12 and 13 ran here.

Role Machine CPU 32 KB write and fdatasync, p50 / p99
Broker, the leader queen Xeon Gold 6548N 0.08 / 0.29 ms
Broker queen-02 Xeon Platinum 8358 0.10 / 0.40 ms
Broker queen-03 Xeon Platinum 8358 0.09 / 0.37 ms
Load queen-04 Xeon Gold 6548N 0.72 to 0.94 / 6.3 to 14.4 ms
Load queen-05 Xeon Gold 6548N 0.76 to 0.95 / 2.6 to 3.2 ms
Load queen-06 Xeon Platinum 8358 0.20 / 1.17 ms

Each has 16 vCPU and 31 GB, Ubuntu 24.04 and one ext4 disk. The roles follow the disks: two of the three faster CPUs sync ten times slower than the others, and the first runs, with those two as followers, read a push p99 of 242 ms at 300,000 msg/s that belonged to one follower’s clients alone. With the three fast disks as brokers the same build read 5 ms. Measure every disk before giving roles (scripts/fsync.py in the archive).

QueenMQ ran as release builds (cargo build --release --locked, the profile of the published image) of the commits that became 2.3.0, three voters, one tenant and one raft group, every broker setting at its default, with the brokers’ kernel keeping 512 MB free (vm.min_free_kbytes, with vm.watermark_scale_factor 200). The load generator is qload, nine processes on the three load machines, open loop with the rules below.

# from benchmark-queen/2026-10-12-three-node-new-forms/scripts, hosts.env filled in
./txnrun.sh txn-bin-10 bin 10 40 60000 120000 180000            # transactions of 10, binary body
QBIN=220 ./txnrun.sh txn-220-10 json 10 40 60000 120000 180000  # the same on 2.2.0

The folder’s README.md has the runs of 2026-10-12 (2.2.0 against the new wire forms, what stops one raft group, the transactions) and NIGHT.md the log of the night that followed: each stall, its cause, the build that removed it and the tables of every matrix.

The machines of the September runs

Role Count Shape
Brokers 3 DigitalOcean, 16 vCPU, 31 GB, one local disk with ext4. Two Xeon Gold 6548N, one Xeon Platinum 8358
Load machines 3, and a fourth for QueenMQ’s native runs 16 vCPU, 31 GB, Xeon Platinum 8358

All of them ran Ubuntu 24.04 in one VPC, with the same kernel settings on every machine for every system (common/os-tune.sh): swappiness 1, larger socket buffers and backlogs, a million open files, noatime, transparent huge pages left at Ubuntu’s madvise, and no IRQ pinning. Clocks were synced with chrony, each within a third of a millisecond of NTP. Only one system ran on the brokers at a time, and every start script refuses to start while another system’s ports are open.

The older Platinum broker caps a raft leader about 15 to 20% lower, so QueenMQ’s harness stops that node whenever it leads and starts it again as a follower, until another node leads. Kafka, Redpanda and Pulsar spread partition leadership over all three brokers, the older one included.

The load generators

QueenMQ’s native runs of September used goload, a Go program built on QueenMQ’s Go client. Kafka, Redpanda and Pulsar used kload (franz-go) and pload (pulsar-client-go), and the transactional runs added qload for QueenMQ. The three share one core copied from goload, so all four systems are driven and measured the same way.

The producers run open loop. Each unit, 100 messages for one partition, leaves at its scheduled instant, and a unit that would pass the in-flight cap of 5,000 per process is shed and counted, never queued, so a broker that slows down shows up as latency and shedding instead of as a smaller load. Producers never retry, since a retry would offer the same messages twice. Produce latency runs from a unit’s scheduled instant to its last acknowledgement, so time spent waiting to be sent counts.

Every message carries the instant its unit was scheduled, and e2e is the consumer’s receive time minus that instant, taken before the ack. The two ends are on different machines, so e2e includes the clock offset between them, a fraction of a millisecond here; the Kafka and Pulsar loaders also print a same-machine e2e with no offset at all, and it agreed with the cross-machine one. All of them send the same JSON payload, about 256 bytes (289 on average), and keep latencies in log-linear histograms with the same buckets, so their percentiles compare.

How one unit of load is timed. A producer schedules a unit of 100 messages for one partition at instant t0. It is sent if the process has fewer than 5,000 units in flight, otherwise it is shed and counted, never queued. The broker acknowledges it, and produce latency is the acknowledgement time minus t0. A consumer later receives the messages, which carry t0, and e2e latency is the receive time minus t0, taken before the ack.producerload machinebrokerthree nodesconsumerload machineunit scheduled at t0:100 messages, one partitionpush, if under 5,000 in flightotherwise shed and countedackproduce latency = ack − t0pop or fetchmessages carrying t0e2e = receive − t0,before the ack
Both latencies start at the scheduled instant, not at the send, so a broker that slows down shows up as latency and shedding instead of as a smaller load. Source: benchmark-queen/2026-09-30-kafka-pulsar (goload, kload, pload)

Every 10 seconds each process prints one line. A figure “over the last 30 seconds” is the three full windows before the process’s final one: rates are summed over the processes, a latency is the worst process’s. CPU is the system’s own processes, from 2-second samples over the loaded window, summed over the three brokers. A system fell behind when it consumed under 97% of the offered rate or its e2e p99 passed 5 seconds.

QueenMQ’s settings in the September runs

Each benchmark binary was built with Cargo’s fastrel profile, which is release without link-time optimization and with 16 codegen units, so it builds in minutes between runs; the published images use release, with thin LTO and one codegen unit. The cluster had three voters on the openraft replicator, with the data directory on the local disk and authentication and tenancy headers off. Every run used one tenant, so one raft group. Four settings were set explicitly, and three of them differ from the defaults (all are in the configuration reference):

Setting Default These runs Why
QUEEN_RAFT_LOG_CACHE_MB 512 4096 Applied entries leave the leader’s memory past this size even if a follower still needs them, and a follower behind the cache is fed from disk. At a million messages a second, 512 MB is a second or two of log.
QUEEN_RAFT_TXN_WINDOW_MIN_S 900 300 The least time a message’s hash list and its queue-log record are kept, so no log file is freed sooner. The leader wrote about 250 MB/s at a million messages a second, so 900 seconds of log would pass 200 GB a node.
QUEEN_QLOG_SHARDS 0 4 Every queue’s records go to one of four shared logs, so a group commit fsyncs at most four files however many queues it touches.
QUEEN_LANES 8 8; 16 in the transaction runs The planning threads that plan pushes in parallel (now QUEEN_PLANNER_THREADS).

The September matrix had deduplication off (dedupWindowSeconds: 0) and the soak a 60-second window. Completed messages were kept 1,800 seconds in the 60-second runs, so retention deleted nothing during a point, and 300 seconds in the soak. Leases were 30 seconds. Consumers popped up to 10 partitions and 1,000 messages at a time, long-polling for 2 seconds, and acked asynchronously with up to 256 acks in flight per process; in the soak the leader chose the pop width. There were 300 consumers on one queue, 1,008 at 1,000 queues and 10,008 at 10,000, where the other systems ran 198 consumers on 200 partitions and 297 above. Partitions were created before each timed run by pushing one message into each and draining it.

The other systems’ settings

SPEC.md in the Kafka and Pulsar folder is the contract the runs followed, with every setting and the reason for it, and the rule that tuning may change performance knobs only, never durability or ordering. The settings of the GCP runs are on the comparison; September’s, in short:

  • Kafka 4.3.1, KRaft with three combined broker and controller nodes. Replication factor 3, min.insync.replicas 2, acks=all, idempotent producers, lz4, 5 ms linger, 8 replica fetchers, 8 network and 16 I/O threads, message.max.bytes 16 MiB, a 6 GB heap (10 GB from 50,000 partitions) on G1, KIP-848 consumer groups, no fsync (page cache, Kafka’s production model). On the GCP nodes: XFS, 12 replica fetchers, 12 network and 24 I/O threads, writeback from 1 GiB.
  • Redpanda 26.2.3, production mode with rpk redpanda tune all and rpk iotune, write caching off, classic consumer groups (26.2.3 does not answer KIP-848), its partition memory limits raised for 75,000 partitions and more. Its tuners were undone before any other system ran.
  • Pulsar 4.2.4, ZooKeeper, a bookie and a broker on each node, ensemble 3, write quorum 3, ack quorum 2, journal fsync on, 48 namespace bundles, generational ZGC, producer batching of 5 ms or 1,000 messages with lz4, failover subscriptions. On the GCP nodes its memory was sized for 88 GB.

What each system acknowledges after, side by side, is on the comparison.

Running it again

Each folder below has the scripts that ran and a README or SPEC.md. Fill in the machines’ addresses first: cluster.env for QueenMQ’s harness, hosts.env for the others.

The GCP runs. VMs and disks from benchmark-queen/2026-10-15-bytes-per-msg/scripts/gcp/ (hosts.sh filled in from create.sh’s table), QueenMQ from benchmark-queen/2026-10-12-three-node-new-forms/scripts/ with the broker binary in /root/p3/bin/<name>/queen on each broker, the others from benchmark-queen/2026-09-30-kafka-pulsar/:

OWNER=<you> GCP_PROJECT=<project> ./create.sh && ./ssh-ready.sh && ./disks.sh && ./fsync-all.sh
QBIN=240 PFX=a24 RATES=3000000 ./grid3.sh 10 100 1000 10000 100000 1000000   # Queen, a fresh cluster per point
kafka/run.sh 1x1000-3000k 3000000 PARTITIONS=1000 CONSUMERS=33 KPROPS=num.io.threads=24,num.network.threads=12,num.replica.fetchers=12
pulsar/run.sh 1x1000-3000k 3000000 PARTITIONS=1000 SUB_TYPE=failover CONSUMERS=33
redpanda/run.sh 1x1000-3000k 3000000 PARTITIONS=1000 CONSUMERS=33 && redpanda/cluster.sh untune

The soak and the September runs of QueenMQ, from benchmark-queen/2026-09-30-stage03-3node/, with the broker binary in /root/qr/ on each broker:

./setup30.sh                    # scripts and goload onto every machine
BIN=/root/qr/queen-b1 SECS=36000 ./soak.sh soak6   # 10 hours, a fault every 90 minutes
./soak-status.sh soak6                             # the latest window of every process
./soak-collect.sh soak6                            # fetch everything into runs/soak6

Kafka, Redpanda, Pulsar and the transactions of September (and the deploy step of the GCP runs), from benchmark-queen/2026-09-30-kafka-pulsar/:

(cd mqload && GOWORK=off ./build.sh)
common/deploy.sh harness tune kafka pulsar redpanda clock check
kafka/grid.sh && report/report.py runs/kafka/* --loaders
redpanda/grid.sh && redpanda/cluster.sh stop && redpanda/cluster.sh untune
pulsar/grid.sh
txn/deploy.sh && txn/grid.sh && report/report.py --txn runs/txn/*/*

The archive

Folder under benchmark-queen/ What it holds
2026-10-15-bytes-per-msg/ QueenMQ 2.4 on the GCP nodes: the 3,000,000 msg/s and 1 KB matrices (runs/gcp/fm3-*, f1k-*), the ceiling (fxc-*), the A/B against 2.3.0, the scripts that made the VMs, and the measurement of what each message writes
2026-10-12-three-node-new-forms/ QueenMQ’s harness for the GCP runs and QueenMQ 2.4’s points of 2026-10-16 (runs/a24-*); the 2.3.0 runs: README.md, NIGHT.md, the DigitalOcean matrix rows (runs/grid-final.jsonl), 2.2.0 on the same machines, the transactions, and the probes that found each stall
2026-09-30-stage03-3node/ QueenMQ’s harness for 2026-09-30 and 10-01 and every QueenMQ run on these pages: the 2.0.0-beta.2 matrix (runs/mb2c-*) and the soaks (runs/soak1 to soak5), with the goload source
2026-09-30-kafka-pulsar/ SPEC.md, the runbook, the mqload loaders (qload among them), each system’s install and configuration scripts, report/report.py, and the Kafka, Redpanda, Pulsar, transaction and Kafka-client runs; the folders tagged -gcp24 are the GCP runs (Kafka’s of 10-14, Pulsar’s and Redpanda’s 1x<partitions>-3000k-gcp24 of 10-16), and -oct13 and b1- the 2026-10-13 runs on the second set of machines
2026-09-29-1m-3node/ the 2026-09-29 runs at 1M msg/s on 500,000 partitions, deduplication on and off, and their harness
2026-09-27-jepsen-archive/ Jepsen P8 to P10: every run, the Jepsen stores and the binaries they tested

Three older 2.0 folders are kept and not quoted, because the engine they measured has been rewritten since: 2026-09-23-raft-32core/ (one 32-vCPU node), 2026-09-24-raft-3node/ (QueenMQ and Kafka by partition count at 300,000 msg/s on 32-vCPU nodes) and 2026-09-25-queue-count/ (one node, the queue-count problem that led to shared queue logs). Results of QueenMQ 1.x, whose storage was PostgreSQL, are in the git history of this site.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close