Every number in this section comes from a folder of logs, and most can be produced again with one command. This page is how: the machines, the load generators and what they count, the settings each system ran with, and the scripts.
Three sets of machines
The section has three generations of runs, and no page compares a number across two of them.
- GCP, 2026-10-14 to 16. QueenMQ 2.4 against Kafka 4.3.1, Pulsar 4.2.4 and Redpanda 26.2.3: the comparison and partition count.
- DigitalOcean, 2026-10-12 and 13. Pre-release builds of 2.3.0: the transactions against 2.2.0 and five Kafka-client points.
- DigitalOcean, September and 2026-10-01 and 02. QueenMQ 2.0.0-beta builds with Kafka, Redpanda and Pulsar in transactions, the soak, the first Kafka-client runs, and Laravel on one node.
The two DigitalOcean sets are six droplets of the same size, not the same droplets.
The GCP runs
| Role | Machine type | CPU | Memory | Disk for the data |
|---|---|---|---|---|
| 3 brokers | c4-standard-24-lssd |
24 vCPU, Intel Xeon 6985P-C | 88 GB | four local NVMe disks (4 × 375 GB) in one RAID0, XFS noatime |
| 3 load generators, 6 for Pulsar’s largest points | c4-highcpu-16 |
16 vCPU, Intel Xeon Platinum 8581C | 32 GB | boot disk only |
One zone (europe-west1-b), one VPC, Ubuntu 24.04, the same kernel settings for every system,
clocks synced with chrony. The VMs were made three times, on 2026-10-14, 15 and 16, and every
broker’s disks were measured before use: a 32 KB write and fdatasync took 0.05 to 0.06 ms at the
median on every broker of every set, 16,000 to 17,700 a second. A local SSD that has been written
for hours without TRIM slows down (on 10-14 one disk of one broker went to 30 ms a write), so the
RAID gets an fstrim before a series. On 10-16 QueenMQ and Kafka ran their 1,000-partition point
again and read what they had read on the earlier sets.
QueenMQ ran as 2.4.0-beta.1 (the image ghcr.io/queen-mq/queen:2.4.0-beta.1; the runs of 10-15 are
the same code built before the version commit), three voters, one tenant and one raft group,
every broker setting at its default except QUEEN_CONSUME_IDLE_UNLOAD_S=30 at ten million
partitions. Its load generator turns on the wire forms that are off by default in every client
(-push-wire bin -push-zstd -ack-counts -pop-frames: the binary push, a client-compressed group,
acks answered with counts, framed pops). Each queue had deduplication off and kept completed
messages 1,800 seconds, so nothing was deleted during a point.
A point is a fresh cluster with its partitions created and warmed, 70 seconds of load with a 10-second ramp in them (90 for QueenMQ’s runs of 10-16), nine load processes. A latency is the median of the three 10-second windows in seconds 30 to 60, each window taking the slowest process, with the worst window in brackets; CPU is the three brokers’ cores over the loaded window. A point is carried when at least 99.5% of the offered rate is pushed, nothing is shed or errs, and 99% of what was pushed is consumed. Each system’s settings are on the comparison.
The 2.3.0 runs on DigitalOcean
The transactions against 2.2.0 and the Kafka-client points of 2026-10-12 and 13 ran here.
| Role | Machine | CPU | 32 KB write and fdatasync, p50 / p99 |
|---|---|---|---|
| Broker, the leader | queen |
Xeon Gold 6548N | 0.08 / 0.29 ms |
| Broker | queen-02 |
Xeon Platinum 8358 | 0.10 / 0.40 ms |
| Broker | queen-03 |
Xeon Platinum 8358 | 0.09 / 0.37 ms |
| Load | queen-04 |
Xeon Gold 6548N | 0.72 to 0.94 / 6.3 to 14.4 ms |
| Load | queen-05 |
Xeon Gold 6548N | 0.76 to 0.95 / 2.6 to 3.2 ms |
| Load | queen-06 |
Xeon Platinum 8358 | 0.20 / 1.17 ms |
Each has 16 vCPU and 31 GB, Ubuntu 24.04 and one ext4 disk. The roles follow the disks: two of the
three faster CPUs sync ten times slower than the others, and the first runs, with those two as
followers, read a push p99 of 242 ms at 300,000 msg/s that belonged to one follower’s clients
alone. With the three fast disks as brokers the same build read 5 ms. Measure every disk before
giving roles (scripts/fsync.py in the archive).
QueenMQ ran as release builds (cargo build --release --locked, the profile of the published
image) of the commits that became 2.3.0, three voters, one tenant and one raft group, every
broker setting at its default, with the brokers’ kernel keeping 512 MB free
(vm.min_free_kbytes, with vm.watermark_scale_factor 200). The load generator is qload, nine
processes on the three load machines, open loop with the rules below.
# from benchmark-queen/2026-10-12-three-node-new-forms/scripts, hosts.env filled in
./txnrun.sh txn-bin-10 bin 10 40 60000 120000 180000 # transactions of 10, binary body
QBIN=220 ./txnrun.sh txn-220-10 json 10 40 60000 120000 180000 # the same on 2.2.0The folder’s README.md has the runs of 2026-10-12 (2.2.0 against the new wire forms, what stops
one raft group, the transactions) and NIGHT.md the log of the night that followed: each stall,
its cause, the build that removed it and the tables of every matrix.
The machines of the September runs
| Role | Count | Shape |
|---|---|---|
| Brokers | 3 | DigitalOcean, 16 vCPU, 31 GB, one local disk with ext4. Two Xeon Gold 6548N, one Xeon Platinum 8358 |
| Load machines | 3, and a fourth for QueenMQ’s native runs | 16 vCPU, 31 GB, Xeon Platinum 8358 |
All of them ran Ubuntu 24.04 in one VPC, with the same kernel settings on every machine for every
system (common/os-tune.sh): swappiness 1, larger socket buffers and backlogs, a million open
files, noatime, transparent huge pages left at Ubuntu’s madvise, and no IRQ pinning. Clocks
were synced with chrony, each within a third of a millisecond of NTP. Only one system ran on the
brokers at a time, and every start script refuses to start while another system’s ports are open.
The older Platinum broker caps a raft leader about 15 to 20% lower, so QueenMQ’s harness stops that node whenever it leads and starts it again as a follower, until another node leads. Kafka, Redpanda and Pulsar spread partition leadership over all three brokers, the older one included.
The load generators
QueenMQ’s native runs of September used goload, a Go program built on QueenMQ’s Go client. Kafka, Redpanda and
Pulsar used kload (franz-go) and pload (pulsar-client-go), and the transactional runs added
qload for QueenMQ. The three share one core copied from goload, so all four systems are driven
and measured the same way.
The producers run open loop. Each unit, 100 messages for one partition, leaves at its scheduled instant, and a unit that would pass the in-flight cap of 5,000 per process is shed and counted, never queued, so a broker that slows down shows up as latency and shedding instead of as a smaller load. Producers never retry, since a retry would offer the same messages twice. Produce latency runs from a unit’s scheduled instant to its last acknowledgement, so time spent waiting to be sent counts.
Every message carries the instant its unit was scheduled, and e2e is the consumer’s receive time minus that instant, taken before the ack. The two ends are on different machines, so e2e includes the clock offset between them, a fraction of a millisecond here; the Kafka and Pulsar loaders also print a same-machine e2e with no offset at all, and it agreed with the cross-machine one. All of them send the same JSON payload, about 256 bytes (289 on average), and keep latencies in log-linear histograms with the same buckets, so their percentiles compare.
benchmark-queen/2026-09-30-kafka-pulsar (goload, kload, pload)Every 10 seconds each process prints one line. A figure “over the last 30 seconds” is the three full windows before the process’s final one: rates are summed over the processes, a latency is the worst process’s. CPU is the system’s own processes, from 2-second samples over the loaded window, summed over the three brokers. A system fell behind when it consumed under 97% of the offered rate or its e2e p99 passed 5 seconds.
QueenMQ’s settings in the September runs
Each benchmark binary was built with Cargo’s fastrel profile, which is release without link-time
optimization and with 16 codegen units, so it builds in minutes between runs; the published images
use release, with thin LTO and one codegen unit. The cluster had three voters on the openraft
replicator, with the data directory on the local disk and authentication and tenancy headers off.
Every run used one tenant, so one raft group. Four settings were set explicitly, and three of them
differ from the defaults (all are in the configuration reference):
| Setting | Default | These runs | Why |
|---|---|---|---|
QUEEN_RAFT_LOG_CACHE_MB |
512 | 4096 | Applied entries leave the leader’s memory past this size even if a follower still needs them, and a follower behind the cache is fed from disk. At a million messages a second, 512 MB is a second or two of log. |
QUEEN_RAFT_TXN_WINDOW_MIN_S |
900 | 300 | The least time a message’s hash list and its queue-log record are kept, so no log file is freed sooner. The leader wrote about 250 MB/s at a million messages a second, so 900 seconds of log would pass 200 GB a node. |
QUEEN_QLOG_SHARDS |
0 | 4 | Every queue’s records go to one of four shared logs, so a group commit fsyncs at most four files however many queues it touches. |
QUEEN_LANES |
8 | 8; 16 in the transaction runs | The planning threads that plan pushes in parallel (now QUEEN_PLANNER_THREADS). |
The September matrix had deduplication off (dedupWindowSeconds: 0) and the soak a
60-second window. Completed messages were kept 1,800 seconds in the 60-second runs, so retention
deleted nothing during a point, and 300 seconds in the soak. Leases were 30 seconds. Consumers
popped up to 10 partitions and 1,000 messages at a time, long-polling for 2 seconds, and acked
asynchronously with up to 256 acks in flight per process; in the soak the leader chose the pop
width. There were 300 consumers on one queue, 1,008 at 1,000 queues and 10,008 at 10,000, where the
other systems ran 198 consumers on 200 partitions and 297 above. Partitions were created before
each timed run by pushing one message into each and draining it.
The other systems’ settings
SPEC.md in the Kafka and Pulsar folder is the contract the runs followed, with every setting and
the reason for it, and the rule that tuning may change performance knobs only, never durability or
ordering. The settings of the GCP runs are on
the comparison; September’s, in short:
- Kafka 4.3.1, KRaft with three combined broker and controller nodes. Replication factor 3,
min.insync.replicas2,acks=all, idempotent producers, lz4, 5 ms linger, 8 replica fetchers, 8 network and 16 I/O threads,message.max.bytes16 MiB, a 6 GB heap (10 GB from 50,000 partitions) on G1, KIP-848 consumer groups, no fsync (page cache, Kafka’s production model). On the GCP nodes: XFS, 12 replica fetchers, 12 network and 24 I/O threads, writeback from 1 GiB. - Redpanda 26.2.3, production mode with
rpk redpanda tune allandrpk iotune, write caching off, classic consumer groups (26.2.3 does not answer KIP-848), its partition memory limits raised for 75,000 partitions and more. Its tuners were undone before any other system ran. - Pulsar 4.2.4, ZooKeeper, a bookie and a broker on each node, ensemble 3, write quorum 3, ack quorum 2, journal fsync on, 48 namespace bundles, generational ZGC, producer batching of 5 ms or 1,000 messages with lz4, failover subscriptions. On the GCP nodes its memory was sized for 88 GB.
What each system acknowledges after, side by side, is on the comparison.
Running it again
Each folder below has the scripts that ran and a README or SPEC.md. Fill in the machines’
addresses first: cluster.env for QueenMQ’s harness, hosts.env for the others.
The GCP runs. VMs and disks from benchmark-queen/2026-10-15-bytes-per-msg/scripts/gcp/
(hosts.sh filled in from create.sh’s table), QueenMQ from
benchmark-queen/2026-10-12-three-node-new-forms/scripts/ with the broker binary in
/root/p3/bin/<name>/queen on each broker, the others from benchmark-queen/2026-09-30-kafka-pulsar/:
OWNER=<you> GCP_PROJECT=<project> ./create.sh && ./ssh-ready.sh && ./disks.sh && ./fsync-all.sh
QBIN=240 PFX=a24 RATES=3000000 ./grid3.sh 10 100 1000 10000 100000 1000000 # Queen, a fresh cluster per point
kafka/run.sh 1x1000-3000k 3000000 PARTITIONS=1000 CONSUMERS=33 KPROPS=num.io.threads=24,num.network.threads=12,num.replica.fetchers=12
pulsar/run.sh 1x1000-3000k 3000000 PARTITIONS=1000 SUB_TYPE=failover CONSUMERS=33
redpanda/run.sh 1x1000-3000k 3000000 PARTITIONS=1000 CONSUMERS=33 && redpanda/cluster.sh untuneThe soak and the September runs of QueenMQ, from benchmark-queen/2026-09-30-stage03-3node/, with
the broker binary in /root/qr/ on each broker:
./setup30.sh # scripts and goload onto every machine
BIN=/root/qr/queen-b1 SECS=36000 ./soak.sh soak6 # 10 hours, a fault every 90 minutes
./soak-status.sh soak6 # the latest window of every process
./soak-collect.sh soak6 # fetch everything into runs/soak6Kafka, Redpanda, Pulsar and the transactions of September (and the deploy step of the GCP runs), from benchmark-queen/2026-09-30-kafka-pulsar/:
(cd mqload && GOWORK=off ./build.sh)
common/deploy.sh harness tune kafka pulsar redpanda clock check
kafka/grid.sh && report/report.py runs/kafka/* --loaders
redpanda/grid.sh && redpanda/cluster.sh stop && redpanda/cluster.sh untune
pulsar/grid.sh
txn/deploy.sh && txn/grid.sh && report/report.py --txn runs/txn/*/*The archive
Folder under benchmark-queen/ |
What it holds |
|---|---|
2026-10-15-bytes-per-msg/ |
QueenMQ 2.4 on the GCP nodes: the 3,000,000 msg/s and 1 KB matrices (runs/gcp/fm3-*, f1k-*), the ceiling (fxc-*), the A/B against 2.3.0, the scripts that made the VMs, and the measurement of what each message writes |
2026-10-12-three-node-new-forms/ |
QueenMQ’s harness for the GCP runs and QueenMQ 2.4’s points of 2026-10-16 (runs/a24-*); the 2.3.0 runs: README.md, NIGHT.md, the DigitalOcean matrix rows (runs/grid-final.jsonl), 2.2.0 on the same machines, the transactions, and the probes that found each stall |
2026-09-30-stage03-3node/ |
QueenMQ’s harness for 2026-09-30 and 10-01 and every QueenMQ run on these pages: the 2.0.0-beta.2 matrix (runs/mb2c-*) and the soaks (runs/soak1 to soak5), with the goload source |
2026-09-30-kafka-pulsar/ |
SPEC.md, the runbook, the mqload loaders (qload among them), each system’s install and configuration scripts, report/report.py, and the Kafka, Redpanda, Pulsar, transaction and Kafka-client runs; the folders tagged -gcp24 are the GCP runs (Kafka’s of 10-14, Pulsar’s and Redpanda’s 1x<partitions>-3000k-gcp24 of 10-16), and -oct13 and b1- the 2026-10-13 runs on the second set of machines |
2026-09-29-1m-3node/ |
the 2026-09-29 runs at 1M msg/s on 500,000 partitions, deduplication on and off, and their harness |
2026-09-27-jepsen-archive/ |
Jepsen P8 to P10: every run, the Jepsen stores and the binaries they tested |
Three older 2.0 folders are kept and not quoted, because the engine they measured has been rewritten
since: 2026-09-23-raft-32core/ (one 32-vCPU node), 2026-09-24-raft-3node/ (QueenMQ and Kafka by
partition count at 300,000 msg/s on 32-vCPU nodes) and 2026-09-25-queue-count/ (one node, the
queue-count problem that led to shared queue logs). Results of QueenMQ 1.x, whose storage was
PostgreSQL, are in the git history of this site.