How we measure

Every number in this section comes from a folder of logs, and most can be produced again with one command. This page is how: the machines, the load generators and what they count, the settings each system ran with, and the scripts.

The machines

All of them ran Ubuntu 24.04 in one VPC, with the same kernel settings on every machine for every system (common/os-tune.sh): swappiness 1, larger socket buffers and backlogs, a million open files, noatime, transparent huge pages left at Ubuntu’s madvise, and no IRQ pinning. Clocks were synced with chrony, each within a third of a millisecond of NTP. Only one system ran on the brokers at a time, and every start script refuses to start while another system’s ports are open.

The older Platinum broker caps a raft leader about 15 to 20% lower, so Queen’s harness stops that node whenever it leads and starts it again as a follower, until another node leads. Kafka, Redpanda and Pulsar spread partition leadership over all three brokers, the older one included.

The load generators

Queen’s native runs used goload, a Go program built on Queen’s Go client. Kafka, Redpanda and Pulsar used kload (franz-go) and pload (pulsar-client-go), and the transactional runs added qload for Queen. The three share one core copied from goload, so all four systems are driven and measured the same way.

The producers run open loop. Each unit, 100 messages for one partition, leaves at its scheduled instant, and a unit that would pass the in-flight cap of 5,000 per process is shed and counted, never queued, so a broker that slows down shows up as latency and shedding instead of as a smaller load. Producers never retry, since a retry would offer the same messages twice. Produce latency runs from a unit’s scheduled instant to its last acknowledgement, so time spent waiting to be sent counts.

Every message carries the instant its unit was scheduled, and e2e is the consumer’s receive time minus that instant, taken before the ack. The two ends are on different machines, so e2e includes the clock offset between them, a fraction of a millisecond here; the Kafka and Pulsar loaders also print a same-machine e2e with no offset at all, and it agreed with the cross-machine one. All of them send the same JSON payload, about 256 bytes (289 on average), and keep latencies in log-linear histograms with the same buckets, so their percentiles compare.

Every 10 seconds each process prints one line. A figure “over the last 30 seconds” is the three full windows before the process’s final one: rates are summed over the processes, a latency is the worst process’s. CPU is the system’s own processes, from 2-second samples over the loaded window, summed over the three brokers. A system fell behind when it consumed under 97% of the offered rate or its e2e p99 passed 5 seconds.

Queen’s settings

Each benchmark binary was built with Cargo’s fastrel profile, which is release without link-time optimization and with 16 codegen units, so it builds in minutes between runs; the published images use release, with thin LTO and one codegen unit. The cluster had three voters on the openraft replicator, with the data directory on the local disk and authentication and tenancy headers off. Every run used one tenant, so one raft group. Four settings were set explicitly, and three of them differ from the defaults (all are in the configuration reference):

The throughput and partition runs had deduplication off (dedupWindowSeconds: 0) and the soak a 60-second window. Completed messages were kept 1,800 seconds in the 60-second runs, so retention deleted nothing during a point, and 300 seconds in the soak. Leases were 30 seconds. Consumers popped up to 10 partitions and 1,000 messages at a time, long-polling for 2 seconds, and acked asynchronously with up to 256 acks in flight per process; in the soak the leader chose the pop width. There were 300 consumers on one queue, 1,008 at 1,000 queues and 10,008 at 10,000, where the other systems ran 198 consumers on 200 partitions and 297 above. Partitions were created before each timed run by pushing one message into each and draining it.

The other systems’ settings

SPEC.md in the Kafka and Pulsar folder is the contract the runs followed, with every setting and the reason for it, and the rule that tuning may change performance knobs only, never durability or ordering. In short:

• Kafka 4.3.1, KRaft with three combined broker and controller nodes. Replication factor 3, min.insync.replicas 2, acks=all, idempotent producers, lz4, 5 ms linger, 8 replica fetchers, 8 network and 16 I/O threads, message.max.bytes 16 MiB, a 6 GB heap (10 GB from 50,000 partitions) on G1, KIP-848 consumer groups, no fsync (page cache, Kafka’s production model).
• Redpanda 26.2.3, production mode with rpk redpanda tune all and rpk iotune, write caching off, classic consumer groups (26.2.3 does not answer KIP-848), its partition memory limits raised for 75,000 partitions and more. Its tuners were undone before any other system ran.
• Pulsar 4.2.4, ZooKeeper, a bookie and a broker on each node, ensemble 3, write quorum 3, ack quorum 2, journal fsync on, 48 namespace bundles, generational ZGC, producer batching of 5 ms or 1,000 messages with lz4, failover subscriptions.

What each system acknowledges after, side by side, is on throughput and latency.

Running it again

Each folder below has the scripts that ran and a README or SPEC.md. Fill in the machines’ addresses first: cluster.env for Queen’s harness, hosts.env for the others.

Queen’s matrix and single points, from benchmark-queen/2026-09-30-stage03-3node/, with the broker binary in /root/qr/ on each broker:

./setup30.sh                    # scripts and goload onto every machine
./campM.sh queen-b2 mb2c        # the 1M matrix: 60 s per point, deduplication off, pops of 10 partitions
DEDUP=0 GX="-pop-partitions 10" NL=4 RATE=1000000 ./run30.sh mb2c-q1-t10m 1 10000000 queen-b2 60
python3 grid_report.py runs mb2c-q1-t200-r1m mb2c-q1-t10m

The soak, from the same folder:

BIN=/root/qr/queen-b1 SECS=36000 ./soak.sh soak6   # 10 hours, a fault every 90 minutes
./soak-status.sh soak6                             # the latest window of every process
./soak-collect.sh soak6                            # fetch everything into runs/soak6

Kafka, Redpanda, Pulsar and the transactions, from benchmark-queen/2026-09-30-kafka-pulsar/:

(cd mqload && GOWORK=off ./build.sh)
common/deploy.sh harness tune kafka pulsar redpanda clock check
kafka/grid.sh && report/report.py runs/kafka/* --loaders
redpanda/grid.sh && redpanda/cluster.sh stop && redpanda/cluster.sh untune
pulsar/grid.sh
txn/deploy.sh && txn/grid.sh && report/report.py --txn runs/txn/*/*

The archive

Three older 2.0 folders are kept and not quoted, because the engine they measured has been rewritten since: 2026-09-23-raft-32core/ (one 32-vCPU node), 2026-09-24-raft-3node/ (Queen and Kafka by partition count at 300,000 msg/s on 32-vCPU nodes) and 2026-09-25-queue-count/ (one node, the queue-count problem that led to shared queue logs). Results of Queen 1.x, whose storage was PostgreSQL, are in the git history of this site.