Skip to content

Partition scaling to 10M vs Kafka, Pulsar, Redpanda

One queue from 10 to 10,000,000 partitions on three GCP nodes with local NVMe: QueenMQ 2.4 at 3,000,000 msg/s of 256-byte messages and 1,000,000 msg/s of 1 KB messages, beside Kafka 4.3.1, Pulsar 4.2.4 and Redpanda 26.2.3 on the same machines: latency, CPU, disk, and what a partition costs QueenMQ in memory.

Updated View as Markdown

This is the test QueenMQ was built for. Give every entity its own partition and your partition count is your entity count: a million customers, ten million orders. On three GCP nodes with 24 vCPU and local NVMe disks, QueenMQ 2.4 carried 3,000,000 messages a second of 256 bytes in and out of one queue at every partition count from 10 to 10,000,000, with an end-to-end p99 of 15 to 25 ms on 21 to 28 cores, and 1,000,000 messages a second of 1 KB at 16 to 22 ms. On the same machines Kafka was the fastest up to 1,000 partitions and did not carry 50,000, Redpanda read 782 ms at 10,000 and did not carry 50,000, and Pulsar held 20 to 25 ms up to 50,000 and did not carry 100,000 (2026-10-14 to 16). Compare explains how each system gets order per entity.

The shape

The load of QueenMQ vs Kafka, Pulsar and Redpanda on NVMe, where the machines and every setting are: 256-byte JSON messages pushed 100 at a time to one partition by nine open-loop load processes, a fresh cluster for every point, 70 seconds of load (90 for QueenMQ’s runs of 2026-10-16 at 1,000 and 50,000 partitions). A latency is the median of the three 10-second windows in seconds 30 to 60, each window the slowest load process, with the worst window’s p99 in brackets. Every partition existed before the timed run: QueenMQ’s load generator pushed one message into each partition and drained it, and the other systems’ topics were created and warmed first. No system was measured while it created partitions.

QueenMQ ran as 2.4.0-beta.1 with the binary push, compressed groups, framed pops and counted acks, deduplication off and nothing deleted during a point; 9 consumers read 10 partitions, 99 read 100, and 297 read every larger count with pops of up to 10 partitions and 1,000 messages. The ten-million column ran with QUEEN_CONSUME_IDLE_UNLOAD_S=30. Every point was carried: nothing shed, no error, and at least 99% of what was pushed consumed.

Every count at 3,000,000 msg/s

QueenMQ 2.4 with 256-byte messages: end-to-end p50 / p99 in milliseconds, the cores of the three brokers (the leader’s and the two followers’ in brackets), the resident memory of the leader and of a follower at the end of the point, and the mean write rate of a broker’s disks with the bytes it wrote for each message.

Partitions e2e p50 / p99 Cores Leader, follower Disk a broker
10 10 / 24 (27) 21.0 (10.9, 5.0, 5.1) 5.2, 4.8 GB 416 MB/s, 138 B a message
100 7.5 / 17 (17) 27.7 (13.9, 6.8, 7.0) 5.1, 4.6 GB 418 MB/s, 139 B
1,000 6.9 / 16.5 (19) 24.6 (12.8, 5.9, 5.9) 6.4, 5.9 GB 443 MB/s, 147 B
10,000 7.1 / 20 (26) 24.8 (12.8, 6.0, 6.0) 5.6, 4.9 GB 561 MB/s, 187 B
50,000 7.7 / 20 (23) 24.8 (13.0, 5.9, 5.9) 7.0, 6.1 GB 497 MB/s, 166 B
100,000 7.1 / 15 (17) 25.0 (13.0, 5.9, 6.1) 6.0, 5.0 GB 468 MB/s, 156 B
1,000,000 7.5 / 20 (21) 24.9 (13.0, 5.9, 6.0) 7.7, 5.8 GB 438 MB/s, 146 B
10,000,000 8.9 / 25 (29) 25.4 (13.3, 6.1, 6.0) 17.2, 14.2 GB 430 MB/s, 143 B
p99 end-to-end latency at 3,000,000 msg/s against the partition count, both on log scales, on three GCP nodes with local NVMe. QueenMQ 2.4 stays between 15 and 25 ms from 10 to 10,000,000 partitions. Kafka reads 7.5 to 11 ms up to 1,000 partitions, 21 at 10,000 and 105 at 25,000, and does not carry 50,000 (32 s) or 100,000 (38 s). Pulsar reads 20 to 25 ms up to 50,000 and does not carry 100,000 (28 s). Redpanda reads 14 to 44 ms up to 1,000 partitions and 782 at 10,000, and does not carry 50,000 (8.8 s).p99 end-to-end latency10 ms100 ms1 s10 s101001k10k100k1M10Mpartitions in the queue or topicQueenMQ 2.4Kafka 4.3.1Pulsar 4.2.4Redpanda 26.2.3
3,000,000 msg/s into one queue or topic, p99 end to end. QueenMQ 2.4 stays at 15 to 25 ms from 10 partitions to ten million. Hollow: the system did not carry the rate. Kafka, Pulsar and Redpanda were not run past 100,000 partitions. Source: benchmark-queen/2026-10-15-bytes-per-msg/runs/gcp (fm3-*), benchmark-queen/2026-10-12-three-node-new-forms/runs (a24-*), benchmark-queen/2026-09-30-kafka-pulsar/runs (kafka, pulsar, redpanda)

From 100 partitions to ten million the median stays between 6.9 and 8.9 ms and the p99 between 15 and 25 ms, on about 13 cores of the leader and 6 of each follower. Ten partitions are the slowest row: a partition goes to one consumer at a time, so nine consumers take more than 300,000 messages a second each. The 1,000- and 50,000-partition rows ran 90 seconds instead of 70, so their nodes held 20 seconds more of messages’ rows (nothing is deleted during a point). Six runs of the 1,000-partition point on two days read a p99 of 15, 16.5, 17, 18 and 20 ms, and once 65 ms, in a run where the log’s group commit settled on fewer, larger groups from the first seconds; why, we did not find. A broker wrote 138 to 187 bytes for each message; Kafka’s wrote 147 at 1,000 partitions.

With 1 KB messages at 1,000,000 msg/s

The same counts with 1 KB JSON messages (1,027 bytes on average), about 1 GB/s of messages, everything else as above. Kafka’s numbers for this load are on the comparison; Pulsar and Redpanda were not run with 1 KB messages.

Partitions e2e p50 / p99 Cores Leader, follower Disk a broker
10 9.7 / 20 (24) 15.2 2.3, 1.9 GB 539 MB/s
100 6.7 / 19 (19) 18.2 2.4, 1.9 GB 541 MB/s
1,000 6.8 / 16 (17) 17.1 3.3, 2.1 GB 559 MB/s
10,000 6.9 / 16.5 (17) 17.4 3.7, 2.1 GB 580 MB/s
100,000 6.8 / 20 (21) 17.5 4.2, 2.2 GB 551 MB/s
1,000,000 7.4 / 17 (17) 17.4 6.5, 3.1 GB 544 MB/s
10,000,000 7.0 / 22 (25) 17.7 14.1, 11.5 GB 546 MB/s

The median is 6.7 to 7.4 ms from 100 partitions up and the p99 16 to 22 ms at every count, on 15 to 18 cores. Kafka with 1 KB messages read 9.4 ms at p99 up to 1,000 partitions and 19 ms at 10,000, on 31 cores against QueenMQ’s 17, and did not carry 50,000.

The other systems on the same machines

Kafka 4.3.1, Pulsar 4.2.4 and Redpanda 26.2.3 at 3,000,000 msg/s of 256-byte messages, on the same machine types, each with the settings on the comparison. p50 / p99 in milliseconds (worst window in brackets) and the cores of the three brokers; Redpanda’s cores are its shards’ busy time, because its reactors poll and the operating system counts 66 to 71 of the 72.

Partitions Kafka Pulsar Redpanda
10 6.4 / 11 (45), 14.9 13 / 25 (26), 5.4 14 / 44 (45), 8.8
100 6.0 / 7.5 (8), 18.5 13 / 25 (26), 11.4 6.9 / 14 (14), 19.9
1,000 6.0 / 8.1 (8), 21.0 6.2 / 20 (21), 15.1 6.8 / 14 (20), 22.4
10,000 8.4 / 21 (78), 52.8 6.7 / 20 (21), 17.2 68 / 782 (815), 30.7
25,000 58 / 105 (146), 57.0 not run not run
50,000 not carried: 601,000 msg/s pushed, p99 32 s; 50.2 6.7 / 21 (21), 21.7 not carried: 2,820,000 msg/s pushed, p99 8.8 s; 33.8
100,000 not carried: 174,000 msg/s pushed, p99 38 s; 60.5 not carried: 2,520,000 msg/s pushed, p99 28 s; 22.8 not run
Cores used by the three brokers together at 3,000,000 msg/s against the partition count (log scale). QueenMQ 2.4 uses 21 to 28 cores from 10 to 10,000,000 partitions. Kafka uses 15 to 21 up to 1,000, 53 at 10,000 and 57 at 25,000, and 50 and 61 at 50,000 and 100,000, where it did not carry the rate. Pulsar uses 5 to 22 up to 50,000 and 23 at 100,000, where it did not carry the rate. Redpanda's shards are busy 9 to 31 cores up to 10,000 and 34 at 50,000, where it did not carry the rate.cores, three brokers together (of 72)0204060101001k10k100k1M10Mpartitions in the queue or topicQueenMQ 2.4Kafka 4.3.1Pulsar 4.2.4Redpanda 26.2.3 (busy shards)
QueenMQ's CPU does not follow the partition count. Kafka's more than doubles from 1,000 partitions to 10,000; Redpanda's grows from 9 to 31 busy cores; Pulsar uses the fewest wherever it carried the rate. Hollow: not carried. Source: the grid rows of the QueenMQ runs; report/report.py over the Kafka, Pulsar and Redpanda runs, Redpanda's busy.txt

Kafka is the fastest up to 1,000 partitions, where it answers a write from memory and the others wait for two fsyncs. From there its CPU follows the count: 21 cores at 1,000, 53 at 10,000 for a p99 of 21 ms, 57 at 25,000 for 105 ms, and at 50,000 and 100,000 it pushed 601,000 and 174,000 of the 3,000,000 messages a second. Its topic of 100,000 partitions took about four minutes to create. Redpanda reads 14 ms at 100 and 1,000 partitions and 782 ms at 10,000, and fell behind at 50,000. Pulsar uses the fewest cores and holds 20 to 25 ms up to 50,000 partitions; at 50,000 and 100,000 its Go client needed six load generators instead of three, and at 100,000 it pushed 2,520,000 of the 3,000,000 with its brokers at 23 of 72 cores and the load generators at about 13 of their 16, so these runs do not tell whether its brokers or its client stopped it. None of the three was run past 100,000 partitions.

What a partition costs

Memory is the one thing in QueenMQ that follows the partition count. From 10 partitions to ten million the leader went from 5.2 to 17.2 GB and a follower from 4.8 to 14.2: about 1.2 KB per partition on the leader and 0.9 KB on a follower, with 1 KB messages the same. The leader holds a little more because it also keeps each consumer group’s state for the partitions it has read, for QUEEN_CONSUME_IDLE_UNLOAD_S after their last use (600 s by default; the ten-million column ran with 30 s). Up to 100,000 partitions the difference is under a gigabyte. Memory is what sets the partition ceiling: ten million partitions took 17 GB of a node’s 88.

Resident memory of QueenMQ 2.4's nodes against the partition count (log scale). At 3,000,000 msg/s of 256-byte messages the leader holds 5.1 to 6.0 GB from 10 to 100,000 partitions, 7.7 GB at 1,000,000 and 17.2 GB at 10,000,000, and a follower 4.6 to 5.0 GB, 5.8 GB and 14.2 GB. At 1,000,000 msg/s of 1 KB messages the leader holds 2.3 to 4.2 GB up to 100,000 partitions, 6.5 GB at 1,000,000 and 14.1 GB at 10,000,000.resident memory of a node0 GB5 GB10 GB15 GB20 GB101001k10k100k1M10Mpartitions in the queueLeader, 3M msg/s of 256 BFollower, 3M msg/s of 256 BLeader, 1M msg/s of 1 KB
A partition costs QueenMQ about 1.2 KB on the leader and 0.9 KB on a follower: memory hardly moves up to 100,000 partitions and reaches 17 GB of the leader's 88 at ten million. Source: benchmark-queen/2026-10-15-bytes-per-msg/runs/gcp/grid-fix-m3.jsonl and grid-fix-1m1k.jsonl (mem_gb)

Why the line stays flat

In QueenMQ a partition is a few rows in the store, which every node keeps in memory: the partition itself, its offsets, and a cursor for each consumer group that reads it. Its messages are records in its queue’s log, beside every other partition’s, and one fsync covers everything a group of entries wrote to that log. So a new partition adds memory and nothing else, neither a file nor a replication stream nor an fsync of its own, and the leader’s CPU at ten million partitions is what it was at a thousand.

Kafka keeps a log of its own for every partition on every replica, with its own files and its own place in replication; Redpanda runs a raft group per partition, each served by one core; Pulsar keeps a managed ledger per partition in BookKeeper, though a bookie writes one journal for all of them. That machinery is cheap with a few hundred partitions and grows with every one you add, which is what Kafka’s cores and Redpanda’s latency show from 10,000 on.

The line was flat before 2.4 too, and higher. On these machines 2.3.0 read a p99 of 42 to 76 ms at 3,000,000 msg/s from 10 to ten million partitions; 2.4 writes 147 bytes a message where 2.3.0 wrote 248, its log’s fsync takes about 0.5 ms instead of 3, and the p99 fell to 15 to 25 ms. 2.3.0 had removed the costs that showed at high counts: the leader’s scan for idle consumer state, which held 625,000 partitions under one lock; a table of partition names that hashed every name again each time it doubled; and the merge of the queue log’s directory, synced in one go.

What this page does not show

  • Traffic on every partition at once. Each push carried 100 messages for one partition, so 30,000 pushes a second reached at most 1,800,000 partitions in a minute: all ten million were created, written and drained in the prefill, and the timed load touched a part of them. Traffic on all of them within a minute means pushes of a message or two per partition, which costs more per message and was not run on 2.4.
  • Other rates. Every count ran at 3,000,000 msg/s with 256-byte messages and at 1,000,000 with 1 KB; the ceiling of one queue was measured on 1,000 partitions only (5,000,000 msg/s).
  • Deduplication and retention on, many queues instead of one, and keys hashed onto a fixed set of partitions: measured on 2.3.0 and 2.0.0 betas, not on 2.4; those tables are in the git history of this page.
  • Long runs. Each point ran 70 or 90 seconds; the long runs are on soak and failover, on an older build.
  • Kafka, Pulsar and Redpanda past 100,000 partitions, Redpanda at 100,000, and Pulsar at 100,000 with enough load generators to rule its client out.

Runs: QueenMQ 2.4 in benchmark-queen/2026-10-15-bytes-per-msg/runs/gcp/ (fm3-* at 3,000,000 msg/s, f1k-* with 1 KB messages, grid-fix-m3.jsonl and grid-fix-1m1k.jsonl for every node’s cores, memory and disk) and the a24-* folders of benchmark-queen/2026-10-12-three-node-new-forms/runs/ (1,000 and 50,000 partitions); Kafka, Pulsar and Redpanda in benchmark-queen/2026-09-30-kafka-pulsar/runs/, listed on the comparison.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close