Skip to content

Benchmark: Kafka clients through QueenMQ's facade

Kafka clients (franz-go) producing and consuming through the facade inside QueenMQ, beside QueenMQ's own clients and Kafka 4.3.1 on the same machines: rate, partition count, CPU, what made the facade fast, and what is still open.

Updated View as Markdown

QueenMQ speaks the Kafka wire protocol from inside the broker process (QUEEN_KAFKA_EMBEDDED=true), so an existing producer or consumer can point at QueenMQ and keep its code. We measured what that costs with the same load generator, built on franz-go, that drove Kafka itself. After a night of work on the facade, Kafka clients see the latency QueenMQ’s own clients see, and at a million messages a second a little better. At 1,000,000 msg/s on 200 partitions the facade’s e2e p99 was 91 ms on the final build, against 146 ms for QueenMQ’s own clients (2.0.0-beta.2) and 18 ms for Kafka 4.3.1 itself; the facade ran the code of 2.0.0-beta.5 on 2026-10-02. On 2.3.0-beta.1 the same point reads 48 ms, and the facade no longer stops consuming at 2,000,000 (2.3.0).

Every number on this page was measured on the first facade, the one 2.0 to 2.5 shipped. 2.6 rewrote it as a translation layer that takes produce on the raft leader only, where the first facade’s followers relayed theirs to the leader and spread the Kafka protocol work over the nodes. Measured against 2.5.0 on 2026-10-20 (three 24-vCPU nodes with local NVMe, franz-go, 1 KB records with lz4, one topic of 100 partitions), 2.6 matched it at 500,000 msg/s, with a produce p99 of 14 ms, added about 3 ms of produce latency at 700,000, and at 1,000,000 pushed 30% less. Neither build carried 850,000 with that client.

The shape

The rig and load of the other pages: one topic of 200 partitions unless stated, 256-byte messages produced 100 at a time to one partition, 60 seconds per point, figures over the last 30. The clients were nine franz-go processes on three load machines, with acks=all, idempotent producers, lz4, a 5 ms linger and classic consumer groups. The QueenMQ cluster ran the facade on every node, with the broker settings of the native runs. The builds are those of the facade work that was merged as b51e4419 (2.0.0-beta.5): qk3 on 2026-10-01, and the final build on 2026-10-02 for the row marked so. Runs: benchmark-queen/2026-09-30-kafka-pulsar/runs/queen-kafka/ (README-c3.txt there maps tags to builds).

2.3.0: five points

Five points of the tables below ran again on 2026-10-13, on 2.3.0-beta.1 and on the machines of the 2.3.0 pre-release runs, to see that the release had broken nothing: the same load generator and client settings, a fresh cluster per point. End-to-end p50 / p99 in milliseconds, the worst load process. The machines are others of the same size, so the columns show a direction and not an exact gain. Runs: the b1-* folders of the archive above.

Point In / out 2.3.0-beta.1 2.0.0-beta.5, 2026-10-02
300,000 msg/s, 200 partitions 300k / 300k 9 / 11 10 / 19
1,000,000, 200 partitions 1,000k / 1,000k 12 / 48 35 / 91
1,000,000, 10,000 partitions 1,000k / 1,000k 13 / 26 45 / 142
1,500,000, 200 partitions 1,499k / 1,498k 98 / 175 185 / 3,785
2,000,000, 200 partitions 2,007k / 1,992k 334 / 20,578 1.91M in, 105k out

No point had an error or shed anything. At 2,000,000 msg/s the consumers now keep the rate. Seven of the nine load processes end at 220 / 350 ms; two hold consumers that fall steadily behind, to 18 and 25 seconds in the last window, which is where the p99 of that row comes from.

Kafka itself on these machines read 8 / 11 ms at 1,000,000 msg/s on 100 partitions, and QueenMQ’s own clients 10 / 19 ms.

Rate

The tables from here to “Still open” are the runs of 2026-10-01 and 02.

End-to-end latency, p50 / p99 in milliseconds, one topic of 200 partitions. “Behind” means consumption under 97% of the offered rate, or a p99 over 5 seconds.

Offered msg/s Kafka clients through QueenMQ QueenMQ’s own clients Kafka 4.3.1
300,000 10 / 19 11 / 22 8 / 10
600,000 12 / 38 19 / 98 8 / 13
1,000,000 44 / 108; final build 35 / 91 61 / 146 8 / 18
1,500,000 185 / 3,785, all in and out behind: 1.34M out 8 / 35
2,000,000 behind: 1.91M in, 105k out behind: 1.50M in, 1.23M out 9 / 59

QueenMQ’s own clients are the 2.0.0-beta.2 runs of September (in the git history of these pages), driven from four load machines. On the facade build with the same three load machines as the Kafka clients, they measured 89 / 212 ms at a million (recorded in b51e4419’s message): our HTTP load generator needs more CPU per message than franz-go, and on three machines it adds latency of its own.

Partition count

At 1,000,000 msg/s, p50 / p99 in milliseconds.

Partitions Kafka clients through QueenMQ QueenMQ’s own clients Kafka 4.3.1
200 44 / 108 61 / 146 8 / 18
10,000 45 / 142 51 / 144 15 / 46
50,000 78 / 200 49 / 103 692 / 1,466
100,000 134 / 251 50 / 163 behind: 2.7 s / 10.9 s
p99 end-to-end latency at 1,000,000 msg/s against the partition count, log scales. Kafka clients through QueenMQ's Kafka facade: 108 ms at 200 partitions, 142 at 10,000, 200 at 50,000 and 251 at 100,000. QueenMQ's own clients: between 103 and 163 ms. Kafka 4.3.1 itself: 18 ms at 200 partitions, 46 at 10,000, 1.5 s at 50,000, and behind at 100,000 with a p99 of 10.9 s.p99 end-to-end latency10 ms100 ms1 s10 s1k10k100kpartitionsKafka clients on QueenMQQueenMQ clientsKafka 4.3.1
The same Kafka clients, pointed at QueenMQ instead of Kafka. Under 10,000 partitions Kafka is faster; past 50,000, QueenMQ's partition-per-entity store keeps the Kafka clients under a quarter of a second while Kafka falls behind. Source: benchmark-queen/2026-09-30-stage03-3node, benchmark-queen/2026-09-30-kafka-pulsar

With many topics of 10,000 partitions in total, Kafka clients through QueenMQ measured 44 / 140 ms on 10 topics of 1,000, 58 / 165 ms on 100 of 100, and 41 / 100 ms on 1,000 of 10.

CPU

Cores used by QueenMQ’s processes on the three brokers at 1M msg/s:

Partitions Kafka clients through QueenMQ QueenMQ’s own clients
200 22.1 17.2
10,000 25.2 18.4
50,000 36.9 18.0
100,000 49.7 18.3
Cores used by QueenMQ at 1,000,000 msg/s, summed over three brokers, with Kafka clients through the Kafka facade and with QueenMQ's own clients. 200 partitions: 22.1 and 17.2. 10,000: 25.2 and 18.4. 50,000: 36.9 and 18.0. 100,000: 49.7 and 18.3.Kafka clients on QueenMQQueenMQ clients02040cores, three brokers200 partitions10,00050,000100,00022.117.225.218.436.91849.718.3
The gap grows on the fetch path. The facade answers Fetch below fetch sessions, so every fetch names all of its consumer's partitions; a QueenMQ pop asks for work and is handed partitions that have some. Source: benchmark-queen/2026-09-30-stage03-3node

The gap that grows with partitions is on the fetch path. The facade answers Fetch up to version 6, below fetch sessions (KIP-227), so every fetch names all of its consumer’s partitions, and at 100,000 partitions the nine client processes used 22 cores against QueenMQ and 6 against Kafka. We left sessions out on purpose: a session is per-connection state, and the facade keeps none, so it can restart the way a Kafka broker does.

What made it fast

On 2026-09-30 the same load at 1M msg/s on 200 partitions got through the facade at 935,000 in and 924,000 out, with an e2e of 2.0 s at p50 and 3.8 s at p99 (run qk-1x200-1000k). Four changes, all in b51e4419, took that to 44 / 108 ms:

  • Produce and Fetch call the broker’s push and fetch as typed calls: the same commands, admission and reads as POST /api/v1/push and POST /api/v1/fetch, with no JSON in between. A test pins that both paths give byte-equal answers, so the two cannot drift apart unnoticed.
  • A Kafka record travels as bytes in both directions.
  • A topic created through Kafka got dedupWindowSeconds: 0. A Kafka record carried no QueenMQ transactionId, so the window could only ever check a brand-new id; the first facade’s own duplicate suppression, the idempotent producer’s sequence window, did not need it.
  • Produce requests that a connection already had buffered were written as one push and answered in order. QUEEN_KAFKA_COALESCE (default 5, Kafka’s max.in.flight.requests.per.connection) capped how many, and nothing waited for a request that had not arrived.

The last two are gone in 2.6: an idempotent record’s producer id, epoch and sequence are its QueenMQ transaction id, so a topic created through Kafka gets a dedup window of 900 seconds, and each produce request is its own ordered push (how it works).

On the final build, Kafka Streams (WordCountDemo) and Kafka Connect in distributed mode, with a file source into a topic and a file sink out of it, passed four runs out of four on fresh three-node clusters.

Still open

Overload was the first open problem. On 2.0.0-beta.5, at 2M msg/s offered, the facade accepted up to 1.9M a second and its consumers starved: 105,000 consumed in the archived run, and anywhere from 42,000 to 1,550,000 in repeats. 2.3.0 reads a fetch only to what its response can carry (a consumer behind on a thousand partitions had a gigabyte read for the 32 MiB a response holds, on every fetch), and at the same rate consumption now keeps up. What is left is the two late load processes of the 2.3.0 run: some consumers fall behind for the whole run while the rest stay near 300 ms, and we do not know why. The fetch change itself has no before and after measurement on one set of machines.

Long runs are the second, and have not been repeated on 2.3.0. A 10-minute run at 1M msg/s (qk3, 2026-10-01) moved 595 million messages in and out with no errors, and its last 30 seconds held 51 / 111 ms. Three stretches of 20 to 100 seconds, around minutes 3, 6 and 8, had produce p99 of 2 to 8 seconds and put the run’s e2e p99 at 10 to 12 seconds. The first began when free memory on the nodes ran out and the followers fell into direct reclaim, while each node’s anonymous memory grew by about 13 GB over the ten minutes.

The facade is also outside the Jepsen tests, which drive the native HTTP API; it has a compatibility suite of its own.

How to point a Kafka client at QueenMQ, and what the facade supports, is in the Kafka guide.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close