Skip to content

Jepsen correctness tests and results

How QueenMQ's correctness is tested with Jepsen: the workloads and what each checker rejects, the faults, every campaign with its build and result, the bugs the tests found and the commits that fixed them, and the limits of testing.

Updated View as Markdown

A million messages a second is worth nothing if one of them can go missing. We check that with Jepsen: five nodes, clients driving the native API as hard as they can, a nemesis that kills, pauses, partitions, skews clocks and cuts power, and checkers that read the whole history afterwards looking for a message lost, delivered twice, acked twice or seen out of order. The latest campaign, on 2026-11-16 and 17, ran the whole suite on the code of 2.9.0: 568 tests, the standby failovers, the Kafka facade and the routing of 2.9 among them, and all 568 came back valid. 561 were valid at their first run. Four ephemeral tests found the bug at the end of the list below and were valid on its fix, and so was a seek test once its own mistake was fixed. The last two are valid with three wrong answers counted, not judged, that only clocks minutes apart produce (Limits). That is evidence, not proof. Jepsen finds bugs and cannot show there are none, and further down is every bug it found in QueenMQ 2.

How a test runs

Each test starts a fresh five-node cluster, which needs three nodes for a quorum. It runs one workload for 300 seconds at 100 to 200 operations a second while the nemesis injects one class of fault (a few tests combine two), then heals everything, lets the clients finish (drain the queues, read every partition) and hands the full history to the workload’s checkers. A test is valid only when every checker is. The failover workload is the exception in its shape: it starts two clusters of three nodes, one following the other, and the faults run on both. The harness is in test/jepsen/ (Clojure, Jepsen 0.3.14), and run-campaign.sh runs a list of tests and writes one verdict line per test. The checkers judge the history, and three nodes of five can carry a history while the other two are down, so after each test the script also reads every node’s log: a node that refused to start or panicked turns the line into REFUSED, whatever the checkers said. The tests that damage a node’s files on purpose are left out of that, because there a refusal is the right answer. The panic checker inside each test, which looks for a panic, a poisoned lock or NA-QLOG-I1 in every node’s log, leaves out the same refusals since 2.5.0: an NA-QLOG-I1 line the nemesis recorded as its victim’s reason to refuse a boot.

One Jepsen test. Clients run one workload against a fresh five-node QueenMQ cluster for 300 seconds at 100 to 200 operations a second while the nemesis injects one class of fault: kills, pauses, partitions, clock jumps or power loss. Every call and its answer is recorded in a history. After the faults heal and the clients drain the queues, the workload's checkers read the whole history, and the test is valid only when every checker is.clientsone workload, 300 sfive QueenMQ nodesfresh for each testnemesisone class of faulthistoryevery call, every answercheckersafter heal and drainvalidonly if every checker iscallsfaultsrecordedread whole
Every test starts from a fresh cluster and ends with the whole history checked, not a sample of it. Source: test/jepsen/
cd test/jepsen
RUNS=/root/runs BIN=/root/bin/queen ./run-campaign.sh matrix-p11.txt

One test on its own, against nodes n1 to n5 reachable over SSH:

lein run test --nodes n1,n2,n3,n4,n5 --username root --bin /root/bin/queen \
  --concurrency 2n --workload queue --nemesis kill --time-limit 300 --rate 200

What each workload checks

Workload What the clients do What the checker rejects
W1 log push, and read partitions by offset; with --txn, pushes to several partitions in one transaction; with --batch-sends N, pushes of up to N messages of one partition an acknowledged record lost or duplicated, two orders of one partition, a gap in the offsets, a transaction partly visible
W2 queue push, leased pop, one batch ack carrying the lease an acknowledged message lost, a message acked twice, a redelivery after a successful ack, two unexpired leases on one message, order skipped within a partition
W3 register, W3b counter, W3c claim KV get, put and compare-and-set; increments; putIfAbsent a history that is not linearizable; an increment lost or doubled; two winners of one claim
W4 elle KV calls that write and read several keys any history that is not strict serializable, as Elle finds it: G0, G1, G-single, G2 and their real-time forms
W5 pipeline pop one message, then one /api/v1/transaction that acks it with its lease, pushes its output and increments a KV counter an input lost, an input committed twice, output from a transaction that rolled back
W6 dedup pushes that reuse transactionIds, an unanswered one re-sent once to another node; with --batch-sends N, pushes of 2 to N ids of one partition two records for one id, an answer naming an offset that does not hold its record
W7 dlq acks that fail messages under a retry budget, or send them to the DLQ at once a message neither completed nor dead-lettered, a dead letter that was never failed, a dead letter without its payload
W8 retention W2 with completed messages deleted 2 s after their push a message deleted before its ack
W9 streams stream cycles: pop a batch, read the partition’s state, one cycle that acks it, writes output and updates the state a batch counted twice or never, state that disagrees with the output
W10 timers schedule, reschedule and cancel timers, and consume what fires a timer that fires twice or early, a cancelled timer that fires, fires and cancels that do not add up to the timers created
W11 locks, W11b semaphore take a lock, or one permit of a semaphore, for 2 to 4 seconds; commit transactions that carry its guard and push to the partition it protects; renew it, release it and read its holders. No client watches a clock, so each goes on using a token whose lease ran out a guarded transaction that commits after its permit moved on, a token with two owners, a record of a transaction that rolled back, a grant whose token is not above the one before it, a permit handed to another owner before its lease could have ended
W12 failover two clusters of three nodes, one following the other: push to the source, one record or a transaction over several partitions, and keep sending writes to the standby, which must refuse them; then the source stops (a crash, a crash while the standby is 10 s behind, or a planned switch that waits for the standby), the standby is promoted, and the clients write to it a write a standby took before its promotion; a promoted log that is not the source’s, record for record, up to where it ends; a transaction there in part; a planned switch that lost a write; after a crash, an acknowledged write missing while a later one is there; a write of the promoted cluster lost; a write refused as a standby’s and found in the log
W13 feed push to 48 partitions: one message, several of one partition, or a transaction over several partitions. Ten readers read the tenant feed from the start on every node, five whole and five one shard each, with small pages and short long polls, and move to the next node when a read fails. At the end one transaction covers every shard, and each reader reads until it has it. With --feed-relay, every read asks to be relayed, as the proxy’s reads do, so a leader passes it to a follower an acknowledged message that a reader selecting it did not read exactly once, a failed push that was read, two orders, a hole, part of a transaction or of a position in one answer, a message at or after an answer’s next stamped at or before its resolvedTs, a resolvedTs that goes back between two answers of one node with no restart in between, stamps that do not rise with position, a shard reader that reads another shard

Since 2.8 the harness has nineteen more, each with its own checker in test/jepsen/src/jepsen/queen/workload/: W14 the Kafka facade, W15 filtered consumer groups, W16 reading queues, W17 a link in a transaction, W18 transactions that roll back on purpose, W19 the SDKs’ feed streams as processes under test, W20 ephemeral queues, W21 tenants and raft groups side by side, W22 scoped keys and client tokens, W23 ack retries, dead-letter replays and message deletes, W24 delayed processing, window buffers and long-poll waits, W25 KV keys and claims with a ttl, W26 a group sought to the end and back in time, W27 a queue deleted and made again under load, W28 and W29 the PostgreSQL source and sink, W30 the S3 sink, W31 copies of one push sent together, and W32 Elle over /api/v1/transaction.

The faults

  • kill -9 of nodes, restarted after a while; with lazyfs, a kill also loses every write the node had not fsynced, as a power cut does, and a slow-disk variant makes two nodes fsync 300 ms late
  • SIGSTOP pauses
  • network partitions: one node, a minority, a majority, a ring of overlapping majorities, and a bridge where one node sees both halves
  • a deaf leader, whose followers receive its appends while their answers never reach it
  • clock jumps and strobes on any node, the leader included
  • graceful restarts (SIGTERM) of one node, or of every node in turn
  • membership changes through the admin API: a voter removed, wiped and added back, or two at once, also while snapshots are being installed
  • a file bit-flipped or truncated on one node, which must refuse or repair the damage and never serve it
  • packets between nodes delayed, lost, duplicated, reordered or corrupted (tc netem), alone and together, or squeezed to 1 Mbit/s
  • a full disk on one node, which must stop without harm and come back once there is room
  • a rolling upgrade under load: the nodes restarted one at a time from the previous release onto the new one
  • kills of the SDKs’ stream runners, which W19 runs as processes under test

The campaigns

Campaign Date Build Tests Result
P10, two full passes 2026-09-26 ec9d88c4, the commit that removed PostgreSQL 82 per pass, W1 to W6 164 of 164 valid
P11 2026-09-28 2.0.0-alpha.6 125: P10’s tests, W7 to W10, snapshot installs during membership changes, slow-disk power loss 123 valid; the other 2 were false positives of the new DLQ checker, fixed in 71031581
P16 to P24 2026-09-30 and 10-01 the builds that became 2.0.0-beta.1 78 74 valid; the other 4 were the dead-letter payload bug below
P26 2026-10-01 2.0.0-beta.3, bff9f76d 65: queue, pipeline, KV register, counter and claim, log, Elle and timers, under kills, pauses, partitions, clock faults, restarts, membership changes, bridges, a deaf leader and power loss 65 valid
Kafka-facade build 2026-10-02 the code of 2.0.0-beta.5, before a fix inside the facade 65 64 valid, 1 unknown
2.0.1 2026-10-05 2.0.1, bc1b5bfb6 19: queue, pipeline, dead letters, log, dedup and KV register under kills, pauses, partitions, restarts, a deaf leader and power loss, the queue and the pipeline on 2,000 and 1,000 partitions whose idle consumer state leaves the leader’s memory after a second 19 valid
Locks 2026-10-08 2287397a3: 2.0.3 with locks, the KV check op and cluster version 5 41: W11 and W11b (29, three of them without faults), then KV register, counter and claim, Elle and the pipeline again (12), under kills, pauses, partitions, clock faults, restarts, membership changes, bridges, a deaf leader and power loss, and with kills, pauses, partitions and clock faults in one test 41 valid
2.1.0 2026-10-08 and 09 b69470031, the code of 2.1.0: 2.0.4 with the standby cluster, locks and the four fixes of that night 153: the 125 of P11 (W1 to W10) and 28 lock and semaphore tests (W11, W11b), two of them without faults 152 valid; the other was the harness, which had let the nodes’ clocks drift apart, and it ran again valid
Failover 2026-10-08 and 09 the same build 37: W12 on two clusters of three nodes, with crashes of the source, planned switches, transactions and crashes that find the standby 10 s behind, under kills, power loss, pauses, partitions, clock faults, restarts and all of them at once 36 valid; the other was the harness, and the 10 planned switches ran again valid
Rows window, first pass 2026-10-09 e8efda8cb, the first build of 2.2.0’s rows window 41: the queue, the pipeline, retention, the log, dead letters, dedup, the KV register and Elle with rows that live two seconds, on 256 KiB log files sealed after 10 s, so that consumers read messages that have no row; under kills, power loss, pauses, partitions, clock faults, restarts, membership changes, bridges and a deaf leader 37 valid; the other 4 were the two refusals to start below
2.2.0 2026-10-09 and 10 3b1a6d9bb, the code of 2.2.0 without its last three commits 244: the 153 of the 2.1.0 campaign, 40 of the first pass, 12 on the same small log files with rows that stay, 8 on shared logs, and 31 failovers, 4 of them with rows that live two seconds 244 valid, one at its second run: the harness crashed in the first
2.3.0 2026-10-13 and 14 1c7136d4a, the server code of 2.3.0, as the binary of the 2.3.0-beta.2 image 244: the tests of the 2.2.0 campaign 244 valid: three once their histories were checked again with more memory, one at its second run, after the harness crashed in the first
2.4.0 2026-10-15 and 16 ed73dc1cb, the server code of 2.4.0 without its version line and the encryption fix of c42ea1992 257: the 244 of the 2.3.0 campaign and 13 that push several messages of one partition at once 257 valid, all at their first run
2.5.0 2026-10-17 and 18 3646d217f, the server code of 2.5.0 without the boxing of 19b024495 (a lint fix in the Kafka facade), as the binary of the 2.5.0-beta.1 image 257: the tests of the 2.4.0 campaign 257 valid: 256 at their first run; p11-w1-corrupt-all failed the panic checker alone in both of its runs, on the node’s own refusals to boot on a cut queue log, and both runs are valid under the fixed checker
2.6.0 2026-10-21 and 22 229a4c3ea, the server code of 2.6.0 without its version line, as the binary of the 2.6.0-beta.6 image; the setup check and the first 91 tests of the main list ran d3809f2e5, 2.6.0-beta.1 257: the tests of the 2.5.0 campaign 257 valid: 255 at their first run; p11-w5-slow-lz-kill in four reruns, after its first run’s final drain ran out of time with inputs still waiting, and p11-w3-lead-deaf once knossos’ WGL search checked the history its linear search ran out of memory on
2.7.0 2026-10-30 and 31 f9c9a37f2, the server code of 2.7.0 without its version lines, the disk fields of 107aaaf9b, a lint fix in the Kafka facade and the streams list of 97b0f2f8e; the first 19 tests ran 19f3d492c 226: the tests of the 2.6.0 campaign without its 31 standby failovers, the stream tests closing quiet partitions under a claim 226 valid, all at their first run
Feed 2026-11-07 the tenant feed’s code before its commit, with the fix for the first resolvedTs bug below 43: W13 (19: one without faults; kills, power loss, partitions, pauses, clock faults, a deaf leader, restarts, membership changes, partitions with kills, pauses with kills and bridges; kills, power loss and partitions on 256 KiB log files sealed after 10 s; kills and partitions on shared logs; partitions and a deaf leader with the leader serving feed reads), then the log, the queue, the pipeline and streams (24) under kills, partitions, pauses, clock faults, a deaf leader and restarts 40 valid; the other 3 were the second resolvedTs bug below
Feed, on the fix 2026-11-07 the broker code of a6e85381d, the tenant feed’s commit, with an older copy of three files of the embedded proxy, which no test runs 21: the 18 W13 tests with faults, and 3 whose every read asks to be relayed, under kills, partitions and a deaf leader 21 valid, all at their first run
2.9.0 2026-11-16 and 17 4b641a866, the server code of 2.9.0-beta.3; its five reruns ran 72d5e6360, the code of 2.9.0, which adds the ephemeral fix below 568: every workload of the harness, 36 of them, the standby failovers (58), power loss (60), rolling upgrades (9), several raft groups (16), several tenants (18), the Kafka facade (30), the embedded proxy (10) and the binary wire forms (18) among them 568 valid: 561 at their first run; four ephemeral tests on the fix below and a seek test on a fixed test; two with three wrong answers under clock jumps counted, not judged

The unknown of 2026-10-02 is W9 under bridge partitions. After the last heal, a follower’s pops kept timing out for about 90 seconds although it had caught up, so the final reads did not finish within the test’s time limit and the checker had no end state to judge. It was a liveness problem, and its cause was not found. The same test was valid in the 2.1.0 campaign, on 2026-10-09, whose build has the fix for a leader that stops planning while it leads (5d48aef94): such a leader’s requests time out the same way, and whether it was the cause that day is not known. P8 to P10 are archived with every history in benchmark-queen/2026-09-27-jepsen-archive/.

The 2.1.0 campaign started on the build as it was merged (e2f7236bd) and was stopped after 48 tests, 47 of them valid. The other had found a five-node cluster whose leader led on for two minutes and answered nothing (the row for 5d48aef94 below), and the fix was ready. The campaign then ran from its first test on the build with the four fixes, on two sets of five nodes that shared the list. In 51 of its tests a leader was handed an entry when no quorum had acknowledged it for 2 s or more, 116 times in all: each is a write that raft refused before the fix, and each waited until a quorum answered or that leadership ended. The one test judged invalid, a semaphore test under a deaf leader, ran on nodes whose clocks were 0.55 s apart, and is among our own faults below. The verdicts of both runs are in benchmark-queen/2026-10-08-jepsen-full/.

The failover campaign started on the build as it was merged and found three bugs in its first two hours, listed below; a fourth came from the standby’s own tests the same evening. On the build with the four fixes, the sources acknowledged 446,016 writes over 37 tests. No standby took a write before its promotion, no promoted log differed from its source’s, and the ten planned switches lost nothing. A crash of the source lost what the standby had not read: nothing in 19 of 23 runs, the writes of the last 3 to 20 seconds in the other four, where the standby’s own leader was cut off or dead at that moment, and about 10 seconds of writes in the four runs that cut the two clusters apart for 10 seconds first. Every standby was promoted, in 9 ms at the median, and took its first write 10 ms later. The one test that did not end valid was a planned switch during which the harness killed two nodes of each cluster and then waited two minutes for a standby it had left without a quorum; the ten planned tests ran again on a harness that heals before it waits, all valid. The verdicts and counts are in benchmark-queen/2026-10-08-jepsen-failover/.

The locks campaign counted what the fence did. In its 29 lock and semaphore tests 373,312 guarded transactions committed, permits changed hands 18,456 times, and the guard refused 3,245 transactions that carried a token its permit no longer had. No token went back, and no record came from a transaction that rolled back. One thing the checker reports and does not judge: under clock strobes, one semaphore test read the same owner in two slots 51 times, 12 pairs of permits in all. In the one traced, the owner’s permit looked expired while a clock was ahead, its next acquire was given another slot, and the first permit was live again a moment later. No semaphore gave out more permits than its limit. Every test’s verdict and counts are in benchmark-queen/2026-10-08-jepsen-locks/.

The 2.2.0 campaign tests what that release changes: a push’s row leaves the store when its queue’s dedup window has passed, and a consumer behind that reads from the queue log. Its first pass ran the consumption workloads with rows that live two seconds, on log files of 256 KiB that are sealed after ten seconds, where the defaults are 64 MiB and ten minutes, so that every test seals, reclaims and removes log files while its faults run. Four of its 41 tests were not valid, each with nodes that had refused to start: the two rows below. Both bugs are in 2.1.0 too, where a five-minute test does not reach them.

The campaign then ran for sixteen hours on two sets of nodes, and on five builds, because fixes landed while it ran: each test that had run on an older build ran again on the newer one, 352 runs in all, and all 244 tests ran on the last build, 3b1a6d9bb. One run on the first build was judged valid while two of its five nodes had refused to start: three nodes carried the history. It was the second bug again, in a place the first fix had missed, and it is among our own faults below, because the harness should have said so. Every node’s log of both sets was read at the end, the tests that damage a node’s files on purpose left out, and no other run has a refusal or a panic. Eleven of the tests then ran on the final code, 8f5342d29, and all were valid. The verdicts, the builds and the node logs’ scan are in benchmark-queen/2026-10-09-jepsen-220/.

The 2.3.0 campaign ran the same 244 tests on what that release changes: the consumption engine (its scan of idle partitions in turns, and a checkpoint taken as soon as an answer waits for one), when a command’s recorded outcome is retired, retention spread over its interval, the planner’s tables of partition names, a durable point every 250 ms written out in steps, and the dedup filter of busy partitions. It ran on one set of six nodes, n1 to n5 for 213 tests and all six for the 31 failovers, from 2026-10-13 11:58 to 2026-10-14 11:50 UTC. Three tests of the log workload first came back unknown: the checker of their histories ran out of memory on a control of 4 GB, and the other checkers had found nothing. Their stored histories were checked again with 8 GB, and all three are valid, with no anomaly, no duplicate and no torn read. One failover test crashed in the harness before any load ran, for the reason the 2.2.0 campaign met on its sixth node, and it ran again valid. The binary wire forms that 2.3.0 adds were not part of it, as its clients speak JSON. The verdicts are in benchmark-queen/2026-10-13-jepsen-230/.

The 2.4.0 campaign ran the same 244 tests on what that release changes, the bytes a message costs on disk: a push’s outcome written as runs, the checkpoint’s index rows without their hashes and the boot that puts them back, and entry records without the hashes their message records hold, all from cluster version 7. Every test of those 244 pushes one message at a time, and a one-message push keeps the older outcome, so 13 more tests push up to eight messages of one partition at once (--batch-sends): the log workload under kills, partitions, pauses, power loss, a deaf leader, graceful restarts, forced snapshots, clock jumps and rows that live two seconds, and the dedup workload under kills, partitions, restarts and power loss. It ran on one set of six nodes from 2026-10-15 13:20 to 2026-10-16 13:33 UTC, after a rolling upgrade from 2.3.0 on three of them, and every test was valid at its first run; lazyfs was built on every node before the first power loss. The encryption fix that 2.4.0 also carries was not part of it, as no test runs with an encryption key. The verdicts and the upgrade are in benchmark-queen/2026-10-15-jepsen-24/.

The 2.5.0 campaign ran the same 257 tests on what that release changes: openraft 0.10.0-alpha.37 in place of a git revision, a hook in the batcher, the path every push takes, for the Kafka facade’s ordered pushes, and the end of a windowBuffer hold. It ran on one set of six nodes from 2026-10-17 15:05 to 2026-10-18 15:27 UTC, then a rolling upgrade from 2.4.0 to 2.5.0 and back on three of them, at cluster version 7 throughout. 256 tests were valid at their first run. The 257th, p11-w1-corrupt-all, damages one node’s files on purpose and checks that the node refuses them. In both of its runs every checker passed but the panic checker, which failed on NA-QLOG-I1 lines that were that node’s refusals to boot, each within a second of the nemesis cutting one of its queue logs short. Those refusals are the right answer, so the checker now leaves out exactly the lines the nemesis recorded as its victim’s reason to refuse. Both stored runs are valid under it, and taking the nemesis’s records out of their histories makes them fail again. The Kafka facade, which 2.5.0 changes the most, is outside these tests, as their clients use the native API. The verdicts and the upgrade are in benchmark-queen/2026-10-17-jepsen-25/.

The 2.6.0 campaign ran the same 257 tests on what that release changes: a member that comes back recovers from the leader’s log or from a delta snapshot, a boot reads hash sidecars instead of its queue logs, and the raft port is listened on late. It ran on seven GCP VMs from 2026-10-21 07:48 to 2026-10-22 10:00 UTC, then a rolling upgrade from 2.5.0 to 2.6.0 and back on three of them, at cluster version 7 throughout. Tests 1 to 91 of the main list ran 2.6.0-beta.1 (d3809f2e5), and the rest of the campaign 2.6.0-beta.6 (229a4c3ea), the server code of 2.6.0. 255 tests were valid at their first run. p11-w5-slow-lz-kill, the pipeline under kills with two nodes fsyncing late through lazyfs, failed its checker once, on beta.1: its throughput collapsed under the kills (3,798 inputs, where the reruns took 15,500 to 21,000), and the final drain, which the harness caps at 120 seconds, ended with 1,323 acknowledged inputs still waiting, which its checker counts as lost. 1,633 transactions committed after the final reads, and nothing was written twice or without its input. It was valid in four reruns on beta.6, one of them with a ten-minute drain. p11-w3-lead-deaf, the KV register under a leader that stops answering, came back undecided: knossos’ linear search ran out of memory on two of its 117 keys, and stalled again with a 56 GB heap. Its WGL search, which decides the same question, found the stored history linearizable on every key in two seconds. The Kafka facade, which 2.6.0 rewrote, is outside these tests, as their clients use the native API. The verdicts, the upgrade and both checks are in benchmark-queen/2026-10-21-jepsen-26/.

The 2.7.0 campaign ran 226 tests on what that release changes: request bounds, a memory guard, bounded dead letters and state a client grows, KV reads served as their stored text, a transaction’s KV writes at their own rate, and stream claims, under which the stream tests now close a quiet partition’s windows as the runners do. It ran on six GCP VMs from 2026-10-30 11:39 to 2026-10-31 08:54 UTC: every test of the 2.6.0 campaign but its 31 standby failovers, which need two clusters of three nodes. Tests 1 to 19 ran 19f3d492c, the rest f9c9a37f2, which adds KV reads as stored text, a lock per Kafka transactional id and the transaction’s KV rate. All 226 were valid at their first run. A rolling upgrade from the released 2.6.0 to 2.7.0-beta.2 and back then ran on three more VMs under load, at cluster version 7 throughout: a push of 12,000 messages was taken by a 2.6.0 leader and refused by a 2.7.0 node before forwarding, and popped back as 10,000 and 2,000; a stream claim answered 404 from a 2.6.0 leader and 200 after it; KV reads gave the same bytes on both versions; and a 2.6.0 node led a directory 2.7.0 wrote. The verdicts and the upgrade are in benchmark-queen/2026-10-30-jepsen-27/.

The feed campaign tests what 2.8.0 adds: the tenant feed, which reads every push to a tenant’s queues in one order, by position, from any follower, with a resolvedTs below which nothing new will appear. Its workload, W13, checks every answer of ten readers against the pushes. Its checker ran first on crafted histories: a clean one, 18 with a planted bug each, and three for relayed reads. Before the campaign, W13 without faults found the first bug below. The campaign ran on a build with its fix, from 2026-11-07 11:51 to 15:57 UTC on six GCP VMs. Its 24 tests of the other workloads were valid, and so were 16 of its 19 W13 tests. The other three found the second bug: a resolvedTs that went back by one microsecond on one node, after a new leader’s first entry. No test lost a message, read one twice or left a hole. The 18 W13 tests with faults then ran again on the fixed code, until 17:56 UTC, with three more whose every read asks to be relayed, as the proxy’s reads do: a leader passes such a read to a follower instead of answering 421. All 21 were valid. Without that header the leader answered 421 2 to 58 times a test, and with it 0, 0 and 1 times. A relayed answer does not say which follower gave it, so in those three tests the checker lists a step back of resolvedTs and does not judge it. Each W13 test with faults acknowledged 12,867 to 44,336 messages. The campaign ran on run-matrix.sh, which does not read the nodes’ logs after a test the way run-campaign.sh does. Each test’s panic checker read them, and its member check found all five nodes answering as voters at the end. The verdicts are in benchmark-queen/2026-11-07-jepsen-feed/.

The 2.9.0 campaign ran every test the harness has, on four rigs of a control and six nodes each, from 2026-11-16 11:17 to 2026-11-17 01:57 UTC. Each rig began with what changed most since 2.8: reading queues, Elle over /api/v1/transaction, the PostgreSQL source and sink, KV ttl, queue options and scoped keys, 59 tests, all valid. 561 of the 568 were valid at their first run. Four of the other seven were the ephemeral tests under restarts, kills, partitions and pauses, which found the last bug below; on its fix, the code of 2.9.0, all four ran again valid, and so did the seek test whose own mistake is told further down. The last two, the pipeline over 30 minutes and the locks under every fault at once, each had a step answered rolled back that had committed, after a node took over as leader with its clock 202 s and 155 s ahead (Limits). Their checkers now count that answer under clock faults instead of failing the run, and the two stored histories, checked again, are valid. Every verdict, the reruns and the second checks are in benchmark-queen/2026-11-16-jepsen-29/.

What the tests found

Every row is a bug in QueenMQ that a campaign caught, and the commit that fixed it names the test that found it.

Campaign Workload and fault What went wrong Fixed in
P8 W4, pauses a read saw half of one batch’s writes (G-single) 1d2efe84
P8 W4, deaf leader a call that wrote and read was answered seconds later from the node’s current state, and its read saw writes made after its own (G1c) 46a4539f
P9 W3c, pauses an acknowledged write was lost: a paused leader, already replaced, answered its client by log index, and the entry at that index was the new leader’s 0dc42c51
P9 W2, clock jump the leader’s clock leapt about 300 s and ended leases early; a second worker got a message 10 ms after the first dd905cb4
P9 W5 and W2, clock strobe after a leader’s clock excursion, every pop was answered empty for minutes 50f45449
P14 W2, clock strobe the new consumption engine timed leases on the wall clock: 223 overlapping leases d013946d
P15 W2, graceful restart an ack whose answer was lost in a leader change was answered “stale” on the new leader, and its message never came back 7a2a1711
P22 W7, kill -9 1 to 8 of about 7,600 dead letters were filed without their payload 7a2a1711
Failover W12, power loss and partitions a standby’s new leader planned a client’s command that had waited out its election, stamped with its own clock; every node refused the next entry it replayed, and stopped 963f7846b
2.1.0 W2, power loss with two slow disks; W12, partitions a leader that no quorum had acknowledged for 2 s had a write refused by raft, stopped planning and never started again while it led: every request ran into its deadline, and a standby could not be promoted 5d48aef94
Failover W12, partitions a node that had led the standby and was elected again after the promotion answered 503 standby, also to the retry of a write the cluster had already taken 63a4805ed
Rows window W2 and W8, power loss, on log files sealed after 10 s an empty queue log was deleted in place; a removal that stopped half-way left its directory with no file in it, which a start takes for a log that lost its records, and every node refused to start 6f18bd40f
Rows window and 2.2.0 W8 and W2, kills, on log files sealed after 10 s a follower’s log was cut back and its next entry went into an empty file created for a later seq, one the cut had emptied or one sealed right after it; a search by seq skipped that file, and the node refused to start 6f18bd40f, 3b1a6d9bb
Feed W13, no faults a caught-up answer could claim a resolvedTs at or past the stamp of the entry at its next: the node read its applied index before its clock, and an entry that applied in between gave it the clock of the next one a6e85381d
Feed W13, kills and partitions resolvedTs went back by 1 µs on one node: an answer cut short claimed the stamp of its newest entry, and a new leader’s empty entry moves the applied index but not the clock, so the next caught-up answer claimed 1 µs less a6e85381d
2.9.0 W20, restarts, kills, partitions and pauses an ephemeral partition that moved to another node lost messages: the new owner’s ring knew only the groups that had popped there, and freed what the first of them took before the other groups’ places arrived with the old ring. And a ring that landed in a busy one dropped every group’s place, so a group was handed again what it had taken 5543add64

The campaigns on the way to 2.9.0, from 2026-11-13 to 15, found more bugs, all fixed in 2.9.0: the 2.9.0 release notes say of each one that the Jepsen tests found it.

Two of P8’s failures were our own. The claim workload lost ids when a call timed out, and Elle reported version cycles that did not exist, because a set union in its bifurcan dependency modified its inputs. The W4 checker runs with that union replaced (edc699b6), until Elle ships on a fixed bifurcan.

The night of 2026-10-08 had two of our own as well. A planned switch ended unknown because the failover’s steps let the kill nemesis in between them: it killed two nodes of each cluster, and the harness then waited for a standby it had left without a quorum (7bf42d5ac). And a semaphore test was judged invalid because the nodes’ clocks had drifted 0.55 s apart. The harness turns the nodes’ time daemon off for the clock faults, only a clock test set the clocks again, and the last one on those nodes was three hours old; a lease survives a leader change only while the clocks agree within 500 ms. Every test now starts with its nodes’ clocks set (c4525140e).

The 2.2.0 campaign had two more. A run was judged valid while two of its nodes had refused to start: the checkers judge the history, and nothing looked at the nodes. run-campaign.sh reads every node’s log after each test since, and such a run is REFUSED. And a failover test crashed in the harness, the first one that needed lazyfs on six nodes: the sixth had never built it, and the build took longer than a node’s setup may. It ran again valid.

The 2.3.0 campaign had two as well. Its control had 4 GB, where the earlier controls had 31: the log workload’s checker ran out of memory on three histories, which were checked again once the control had 16 GB. And its sixth node met the lazyfs build of the 2.2.0 campaign again, since the harness still builds lazyfs during a test’s setup the first time a node needs it.

The 2.9.0 campaign had one of our own. The seek test sent each of its two seeks once; under kills both met a node that was down and failed, so the run checked nothing about seek. It now sends a seek again until one is answered (fc0b6f978), and it ran again valid.

Limits

This is testing. A valid run shows that these histories, under these faults, broke no rule a checker knows. The clients speak the native HTTP API, the Kafka facade and, in 18 tests of the 2.9.0 campaign, the binary wire forms of 2.3.0 (binary push, compressed groups, framed pops, the binary transaction body); 16 of its tests run several raft groups, 18 several tenants, and 10 go through the embedded proxy. Some paths no test drives: a pop over several partitions at once (autopilot, which the JavaScript client turns on by default), a consumer group that starts at new or at a time (every group in the tests starts at all), KV delete and getPrefix, and the MySQL source. Lease exclusivity across a leader change needs the nodes’ clocks within QUEEN_RAFT_MAX_CLOCK_SKEW_MS (500 ms) of each other. Further apart, the pipeline and lock checkers count one wrong answer and do not judge it (the guarantees list it under clocks): a leader that takes over with its clock minutes ahead forgets the answers kept for retries, so a transaction a node relayed during the change runs again and is refused (its ack was made already, or its push is already there), and the client is told it rolled back although its first run committed. Nothing is lost or committed twice. Losing a majority of the disks for good can lose acknowledged writes, since raft survives the loss of a minority. The full campaign, standby failovers included, last ran on the code of 2.9.0-beta.3 (4b641a866), and its five tests that needed a second run on the code of 2.9.0 (72d5e6360), which adds the ephemeral fix and its version lines and nothing else. No test sets a completed retention above its queue’s dedup window, so none of them checks the rule of 2.2.0 by which a queue’s completed retention no longer extends how long a push keeps its row. A standby’s failover has its own workload since 2026-10-08; since the 2.9.0 campaign it also runs on a seeded standby and across a rolling upgrade, but two clusters in different places were not part of it.

What each API guarantees, written for the developer using it, is on guarantees.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close