Contributing: build, test and change the broker

Queen is one repository with one Rust binary at its center: the broker, which links the proxy and the Kafka facade as libraries and embeds the dashboard. It keeps its state in a data directory, so building it, testing it and running it needs no database and no other service. A clean clone and a Rust toolchain are enough to build a node and run it, and that is a property we work to keep.

Pull requests target master. Run the suite of the component you touched, and read the rules below before you change the engine: each one is there because breaking it is silent until it is expensive.

Every commit carries a Signed-off-by: line from its author, which git commit -s adds. It certifies the Developer Certificate of Origin: the author wrote the change, or has the right to submit it under the Apache 2.0 license. .github/workflows/dco.yml refuses a pull request with a commit that lacks the line, because the project can license only code its contributors had the right to give. How the project is run and who runs it are in GOVERNANCE.md and MAINTAINERS.md.

The repository

Build and run

The committed server/webapp/dist and proxy/console/dist let a clean clone compile without Node. When you change app/, rebuild the dashboard and then the broker, because the assets are embedded at compile time:

npm --prefix app ci && npm --prefix app run build
cargo build --release --manifest-path server/Cargo.toml

When changing shared/timeRanges.mjs, rebuild both frontends before the broker: run npm --prefix proxy/console ci && npm --prefix proxy/console run build alongside the dashboard build above. The common hourly presets feed every picker; extended periods, system metrics, lag thresholds and console usage add their choices in the same module. The Dockerfile copies shared/ into both frontend build stages.

Run one node on a throwaway data directory and check it:

QUEEN_RAFT_DIR=./queen-data ./server/target/release/queen
curl -s localhost:6632/health

/health answers 503 with "status":"settling" until the node knows a leader that hears from a majority and has caught up, then 200. Delete the directory to start empty. The version comes from server/server.json (server/build.rs embeds it), the minimum Rust version is 1.88 (rust-version in server/Cargo.toml), and ./build.sh builds the product image from the root Dockerfile.

Test

test/run.sh builds the broker image as queen:test, gives every suite and topology its own Compose project with fresh volumes, and prints a pass/fail matrix. The tenancy parity gate fails the run when a suite’s single and tenanted results differ, because the tenancy header must change nothing for a client that does not send it. CI runs the same matrix (.github/workflows/tests.yml).

The Jepsen suite is in test/jepsen/ (Clojure and Leiningen, five nodes over SSH): run-matrix.sh runs one test per line of a matrix file from the control node. The failure rehearsals behind the recovery runbook are in test/recovery/. Neither runs in CI.

Rules for engine changes

Apply is pure. rsm/apply.rs, rsm/state/ and rsm/store/ may not read a clock, the environment or a random number: server/clippy.toml lists those calls, and the three modules deny them. Time and ids come from the planner, inside the entry. This is the rule every replica’s correctness rests on, so after you change the list, run test/raft/lint/deny-bites.sh, which plants one call of each kind in rsm/apply.rs and requires clippy to refuse every one.

Bytes on disk are versioned. Effects and outcomes share one catalogue version. A new field is a new version decoded beside the old one, never an edit of an existing encoder, and the golden fixtures in server/src/rsm/tests/golden/ fail when an encoding changes. A cluster raises its version only once every member can read it, which is what lets a rolling upgrade run nodes of two releases side by side.

Nothing blocks the async runtime. Every I/O and RPC has a deadline, store and file calls run on blocking threads, and no std::sync::Mutex is held across an .await. A blocking call on a runtime thread stalls every request scheduled behind it.

Rules for the docs

sourceOfTruth is a debt list. Each page names the files that govern it, and check-sourceoftruth.mjs fails a page whose last commit is older than the last commit of any of them. Update the page in the same change as the code.

The site follows the book. The little book of QueenMQ (book/, LaTeX, built by book/build.sh) and these pages tell the same story in the same order. Change one, change the other. book/build.sh --publish puts a new edition on the site: the PDF, its cover and edition.json in webdoc/public/book/, which the book page reads.

Code on a page ran. A page shows code with <SdkCode id="push" />: the guide:start(push) region of each language’s programs under examples/tutorials/ (00-five-minutes, book-client-guide, book-use-cases), cut by webdoc/scripts/lib/guide-regions.mjs, the same cutter the book, the homepage and the README check use. examples/tutorials/run.sh runs every program against a broker, and each program checks its own results. A region JavaScript lacks fails the build; a region another language cannot express is listed, with its reason, in webdoc/src/lib/book-code.ts. Keep region lines to 92 characters, so they fit the book’s page.

Drawings have one source. Each figure is a module in figures/src/<chapter>/<id>.mjs; node figures/build.mjs writes a light and a dark SVG, the book prints them as PDFs, and the site shows them with <Fig id="..." />. Every figure is 1000 units wide, so they all print at the same type size.

Older snippets. The reference pages still include regions of the client test suites (docs:start(<id>)), turned into partials by webdoc/scripts/gen-snippets.mjs.

Generated partials are not edited by hand. The route tables, the environment variables and the metric families are read out of the Rust by webdoc/scripts/gen-*.mjs; run pnpm --dir webdoc gen. Where a generator mirrors Rust logic in JavaScript (the access levels, the proxy’s route classes), it guards the mirror with a fingerprint of the Rust function and refuses to run when the function changes, until someone re-reads it.

One gate. pnpm --dir webdoc verify runs the generator drift check, the build, the markdown and brief checks, the linter, the prose check, the sourceOfTruth check, the search-surface check and the README check.

Rules for benchmarks

A number is published only with an archived artifact under benchmark-queen/: the harness, the raw output and the conditions (hardware, node count, shape, duration). The site’s benchmark figures are rendered from those artifacts by webdoc/scripts/charts.py, and the drift check fails when a figure no longer matches its data. A number without an artifact is a claim nobody can check, so it stays out.