Errors, backpressure and retries

Three things about error handling in Queen are counter-intuitive enough to be worth stating before the tables: a rejected acknowledgement arrives as HTTP 200, a rolled-back transaction arrives as HTTP 200, and an empty pop arrives as HTTP 204 with no body at all. If your client checks status codes only, it will misread all three.

The envelope

Every error body the broker produces is one JSON object with an error string:

{"error":"Authentication required"}

The proxy adds a machine-readable code alongside it, and that field is the one to branch on. The human text may change, the codes are a contract:

{"error":"request rate limit exceeded","code":"rate_limited"}

Two conventions surround the envelope.

A 204 carries no body whatsoever. Not an empty object, not a zero-length body with a declared length. Nothing. This is deliberate: announcing a content length on a body the HTTP layer then elides made strict HTTP/1.1 clients (Node’s undici) treat the connection as poisoned and drop it, which under empty-poll load turned into connection-reset storms. Every client already early-returns on 204, so the body was pure cost.

An unknown route is a JSON 404, not the dashboard. The broker’s fallback serves the embedded single-page dashboard, but for any path under /api/ and for any non-GET method it answers:

{"error":"Not Found","code":"no_such_route"}

Before that guard existed, a call to a nonexistent endpoint came back HTTP 200 with an HTML document, which every JSON client read as success. A phantom endpoint now fails loudly.

Broker status codes

401 and 403 only occur when JWT validation is switched on. Access levels are a role set, not a ladder: a write-only credential passes POST /api/v1/push and is rejected with 403 on every read and every consume route, including /api/v1/pop. The per-route level is listed in the generated route table in the reference.

Where 200 does not mean success

The SDKs handle all three: the JavaScript, Python and Go clients raise on success: false for transactions and cycles, and normalise ack results per item, so a truncated or misaligned ack response fails loudly rather than being misattributed to the wrong message.

Behind the proxy

A proxy deployment adds its own error surface in front of the broker. Every response carries a code; these are the ones a client should switch on.

Note the asymmetry in how quotas are reported. A rate over-limit is a 429 and you should wait. A quota over-limit is a 403 and waiting will not help: the number is not going to refill this month, and a queue-count ceiling will not raise itself. Retrying a 403 is a busy loop.

The 429 and Retry-After contract

Every 429 from the proxy carries a Retry-After header in seconds. Its value comes from the limiter that denied the request:

• Token-bucket denials (request rate, message rate) compute ceil(deficit / refill_rate), clamped to between 1 and 60 seconds. A denial never debits the bucket, so a shadow-mode deny does not distort later decisions.
• Parked-consumer denials (too many concurrent long-poll pops holding slots) use a fixed 5 seconds.

Retry-After is advice about when capacity returns, not a promise. Honour it, and add jitter.

413 in particular

The proxy enforces three separate size limits, and the message tells you which one you hit:

The per-item cap is checked per item precisely so a batch of many small messages is not rejected by the single-item ceiling. The right response to a 413 is to reshape the request, not to retry it: send smaller batches, or a payload that carries a reference instead of the blob. Note that the broker’s own body limit (QUEEN_MAX_BODY_BYTES, 64 MiB) is separate and larger than the proxy’s default request-body cap (16 MiB), so behind a proxy the proxy’s cap is the one you meet first.

What the SDKs retry for you

The JavaScript, Python and Go clients share the same two-layer policy, configured independently.

Layer one: 5xx and network failures. retryAttempts (default 3) attempts with exponential backoff starting at retryDelayMillis (default 1000 ms). With several backend URLs configured, a 5xx or network failure also marks that backend unhealthy and fails over to another. Any 4xx short-circuits this loop: client errors are not retried.

Layer two: 429 only. A separate retry429 policy retries in place, against the same backend, because a rate limit is not a backend-health signal and must not trigger failover.

The delay is Retry-After × 1000 when the header is present, otherwise min(capMs, baseMs × 2^attempt); either way it is jittered by ±20% to avoid a synchronised thundering herd.

The unbounded default for long-poll pops is deliberate. A consumer’s poll loop is already unbounded, so an individual 429 should be waited out rather than surfaced as a failure after a handful of tries. Ordinary requests (push, admin calls, non-waiting pops) get the bounded budget so a producer does not hang forever.

The streams runtime uses its own smaller HTTP client for /streams/v1/*: up to 3 retries with exponential backoff from 100 ms to 5 s, retrying 5xx, network failures, 408 and 429, and treating every other 4xx as terminal. It does not read Retry-After.

Durability when the database is down

A push that cannot reach PostgreSQL is not simply an error. The broker appends it to an on-disk spool and a background loop replays it later; the same path handles maintenance mode, where every push is diverted and reported with per-item status: "buffered" at HTTP 201. A spool write that itself fails downgrades the response to 500 with the failing items marked failed.

A checklist for a robust client

• Read the per-item success on ack responses; do not infer it from the status code.
• Read success on transaction and stream-cycle responses for the same reason.
• Treat 403 and 413 as terminal. Log the code and stop; do not retry.
• Treat 429 as backpressure. Honour Retry-After, jitter it, and let long-poll pops wait indefinitely.
• Treat 502, 504 and network failures as retriable, with backoff and failover.
• Make handlers idempotent on transactionId. Every rejected ack, every lease expiry and every replay is a redelivery.
• Expect 204 with no body from a pop, and do not attempt to parse it.

Next