When QueenMQ refuses a call it answers a status code and a JSON body with a stable code, and the code is the part your client should branch on: messages get reworded, codes do not. Most of the decisions a client has to make come down to three questions, and the tables below answer them for every code: can I retry this as it is, do I have to fix the request, or do I need to find out whether it already happened.
A rolled-back transaction is not an error. It answers 200 with success: false and a reason, and
those reasons are on transaction.
Error bodies
| Where | Body | Branch on |
|---|---|---|
| Most routes: push, pop, ack, lease, queues, consumer groups, DLQ, status | {"error": "<message>", "code": "<code>"}, plus "leader" on a 503 retry when the leader is known |
code |
| Reads by offset refused for their shape (too many entries, a negative offset, a bad timestamp) | {"error": "<message>"} with 400; a per-entry error (UNKNOWN_TOPIC_OR_PARTITION, OFFSET_OUT_OF_RANGE, BAD_CURSOR) inside a 200 |
the status, then each entry |
POST /api/v1/transaction, rolled back |
{"transactionId", "success": false, "reason", "error", "results": []} |
reason |
/api/v1/kv*, /api/v1/locks and /api/v1/timers* |
{"error": "<code>", "reason": "<finer code>", "detail": "<message>"} |
error, then reason |
/api/v1/ephemeral/* |
{"error": "<message>", "code": "<code>"} |
code |
| A pop refused for its parameters | {"success": false, "error": "<message>", "messages": []} with 400 |
the status |
| Authentication and the tenancy header | {"error": "<message>"} |
the status |
The proxy, and /api/cp |
{"error": "<message>", "code": "<code>"} |
code |
| Raft membership changes | {"ok": false, "code": "<code>", "error": "<message>"} |
code |
The 429 and 503 answers of the engine, KV, lock and timer routes carry Retry-After, in seconds. A 404
for something that does not exist (a queue, a message) may carry only error.
Engine codes
The code of the first shape.
code |
Status | Meaning | Client |
|---|---|---|---|
retry |
503 | This node could not get the call to a leader: a leader change, no quorum lease, or the leader unreachable from the node that forwarded it. Also an ack, or the ack of a transaction, that would dead-letter a message the node that took it has not read yet (a moment after a leader change): nothing moved. | Retry after Retry-After (1 s). A write may have reached the leader, so it must be safe to repeat. |
no_leader |
503 | No leader is known after the hold: an election is running, or a majority is down. | Retry. |
timeout |
503 | The request’s deadline passed before an answer. A write may have committed. | Retry a write that is safe to repeat; otherwise read to check. |
outcome_unknown |
503 | The leader changed while an ack, nack or DLQ move was being made durable, or a forwarded one was lost after it went out. It may or may not have applied. | Send the same ack again: a repeat of an ack that applied is answered as the first one was. |
standby |
503 | The cluster is a standby: it replays another cluster and takes no writes until it is promoted. Every write route answers it, on every node, and nothing was applied. | Retry after Retry-After (1 s) if a switch to this cluster is under way: the call is served once the standby is promoted. Otherwise you are pointed at the wrong cluster. |
feed_on_leader |
421 | POST /api/v1/feed reached the node that leads the tenant’s raft group of a cluster: the feed is read from followers, so its readers add no load to the leader. The body lists followers (nodeId, httpAddr). |
Ask one of the followers; the JavaScript client does. A caller that cannot reach them sends x-queen-feed-relay: 1 and the leader relays the read to a follower; the proxy always does. On a test cluster, QUEEN_FEED_ON_LEADER=true lets the leader serve it. |
overloaded |
429 | The node’s admission budget for writes stayed full for the whole hold. | Retry after Retry-After (1 to 5 s). |
cluster_version |
400 | Headers, a where, a reading queue or link, on a cluster where some node does not read catalogue version 8 yet. |
Upgrade every node; the feature turns on by itself. |
reading_queue |
400 | A push into a reading queue, a keep setting on one, or a link into one. |
Push to its source; set retention on the source. |
where_mismatch |
400 | A filtered group’s pop with another where than the group’s, or a where on a group that already reads the queue without one. |
Send the group’s where (the message names it), change it on purpose, or use another group. |
bad_where |
400 | A where that does not parse; the message says where. |
Fix the condition. |
route_loop, source_not_found, bad_source, too_many_sources, too_many_readers |
400 | A reading queue that would read itself, a source that does not exist or is a filtered group’s, or past the limits. | Fix the configuration. |
has_readers |
409 | A delete of a queue that reading queues read (readers lists them). |
Delete them first, or delete with force=true. |
source_gone, source_mismatch |
200, success: false |
A transaction’s link to a message its source no longer keeps, or whose transaction id is not the one at that offset. |
Nothing to forward; check the message named. |
storage_full |
507 | A node is above its disk high-water mark (QUEEN_RAFT_DISK_HIGH_PCT, 85%). Pushes, KV writes and timer schedules are refused; acks and deletes still work. |
Free disk or add space, then retry. |
name_too_long |
413 | The tenant, queue and partition or group names together exceed the store’s key budget, counting every key the command derives from them. | Shorten the name. |
too_many_items |
400 | A push carries more than QUEEN_PUSH_MAX_ITEMS (10,000) messages. |
Split the push. |
memory_pressure |
503 | The node is near its memory limit: past QUEEN_MEMGUARD_SOFT_PCT (80 %) it refuses page reads and other large reads, past QUEEN_MEMGUARD_HARD_PCT (90 %) also writes that grow the stored state. Pops, acks, renewals and deletes are always served. Retry-After is 1 to 5 s. |
Retry after Retry-After. |
busy |
503 | Every one of the node’s large-read slots (QUEEN_HEAVY_READS_MAX, 32) stayed taken for 2 s. Retry-After is 1 to 5 s. |
Retry after Retry-After. |
bad_body |
400 | The body is not the route’s JSON shape. | Fix the request. |
bad_request |
400 | A field is missing or invalid. | Fix the request. |
| any other code | 400 | The engine refused this request for the reason the message gives. | Fix the request; do not retry it unchanged. |
no_such_route |
404 | No route has that method and path. | Check the path against the routes. |
internal |
500 | A broker fault. | Report it; do not retry in a loop. |
raft_phase1_unsupported |
503 | The node does not serve this call. | Report it. |
A 503 or a timeout on a write does not mean nothing happened. Make writes safe to repeat
before you retry them: a deterministic transactionId on every pushed message, the same lease on a
repeated ack, a required marker in a transaction. See dedup.
After a leader change
What a leader change does to a call in flight depends on where the call is remembered.
Pushes, transactions, KV and timer calls are logged under a request id that the node serving the
call minted. That node follows the leader change itself and resubmits under the same id until the
deadline, and a command that had already committed is answered from its record and does not run
again. You get the normal answer, or 503 timeout if the deadline passes first.
Acks, nacks and DLQ moves are served from the leader’s memory and made durable by its next
checkpoint, a few milliseconds later. If the leader loses its lead while that checkpoint is on its
way to the log, or a forwarded ack is lost after it went out, nobody can say whether it applied, and
the answer is 503 outcome_unknown. Send the same ack, with the same lease, again.
Pops are never in doubt: a claim nobody receives is handed back or ends with its lease, and its
messages are delivered again. A node that cannot reach the leader answers 503 retry or
no_leader, and the request may have reached the leader first.
All of this holds for a cluster. Lease exclusivity across a leader change assumes node clocks
within QUEEN_RAFT_MAX_CLOCK_SKEW_MS (500 ms) of each other. See
guarantees.
KV, lock and timer codes
The error of the third shape. The reason beside it is finer, and stable too. The locks route
is KV underneath, so apart from its own locks_bad_request it answers the KV codes below: the
quota, the grant, the rates and the pause are KV’s.
error |
Status | reason, and when |
Client |
|---|---|---|---|
kv_bad_request |
400 | kv_bad_body, kv_bad_namespace, kv_bad_key, kv_expiry_not_specified, kv_bad_ttl, kv_bad_expect, kv_unknown_op, kv_prefix_required, kv_too_many_ops, kv_too_many_keys, kv_duplicate_key_in_call, kv_tenant_not_an_input, kv_no_query_string, kv_field_in_path. |
Fix the request. |
locks_bad_request |
400 | locks_bad_body, locks_bad_request, locks_unknown_op, locks_bad_name, locks_ttl_required, locks_bad_ttl, locks_bad_owner, locks_token_required, locks_bad_token, locks_bad_slot, locks_bad_limit, locks_too_many_ops, locks_too_many_keys, locks_duplicate_in_call, locks_tenant_not_an_input. |
Fix the request. |
timers_bad_request |
400 | timers_bad_body, timers_bad_op, timers_queue_required, timers_delay_required, timers_payload_required, timers_payload_not_base64, timers_too_many_ops, timers_server_owned_field, or a read-mode reason on the list routes. |
Fix the request. |
payload_too_large |
413 | kv_value_too_large, kv_key_too_large, timers_payload_too_large, timers_call_too_large. |
Shrink it, or raise the ceiling (see limits). |
timer_horizon_exceeded |
403 | timers_horizon: delayMs is beyond the horizon in force. |
Use a shorter delay. |
kv_quota_exceeded, timers_quota_exceeded |
403 | The tenant is over its KV rows or bytes, or its timer count. Deletes and cancels are never refused for this. | Free space or raise the quota; do not retry unchanged. |
feature_gated |
403 | The tenant has no grant for this surface (QUEEN_KV_REQUIRE_GRANT). |
An operator grants it. |
rate_limited |
429 | The tenant’s KV read or write rate, or the transaction KV rate for the KV writes of a transaction. | Retry after Retry-After. |
kv_unavailable, timers_unavailable |
503 | kv_timeout, kv_retry, kv_no_leader, timers_timeout, timers_retry, no_leader; kv_check_needs_cluster_version_5 for a check while a node of the cluster still runs a release without it; or kv_unavailable when the node already tracks QUEEN_KV_MAX_TENANTS (10,000) tenants. |
Retry after Retry-After. A lock call that failed this way may have applied some of its operations: send it again with the same owners. |
kv_disabled, timers_disabled |
503 | An operator paused the surface at runtime. | Retry later. |
kv_error, timers_error |
500 | A broker fault. | Report it. |
A lost precondition on POST /api/v1/kv is not an error either. An ordinary one shows as
applied: false in its result, and a required one answers 200 with
{"ok": false, "reason": "kv_precondition", "failedIndex", "kvReason", "version", "value"}.
A put or putIfAbsent whose TTL was over when the cluster committed it answers applied: false
with reason: "expired": it holds nothing, and can be sent again.
Cancelling a timer that is not pending answers 200 with ok: false. On POST /api/v1/locks, a
lock somebody else holds answers 200 with acquired: false and reason: "held" (or contended
for a busy semaphore), and a renew or release whose token is no longer the lock’s answers
renewed: false or released: false with reason: "lost".
Ephemeral codes
code |
Status | Meaning | Client |
|---|---|---|---|
ephemeral_bad_request |
400 | The request is malformed: a name, a batch, an id list. | Fix the request. |
ephemeral_queue_not_found |
404 | No ephemeral queue by that name exists. | Check the name. |
queue_full |
429 | The queue is at its maxBytes or maxLength and its overflow policy is reject. |
Back off and push again once consumers drain it. |
rate_limited |
429 | The tenant is over its ephemeral messages-per-second allowance. | Retry after Retry-After. |
ephemeral_quota_exceeded |
403 | The tenant is at its ephemeral byte or queue allowance. | Drain, reset or delete a queue. |
feature_gated |
403 | The tenant is not granted the ephemeral class (QUEEN_EPHEMERAL_REQUIRE_GRANT). |
An operator grants it. |
ephemeral_unavailable |
503 | This node is at QUEEN_EPHEMERAL_MAX_BYTES. |
Retry later. |
ephemeral_disabled |
503 | An operator paused the surface at runtime. | Retry after Retry-After. |
owner_moved, ephemeral_forward_failed |
503 | The partition’s owner changed while the call was relayed, or could not be reached. | Retry. |
internal_only |
403 | A broker-to-broker route was called without the forward mark. | Not a client route. |
Authentication and tenancy
| Status | error |
Client |
|---|---|---|
| 401 | Authentication required, Token has expired, Token not yet valid, Token issued in the future, Invalid token signature, Invalid token issuer, Invalid token audience, Unknown key ID, Bad token header, Unsupported token algorithm |
Get a valid token. |
| 403 | Insufficient permissions |
Use a token whose role covers the route’s access level (HTTP API). |
| 400 | invalid x-queen-tenant header (must be a UUID) |
Fix whatever sets the header (normally the proxy). |
Proxy codes
Behind the embedded proxy, these answers come before the request reaches the broker. Plans and limits are on tenants.
code |
Status | Meaning |
|---|---|---|
unauthorized |
401 | No valid API key or session. |
forbidden |
403 | The credential has no role on this cluster, or is for another cluster. |
rate_limited |
429 | The request rate or the byte rate of the plan, with Retry-After. |
quota_exceeded |
403 | A queue, partition or monthly message quota is reached. |
storage_quota_exceeded |
403 | The retained-bytes quota is reached; pushes are blocked. |
push_blocked |
403 | Pushes are blocked on this cluster (billing hold). |
cluster_suspended |
403 | The cluster is suspended. |
feature_gated |
403 | The route is not in the plan. |
route_blocked |
404 | A broker route that is not served to tenants. |
cluster_unknown |
421 | The Host names no cluster. |
payload_too_large |
413 | The body is over the proxy’s body cap (QUEEN_PROXY_MAX_BODY_BYTES, 16 MiB). |
invalid_request |
400 | A binary push or transaction body whose headers do not say what it weighs uncompressed (binary body: ...), or a DLQ replay that names half a destination. |
bad_gateway, upstream_timeout |
502, 504 | The broker did not answer. |
token_expired |
401 | A client token past its exp. A token function fetches a new one and sends the request again. |
client_blocked |
403 | The client (sub) is on its key’s block list: its tokens are refused, and none is minted for it. |
client_rate_limited |
429 | A scoped key’s own limit, for all its clients or for this one, with Retry-After. Applies in shadow mode too. |
scope_denied |
403 | From the broker: a client token’s request names a queue, partition, group, KV key or timer outside its grants. The message names it. |
scope_invalid, scope_unsupported |
403, 503 | The scope the broker received cannot be read; a proxy that does not run in the broker’s process cannot carry a client token. |
key_unknown, key_ambiguous |
404, 409 | POST /api/v1/tokens by an admin names no scoped key, or a name two keys share: use its id. |
not_configured |
503 | POST /api/v1/tokens on a proxy with no signing key (QUEEN_PROXY_JWT_SECRET or the Ed25519 key). |
Control-plane codes
The /api/cp routes answer the proxy envelope with these codes. With QUEEN_PROXY_CP_TOKEN unset
every route answers 404 not_found; a wrong token answers 401 unauthorized, and a body over
1 MiB 413 payload_too_large.
code |
Status | Meaning |
|---|---|---|
invalid |
400 | A field, a slug or a UUID is not valid. |
conflict |
409 | The record exists, or changed under the call. |
tenant_unknown, cluster_unknown, key_unknown |
404 | No such tenant, cluster or key. |
not_deleting |
409 | A purge or delete of a tenant whose status is not deleting. |
not_purged |
409 | A delete of a tenant whose clusters still list queues at the broker. |
deleting |
409 | A change to a tenant or cluster that is being torn down. |
system_tenant |
409 | The call would act on the broker’s default tenant or the proxy’s own, or the tenant header is off. |
unavailable, no_store |
503 | The proxy’s state is not readable yet. Retry. |
Raft membership codes
/api/v1/system/raft/membership/* answers a refused change with one of these codes: 400 for
bad_request, not_a_learner and not_a_member; 409 for in_flight, no_quorum, last_voter,
learner_behind, already_voter, address_mismatch, membership_changed and leader_changed;
503 for leader_unreachable; and 504 timeout when the deadline (?timeoutMs=, default 30 s)
passed, in which case the leader goes on with the change and refuses another (in_flight) until it
ends, so read the membership before repeating it. A
request that carries a tenant answers 403 forbidden: the membership belongs to the whole cluster.
See cluster.
What the JS client does
- It retries a 5xx or a network error: with one URL up to three attempts, waiting 1 s and then 2 s;
with several URLs and failover on, it tries each URL once. Every 5xx is treated alike, 507 and 503
outcome_unknownincluded. - It retries a 429 in place after
Retry-After, up to 10 attempts, and without limit for a long-poll pop. consume()goes on when a pop still fails after those retries with a 408, a 429, any 5xx but a 503raft_phase1_unsupported, a timeout or a lost connection: it waitsRetry-After, or 1 s doubling up to 30 s when there is none, and polls again, for as long as it takes. Any other error stops it, andconsume()rejects with that error. A request undici refuses before sending it (fetch failedwithUND_ERR_INVALID_ARG) is not a lost connection: it fails the same way every time, so it stopsconsume()too.- A request its caller aborted (a
consume()being stopped) is neither retried nor sent to another URL, and does not mark its node unhealthy: it rejects witherror.abortedset, and a wait between attempts ends there. - Any other 4xx is thrown at once, with
error.statusanderror.codefrom the body. - A transaction rolled back by a lost
requiredprecondition is returned fromcommit(); every other rollback is thrown witherror.reasonset.
Branch on the rollback reasons the broker sends, listed on
transaction. The reason list in the Rust protocol
crate (crates/queen-protocol) still names the 1.x codes ack_rejected, misaligned and
db_error, which 2.0 never sends: a fenced ack that fails is rejected_ack.