Skip to content

API error codes and when to retry

Every error body, status code and code across the API, what each one means, and whether a client should retry, stop, or check the outcome.

Updated View as Markdown

When QueenMQ refuses a call it answers a status code and a JSON body with a stable code, and the code is the part your client should branch on: messages get reworded, codes do not. Most of the decisions a client has to make come down to three questions, and the tables below answer them for every code: can I retry this as it is, do I have to fix the request, or do I need to find out whether it already happened.

A rolled-back transaction is not an error. It answers 200 with success: false and a reason, and those reasons are on transaction.

Error bodies

Where Body Branch on
Most routes: push, pop, ack, lease, queues, consumer groups, DLQ, status {"error": "<message>", "code": "<code>"}, plus "leader" on a 503 retry when the leader is known code
Reads by offset refused for their shape (too many entries, a negative offset, a bad timestamp) {"error": "<message>"} with 400; a per-entry error (UNKNOWN_TOPIC_OR_PARTITION, OFFSET_OUT_OF_RANGE, BAD_CURSOR) inside a 200 the status, then each entry
POST /api/v1/transaction, rolled back {"transactionId", "success": false, "reason", "error", "results": []} reason
/api/v1/kv*, /api/v1/locks and /api/v1/timers* {"error": "<code>", "reason": "<finer code>", "detail": "<message>"} error, then reason
/api/v1/ephemeral/* {"error": "<message>", "code": "<code>"} code
A pop refused for its parameters {"success": false, "error": "<message>", "messages": []} with 400 the status
Authentication and the tenancy header {"error": "<message>"} the status
The proxy, and /api/cp {"error": "<message>", "code": "<code>"} code
Raft membership changes {"ok": false, "code": "<code>", "error": "<message>"} code

The 429 and 503 answers of the engine, KV, lock and timer routes carry Retry-After, in seconds. A 404 for something that does not exist (a queue, a message) may carry only error.

Engine codes

The code of the first shape.

code Status Meaning Client
retry 503 This node could not get the call to a leader: a leader change, no quorum lease, or the leader unreachable from the node that forwarded it. Also an ack, or the ack of a transaction, that would dead-letter a message the node that took it has not read yet (a moment after a leader change): nothing moved. Retry after Retry-After (1 s). A write may have reached the leader, so it must be safe to repeat.
no_leader 503 No leader is known after the hold: an election is running, or a majority is down. Retry.
timeout 503 The request’s deadline passed before an answer. A write may have committed. Retry a write that is safe to repeat; otherwise read to check.
outcome_unknown 503 The leader changed while an ack, nack or DLQ move was being made durable, or a forwarded one was lost after it went out. It may or may not have applied. Send the same ack again: a repeat of an ack that applied is answered as the first one was.
standby 503 The cluster is a standby: it replays another cluster and takes no writes until it is promoted. Every write route answers it, on every node, and nothing was applied. Retry after Retry-After (1 s) if a switch to this cluster is under way: the call is served once the standby is promoted. Otherwise you are pointed at the wrong cluster.
feed_on_leader 421 POST /api/v1/feed reached the node that leads the tenant’s raft group of a cluster: the feed is read from followers, so its readers add no load to the leader. The body lists followers (nodeId, httpAddr). Ask one of the followers; the JavaScript client does. A caller that cannot reach them sends x-queen-feed-relay: 1 and the leader relays the read to a follower; the proxy always does. On a test cluster, QUEEN_FEED_ON_LEADER=true lets the leader serve it.
overloaded 429 The node’s admission budget for writes stayed full for the whole hold. Retry after Retry-After (1 to 5 s).
cluster_version 400 Headers, a where, a reading queue or link, on a cluster where some node does not read catalogue version 8 yet. Upgrade every node; the feature turns on by itself.
reading_queue 400 A push into a reading queue, a keep setting on one, or a link into one. Push to its source; set retention on the source.
where_mismatch 400 A filtered group’s pop with another where than the group’s, or a where on a group that already reads the queue without one. Send the group’s where (the message names it), change it on purpose, or use another group.
bad_where 400 A where that does not parse; the message says where. Fix the condition.
route_loop, source_not_found, bad_source, too_many_sources, too_many_readers 400 A reading queue that would read itself, a source that does not exist or is a filtered group’s, or past the limits. Fix the configuration.
has_readers 409 A delete of a queue that reading queues read (readers lists them). Delete them first, or delete with force=true.
source_gone, source_mismatch 200, success: false A transaction’s link to a message its source no longer keeps, or whose transaction id is not the one at that offset. Nothing to forward; check the message named.
storage_full 507 A node is above its disk high-water mark (QUEEN_RAFT_DISK_HIGH_PCT, 85%). Pushes, KV writes and timer schedules are refused; acks and deletes still work. Free disk or add space, then retry.
name_too_long 413 The tenant, queue and partition or group names together exceed the store’s key budget, counting every key the command derives from them. Shorten the name.
too_many_items 400 A push carries more than QUEEN_PUSH_MAX_ITEMS (10,000) messages. Split the push.
memory_pressure 503 The node is near its memory limit: past QUEEN_MEMGUARD_SOFT_PCT (80 %) it refuses page reads and other large reads, past QUEEN_MEMGUARD_HARD_PCT (90 %) also writes that grow the stored state. Pops, acks, renewals and deletes are always served. Retry-After is 1 to 5 s. Retry after Retry-After.
busy 503 Every one of the node’s large-read slots (QUEEN_HEAVY_READS_MAX, 32) stayed taken for 2 s. Retry-After is 1 to 5 s. Retry after Retry-After.
bad_body 400 The body is not the route’s JSON shape. Fix the request.
bad_request 400 A field is missing or invalid. Fix the request.
any other code 400 The engine refused this request for the reason the message gives. Fix the request; do not retry it unchanged.
no_such_route 404 No route has that method and path. Check the path against the routes.
internal 500 A broker fault. Report it; do not retry in a loop.
raft_phase1_unsupported 503 The node does not serve this call. Report it.

A 503 or a timeout on a write does not mean nothing happened. Make writes safe to repeat before you retry them: a deterministic transactionId on every pushed message, the same lease on a repeated ack, a required marker in a transaction. See dedup.

After a leader change

What a leader change does to a call in flight depends on where the call is remembered.

Pushes, transactions, KV and timer calls are logged under a request id that the node serving the call minted. That node follows the leader change itself and resubmits under the same id until the deadline, and a command that had already committed is answered from its record and does not run again. You get the normal answer, or 503 timeout if the deadline passes first.

Acks, nacks and DLQ moves are served from the leader’s memory and made durable by its next checkpoint, a few milliseconds later. If the leader loses its lead while that checkpoint is on its way to the log, or a forwarded ack is lost after it went out, nobody can say whether it applied, and the answer is 503 outcome_unknown. Send the same ack, with the same lease, again.

Pops are never in doubt: a claim nobody receives is handed back or ends with its lease, and its messages are delivered again. A node that cannot reach the leader answers 503 retry or no_leader, and the request may have reached the leader first.

All of this holds for a cluster. Lease exclusivity across a leader change assumes node clocks within QUEEN_RAFT_MAX_CLOCK_SKEW_MS (500 ms) of each other. See guarantees.

KV, lock and timer codes

The error of the third shape. The reason beside it is finer, and stable too. The locks route is KV underneath, so apart from its own locks_bad_request it answers the KV codes below: the quota, the grant, the rates and the pause are KV’s.

error Status reason, and when Client
kv_bad_request 400 kv_bad_body, kv_bad_namespace, kv_bad_key, kv_expiry_not_specified, kv_bad_ttl, kv_bad_expect, kv_unknown_op, kv_prefix_required, kv_too_many_ops, kv_too_many_keys, kv_duplicate_key_in_call, kv_tenant_not_an_input, kv_no_query_string, kv_field_in_path. Fix the request.
locks_bad_request 400 locks_bad_body, locks_bad_request, locks_unknown_op, locks_bad_name, locks_ttl_required, locks_bad_ttl, locks_bad_owner, locks_token_required, locks_bad_token, locks_bad_slot, locks_bad_limit, locks_too_many_ops, locks_too_many_keys, locks_duplicate_in_call, locks_tenant_not_an_input. Fix the request.
timers_bad_request 400 timers_bad_body, timers_bad_op, timers_queue_required, timers_delay_required, timers_payload_required, timers_payload_not_base64, timers_too_many_ops, timers_server_owned_field, or a read-mode reason on the list routes. Fix the request.
payload_too_large 413 kv_value_too_large, kv_key_too_large, timers_payload_too_large, timers_call_too_large. Shrink it, or raise the ceiling (see limits).
timer_horizon_exceeded 403 timers_horizon: delayMs is beyond the horizon in force. Use a shorter delay.
kv_quota_exceeded, timers_quota_exceeded 403 The tenant is over its KV rows or bytes, or its timer count. Deletes and cancels are never refused for this. Free space or raise the quota; do not retry unchanged.
feature_gated 403 The tenant has no grant for this surface (QUEEN_KV_REQUIRE_GRANT). An operator grants it.
rate_limited 429 The tenant’s KV read or write rate, or the transaction KV rate for the KV writes of a transaction. Retry after Retry-After.
kv_unavailable, timers_unavailable 503 kv_timeout, kv_retry, kv_no_leader, timers_timeout, timers_retry, no_leader; kv_check_needs_cluster_version_5 for a check while a node of the cluster still runs a release without it; or kv_unavailable when the node already tracks QUEEN_KV_MAX_TENANTS (10,000) tenants. Retry after Retry-After. A lock call that failed this way may have applied some of its operations: send it again with the same owners.
kv_disabled, timers_disabled 503 An operator paused the surface at runtime. Retry later.
kv_error, timers_error 500 A broker fault. Report it.

A lost precondition on POST /api/v1/kv is not an error either. An ordinary one shows as applied: false in its result, and a required one answers 200 with {"ok": false, "reason": "kv_precondition", "failedIndex", "kvReason", "version", "value"}. A put or putIfAbsent whose TTL was over when the cluster committed it answers applied: false with reason: "expired": it holds nothing, and can be sent again. Cancelling a timer that is not pending answers 200 with ok: false. On POST /api/v1/locks, a lock somebody else holds answers 200 with acquired: false and reason: "held" (or contended for a busy semaphore), and a renew or release whose token is no longer the lock’s answers renewed: false or released: false with reason: "lost".

Ephemeral codes

code Status Meaning Client
ephemeral_bad_request 400 The request is malformed: a name, a batch, an id list. Fix the request.
ephemeral_queue_not_found 404 No ephemeral queue by that name exists. Check the name.
queue_full 429 The queue is at its maxBytes or maxLength and its overflow policy is reject. Back off and push again once consumers drain it.
rate_limited 429 The tenant is over its ephemeral messages-per-second allowance. Retry after Retry-After.
ephemeral_quota_exceeded 403 The tenant is at its ephemeral byte or queue allowance. Drain, reset or delete a queue.
feature_gated 403 The tenant is not granted the ephemeral class (QUEEN_EPHEMERAL_REQUIRE_GRANT). An operator grants it.
ephemeral_unavailable 503 This node is at QUEEN_EPHEMERAL_MAX_BYTES. Retry later.
ephemeral_disabled 503 An operator paused the surface at runtime. Retry after Retry-After.
owner_moved, ephemeral_forward_failed 503 The partition’s owner changed while the call was relayed, or could not be reached. Retry.
internal_only 403 A broker-to-broker route was called without the forward mark. Not a client route.

Authentication and tenancy

Status error Client
401 Authentication required, Token has expired, Token not yet valid, Token issued in the future, Invalid token signature, Invalid token issuer, Invalid token audience, Unknown key ID, Bad token header, Unsupported token algorithm Get a valid token.
403 Insufficient permissions Use a token whose role covers the route’s access level (HTTP API).
400 invalid x-queen-tenant header (must be a UUID) Fix whatever sets the header (normally the proxy).

Proxy codes

Behind the embedded proxy, these answers come before the request reaches the broker. Plans and limits are on tenants.

code Status Meaning
unauthorized 401 No valid API key or session.
forbidden 403 The credential has no role on this cluster, or is for another cluster.
rate_limited 429 The request rate or the byte rate of the plan, with Retry-After.
quota_exceeded 403 A queue, partition or monthly message quota is reached.
storage_quota_exceeded 403 The retained-bytes quota is reached; pushes are blocked.
push_blocked 403 Pushes are blocked on this cluster (billing hold).
cluster_suspended 403 The cluster is suspended.
feature_gated 403 The route is not in the plan.
route_blocked 404 A broker route that is not served to tenants.
cluster_unknown 421 The Host names no cluster.
payload_too_large 413 The body is over the proxy’s body cap (QUEEN_PROXY_MAX_BODY_BYTES, 16 MiB).
invalid_request 400 A binary push or transaction body whose headers do not say what it weighs uncompressed (binary body: ...), or a DLQ replay that names half a destination.
bad_gateway, upstream_timeout 502, 504 The broker did not answer.
token_expired 401 A client token past its exp. A token function fetches a new one and sends the request again.
client_blocked 403 The client (sub) is on its key’s block list: its tokens are refused, and none is minted for it.
client_rate_limited 429 A scoped key’s own limit, for all its clients or for this one, with Retry-After. Applies in shadow mode too.
scope_denied 403 From the broker: a client token’s request names a queue, partition, group, KV key or timer outside its grants. The message names it.
scope_invalid, scope_unsupported 403, 503 The scope the broker received cannot be read; a proxy that does not run in the broker’s process cannot carry a client token.
key_unknown, key_ambiguous 404, 409 POST /api/v1/tokens by an admin names no scoped key, or a name two keys share: use its id.
not_configured 503 POST /api/v1/tokens on a proxy with no signing key (QUEEN_PROXY_JWT_SECRET or the Ed25519 key).

Control-plane codes

The /api/cp routes answer the proxy envelope with these codes. With QUEEN_PROXY_CP_TOKEN unset every route answers 404 not_found; a wrong token answers 401 unauthorized, and a body over 1 MiB 413 payload_too_large.

code Status Meaning
invalid 400 A field, a slug or a UUID is not valid.
conflict 409 The record exists, or changed under the call.
tenant_unknown, cluster_unknown, key_unknown 404 No such tenant, cluster or key.
not_deleting 409 A purge or delete of a tenant whose status is not deleting.
not_purged 409 A delete of a tenant whose clusters still list queues at the broker.
deleting 409 A change to a tenant or cluster that is being torn down.
system_tenant 409 The call would act on the broker’s default tenant or the proxy’s own, or the tenant header is off.
unavailable, no_store 503 The proxy’s state is not readable yet. Retry.

Raft membership codes

/api/v1/system/raft/membership/* answers a refused change with one of these codes: 400 for bad_request, not_a_learner and not_a_member; 409 for in_flight, no_quorum, last_voter, learner_behind, already_voter, address_mismatch, membership_changed and leader_changed; 503 for leader_unreachable; and 504 timeout when the deadline (?timeoutMs=, default 30 s) passed, in which case the leader goes on with the change and refuses another (in_flight) until it ends, so read the membership before repeating it. A request that carries a tenant answers 403 forbidden: the membership belongs to the whole cluster. See cluster.

What the JS client does

  • It retries a 5xx or a network error: with one URL up to three attempts, waiting 1 s and then 2 s; with several URLs and failover on, it tries each URL once. Every 5xx is treated alike, 507 and 503 outcome_unknown included.
  • It retries a 429 in place after Retry-After, up to 10 attempts, and without limit for a long-poll pop.
  • consume() goes on when a pop still fails after those retries with a 408, a 429, any 5xx but a 503 raft_phase1_unsupported, a timeout or a lost connection: it waits Retry-After, or 1 s doubling up to 30 s when there is none, and polls again, for as long as it takes. Any other error stops it, and consume() rejects with that error. A request undici refuses before sending it (fetch failed with UND_ERR_INVALID_ARG) is not a lost connection: it fails the same way every time, so it stops consume() too.
  • A request its caller aborted (a consume() being stopped) is neither retried nor sent to another URL, and does not mark its node unhealthy: it rejects with error.aborted set, and a wait between attempts ends there.
  • Any other 4xx is thrown at once, with error.status and error.code from the body.
  • A transaction rolled back by a lost required precondition is returned from commit(); every other rollback is thrown with error.reason set.

Branch on the rollback reasons the broker sends, listed on transaction. The reason list in the Rust protocol crate (crates/queen-protocol) still names the 1.x codes ack_rejected, misaligned and db_error, which 2.0 never sends: a fenced ack that fails is rejected_ack.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close