Limits and non-goals

This page states Queen’s opinions in one place: the design decisions, how to work with each one, and the operational boundaries. Everything below is a property of the build you can run today, not a roadmap item. Most of these are how we believe a queue should behave; the point of collecting them here is that you can check them against your requirements in one read.

Per-tenant limits belong to the gateway

A bare broker enforces no per-caller quota: it is one namespace and one resource pool. Put queen_proxy in front and every cluster carries a plan: request and message rates with burst, queue and partition counts, payload and batch size, parked long-poll slots, retained bytes and a retention ceiling, plus an optional monthly message allowance. Enforcement is per cluster, and the wire contract is one the SDKs already handle: 429 with Retry-After for a rate limit, which every client retries on its own with backoff, and 403 with a code naming the quota for a hard cap. What each number does at request time, in the order the gateway evaluates them, is in Quotas and rate limits.

Ordering granularity is your partition key

A message’s position is a per-partition BIGINT offset, allocated under that partition’s row lock, so the order is exactly as coarse or as fine as the key you push with. One partition for a whole queue is total ordering across everything, at one writer and one in-flight leased batch per consumer group: a correct configuration, with one writer’s worth of throughput. A partition per customer is a million independent orders. Nothing about this is fixed when the queue is created, because partitions are created by the push that names them.

What you cannot have is both at once. There is no cross-partition sequence number, no global log, and no way to ask “what happened first” across two partitions. Two consumers draining two partitions of the same queue observe two independent orders. That is not a property of this implementation: a total order has to be serialised through one writer, whatever the system underneath it.

The one extra guarantee you get inside a partition is that created_at is monotone, because the timestamp is written under the same FOR UPDATE lock that bumps the offset. Time-based retention and timestamp subscriptions depend on that; it does not extend into a global clock.

Delivery order is arrival order

There is no priority lane, no reordering at pop time, and no per-message deadline: a message’s position is fixed at push. If some work is more urgent than other work, that is a different queue or a different partition, and you dedicate consumers to it.

What you can shift is when a whole lane becomes visible. delayedProcessing defers every message of a queue by N seconds after commit, and windowBuffer holds a bursty partition until it goes quiet or the accumulated batch fills. Both are per-queue options and neither reorders anything. delayedProcessing is enforced in SQL on every pop path and is never skipped; windowBuffer is enforced either by the same SQL debounce or, with QUEEN_HOTLIST on, by the broker’s timer wheel, which is what releases a continuously written partition at first mark plus the window. Both are described in Queue options.

Parallelism inside a group is bounded by partition count

queen.log_consumers is keyed (partition_id, consumer_group) and carries one batch_end and one lease. So exactly one in-flight leased batch exists per (partition, group).

The rule that follows: in-group parallelism equals partition count. A single-partition queue is drained by one worker at a time within a consumer group, and further workers started against it get empty pops. Concurrency comes from partitions. If you want more parallelism, choose a partition key with more distinct values, or add a second consumer group, which is fan-out, not work-sharing, because every group receives every message.

One PostgreSQL is one failure domain and the ceiling

Brokers are stateless and scale horizontally. The database does not. Everything Queen knows lives in one PostgreSQL, so that instance is simultaneously the durability story, the coordination substrate, the throughput ceiling and the single point of failure. High availability means PostgreSQL high availability.

While the database is unreachable, Queen degrades in a specific, asymmetric way. Know which half keeps working before you rely on it:

• Pushes are spooled. When the transaction that would have stored a batch fails, the broker writes those messages to the on-disk spool (FILE_BUFFER_DIR) and answers HTTP 201 with per-item status: "buffered" instead of losing them. A background drain replays them oldest-first when the database returns, preserving each original transactionId so deduplication makes replay idempotent.
• Reads need the database. Pops, acks, transactions and every management route require PostgreSQL, so nothing is consumable while it is away. Buffered messages are not readable until they are drained.
• The spool is bounded by your disk, and by the fact that startup recovery stops after one hour of draining and hands the rest to the background loop. Size the volume for the longest outage you intend to absorb: a long outage is a large directory.

Retention deletes segments, not messages

Retention is opt-in: it requires retentionEnabled plus a positive window. Without both, segments live forever and the disk grows monotonically. When it is on, it runs as bounded autocommitting steps every RETENTION_INTERVAL milliseconds (default 5000), on whichever replica claims the retention row in queen.maintenance_leases for that period.

Granularity is a whole segment, one row holding many frames. So:

• “Delete exactly this one message” is not an operation Queen has. Retention drops the oldest whole segments of a partition, for every consumer group at once.
• DELETE /api/v1/messages/:partitionId/:transactionId removes a dead-letter row. On a queue served by the log engine it cannot remove a live message from a segment.
• Replay is likewise segment-granular. A seek to a timestamp lands on a segment boundary, not on the exact message.
• A partition is deleted only once it is empty and has been idle for PARTITION_CLEANUP_DAYS (30). Emptying it via retention is not enough, and the recreated lane is a new partition starting at offset 0. Never assume offset continuity across a month of silence.

maxWaitTimeSeconds deletes data, including messages currently leased

Treat maxWaitTimeSeconds as an age-based shedding valve for data you are willing to lose, never as a timeout mechanism. If a queue sets it, the maintenance loop deletes whole segments older than the cutoff, for every consumer group, in-flight leases included, regardless of whether retentionEnabled is set. That is data loss by design. It does not dead-letter what it drops, and a consumer whose cursor was inside the deleted range resumes at the next surviving offset without an error: the pop scan tolerates the gap. The eviction is counted in retention analytics; it is not reported to the consumer that lost the messages.

Dead letters are removed only when you remove them

queen.log_dlq grows until you clear it: no retention rule touches it, not the retention window, not the completed-retention window, not maxWaitTimeSeconds. Clearing is one address at a time, or wholesale by deleting the queue, which deletes its dead-letter rows explicitly (the table has no foreign key, so nothing cascades).

What you get for working through it is GET /api/v1/dlq to read, and per-address routes to act on one message. A replay re-pushes the stored snapshot as a new message with a new identity at the tail of the partition, then drops the dead-letter row. It does not reinsert the message at its original position, and redrive is per address, so reprocessing a large dead-letter backlog is a loop you write, and a dead-letter table nobody works through keeps occupying disk.

/configure is a full replace

POST /api/v1/configure writes every queue option from the body it receives, and any option you omit is written back at its default. It is not a patch. Sending {"queue":"orders","options":{"leaseTime":600}} to a queue that already had retention configured switches retention off, and resets dedupWindowSeconds to 3600.

Always send the complete intended configuration. Related, because two defaults circulate: leaseTime is 60 seconds on a queue created implicitly by a push, and 300 seconds once you have called /configure without overriding it.

Transactions are one database transaction, not exactly-once

POST /api/v1/transaction bundles pushes and acks into one PostgreSQL transaction, all-or-nothing. That is a real atomicity guarantee and it is not end-to-end exactly-once delivery. If the response is lost and the client retries, the work is done twice unless the transactionId is deterministic and still inside the deduplication window.

Two related edges: deduplication is bounded by dedupWindowSeconds (default 3600) and is exact only inside it, and autoAck=true commits the cursor inside the pop transaction, which is at-most-once: a crash after the commit and before your handler finishes loses the message.

The broker expects a trusted network segment

It is built to run behind a terminator, on a network only your own services can reach. The properties that follow from that:

• No TLS listener. The broker binds plain HTTP on PORT. Terminating TLS is the proxy’s job (QUEEN_PROXY_TLS_CERT / QUEEN_PROXY_TLS_KEY) or a reverse proxy’s. Broker-to-PostgreSQL TLS is separate and does exist (PG_USE_SSL, PG_SSL_REJECT_UNAUTHORIZED).
• No CORS layer. No middleware sets access-control headers, so a browser on another origin cannot call the broker directly.
• JWT authentication is off by default (JWT_ENABLED=false), and with it off the middleware passes every request through. Access levels are a role set, not a ladder: WriteOnly passes /api/v1/push and is rejected on every read and consume route.
• The mesh port must be firewalled if you run more than one broker. It is framed TCP; the HELLO handshake is HMAC’d, but the nonce is chosen by the dialer and never tracked, so a captured handshake is replayable, and post-handshake frames are unauthenticated JSON, including the frame that sets maintenance mode. This is a requirement, not a hardening tip.
• /metrics/prometheus is not tenant-safe. Per-queue series sum across tenants.
• A malformed or wrong-length QUEEN_ENCRYPTION_KEY disables encryption and stores plaintext on queues flagged for encryption, with only a warning in the log.

The embedded engine is beta, and one per process

The broker can run inside a Rust application (Embed the engine). Its boundaries are its own: one Broker per process lifetime (the admission arbiter is a process-wide singleton, and a start, shutdown, start cycle keeps metering maintenance through the first instance’s arbiter); shutdown is best effort (engine loops without handles run until process exit, with the connection pool closed under them); the outage spool defaults to a per-instance temp directory, so status: "buffered" pushes do not survive a restart unless a stable spool_dir is configured; and the surface is the data plane plus the DLQ, with no consumer-group administration, listings, traces or streams. The Rust API itself is beta: Versions and compatibility states what that means.

Timers and KV state have their own boundaries

Both surfaces are part of the engine and every broker serves them: there is no variable that turns them on, any more than there is one for push and pop, and the boundaries below always apply. What an operator can do is pause a live surface with a runtime kill switch, which answers 503, and bound a tenant with grants and quotas, which answer 403; both are on KV and timers. The routes are on kv and timers; the mechanics are in Timer internals, KV internals and Sweeper internals.

• deliverAt is a floor, never an appointment. A timer is delivered no earlier than the instant you name, and a wake-up that reaches only the broker that scheduled it makes the worst case on another broker one sleep ceiling away, QUEEN_SWEEPER_MAX_SLEEP_MS, which defaults to one second. There is no per-message deadline anywhere in Queen, and this is not one.
• There are no recurring timers. One timer is one delivery. A repeating schedule is a consumer that schedules the next one, which keeps the cancellation story to a single row.
• A cancel after the fire answers absent. The fire deletes the staging row in the same transaction that pushes the message, so there is no tombstone afterwards. absent means no longer pending. It never means “not delivered”, and a compensation path has to check state rather than read a cancel result as a verdict.
• Every KV write carries an expiry. Exactly one of a TTL in seconds or an explicit forever is required on every write; zero or two of them is an error. A store with no natural retention whose TTL is optional grows in silence, and forever is deliberately something you have to type. This is also the difference from streams state, which is scoped to a partition and has no expiry at all.
• The KV read boundary is a key or a prefix, both capped, and prefix reads exist only on the batch endpoint. There is no query language, no secondary index over values, no scan of a whole namespace, and no cross-namespace read.
• A quota on either surface is soft and late. Enforcement is an in-process count refreshed from a rollup, because an exact count would mean one row per tenant locked by every write, in the outermost lock space, held for a whole transaction. Overshoot is bounded, not zero.

Smaller edges

• Payloads must be JSON. The push path parses each payload as a JSON value. Opaque binary is not a payload type. Encode it yourself. The whole request body is capped by QUEEN_MAX_BODY_BYTES (default 64 MiB).
• Every 204 has no body at all, deliberately. An empty pop is a bare status code.
• Consumers poll. Long-polling parks a pop and wakes it, but there is no server-initiated push and no persistent subscription stream.
• Retention cannot delete stream state, and a partition carrying open accumulators is never reclaimed. queen_streams.state keys on partition_id and declines the foreign key on purpose, so partition cleanup checks it explicitly and refuses to delete: it is one of three vetoes, with dead-letter rows and a live lease (Retention). The limitation that follows is that accumulators have no TTL. Nothing purges queen_streams.queries or queen_streams.state, and both are GRANTed to PUBLIC.
• The bundled dashboard and JWT auth do not combine broker-direct. With auth off the broker serves a fully working dashboard as an anonymous operator; with JWT_ENABLED=true it serves an explanation page instead, because the app sends no bearer token and the broker holds no sessions. A dashboard with logins and roles is queen_proxy’s.
• The broker needs a session it keeps. Schema apply (advisory lock 778120010) and the retention leader election (737001) hold session-scoped advisory locks, which live on one backend session. Transaction pooling breaks that and is the only pooling mode that does; session pooling keeps the session intact and is supported, which is how PgBouncer is normally put in front. The residual risk in session mode is a pooler that recycles the server connection underneath a live session, dropping a lock the broker still holds: it surfaces as pg_advisory_unlock reported not-held in the retention log, and connecting directly is what rules it out.
• The broker does not serve an OpenAPI document at runtime. The OpenAPI 3.1 documents published with this site are generated from the router at documentation build time, not by the broker. There is also no data-migration tool: the binary’s migrate subcommand is a retired stub that exits non-zero without touching the database. Backup is pg_dump and an ordinary PostgreSQL restore.
• SDKs exist for JavaScript, Python, Go, Rust, PHP/Laravel and C++, plus a Go operator CLI. There is no Java or .NET client. Those languages use the HTTP API directly.

Where another tool fits better

The measured crossover behind that last row, in both directions, is in Cross-broker comparison.

If none of those is disqualifying, the story behind the trade-offs is in History, the architectural comparison is in Comparison, and the shortest path to a running broker is the Quickstart.