Everything QueenMQ does, on one map
Write messages. The first push that names a queue and a partition creates them: there is no partition count to choose.
When to use it
Every producer starts here.
Other names
- Kafka produce, producer
- RabbitMQ publish
- Amazon SQS SendMessage
- Redis XADD
await queen.queue('orders')
.partition('customer-42') // one ordered partition per customer, created by this push
.push([{
transactionId: 'order-1001', // the dedup key: pushing the same id again is a no-op
data: { orderId: 1001, customer: 42, total: 59.9 },
}])A partition is one entity: a customer, an order, a conversation. Order holds inside it, and a slow entity holds up no other. A push with no partition goes to Default.
| Setting | Default |
|---|---|
.partition(name) | Default |
When to use it
Name it after the entity whose events must stay in order, like customer-42.
Other names
- Kafka message key, partition key
- Amazon SQS FIFO queue, message group ID
- Azure Service Bus session
- Google Pub/Sub ordering key
await queen.queue('orders')
.partition('customer-42') // one ordered partition per customer, created by this push
.push([{
transactionId: 'order-1001', // the dedup key: pushing the same id again is a no-op
data: { orderId: 1001, customer: 42, total: 59.9 },
}])A message's transactionId. The same id in the same partition within the dedup window is not written again: the push answers duplicate.
| Setting | Default |
|---|---|
dedupWindowSeconds | 3600 s, set on the queue; at most 7 days, 0 turns it off |
When to use it
Derive it from the work (receipt-${orderId}), never a fresh UUID: then a retry after a timeout is safe.
Other names
- Amazon SQS message deduplication ID
- Kafka idempotent producer
- NATS JetStream Nats-Msg-Id
- Azure Service Bus duplicate detection
await queen.queue('orders')
.partition('customer-42') // one ordered partition per customer, created by this push
.push([{
transactionId: 'order-1001', // the dedup key: pushing the same id again is a no-op
data: { orderId: 1001, customer: 42, total: 59.9 },
}])One push carries up to 10,000 messages, and the broker answers each one: queued, duplicate or error. The HTTP status is 201 even when some are refused.
| Setting | Limit |
|---|---|
messages per push | up to 10,000 |
When to use it
Read each message's status: retry only what was refused, and take a duplicate as an earlier attempt that got through.
Other names
- RabbitMQ publisher confirms
- Kafka delivery report
await queen.queue('emails')
.partition('customer-42')
.push([{ transactionId: 'welcome-42', data: { to: 'ada@example.com' } }])
.onSuccess((items) => console.log('stored', items.length))
.onDuplicate((items) => console.log('already stored', items[0].transactionId))
.onError((items, error) => console.error('refused', error.message))The client collects messages and sends them in batches: when a buffer holds messageCount messages, or when timeMillis has passed.
| Setting | Default |
|---|---|
.buffer({ messageCount, timeMillis }) | off |
bufferMaxMessages | 100,000 per client (JavaScript) |
When to use it
Thousands of small messages a second. A crash loses what is still in the buffer, so call flushAllBuffers() before the process exits.
Other names
- Kafka linger.ms, batch.size
for (let i = 0; i < 1000; i++) {
await queen.queue('events')
.partition(`device-${i % 10}`)
.buffer({ messageCount: 100, timeMillis: 1000 }) // 100 at a time, or every second
.push([{ transactionId: `reading-${i}`, data: { device: i % 10, value: i } }])
}
await queen.flushAllBuffers() // before exiting, send what is leftA push travels as one binary body instead of JSON; with compression the client packs and zstd-compresses each partition's batch. A framed pop reads the records as stored.
| Setting | Default |
|---|---|
binaryPush | false |
pushCompression | false, needs binaryPush |
framedPop | false |
When to use it
A high message rate, when the broker's CPU or the network is the limit. Otherwise leave them off: JSON is easier to read when you debug.
Other names
- Kafka compression.type
const fast = new Queen({
url: 'http://localhost:6632',
binaryPush: true, // a push travels as one binary body, not JSON
pushCompression: true, // with binaryPush: each partition's batch is zstd-compressed here
framedPop: true, // a pop gets the records as stored, and unpacks them here
})
await fast.queue('invoices').partition('customer-42')
.push([{ transactionId: 'invoice-2001', data: { orderId: 2001, total: 12.5 } }])
await fast.queue('invoices').each().limit(1).consume(async (msg) => {
console.log(msg.data) // { orderId: 2001, total: 12.5 }, as with JSON
})A small map of strings stored in clear beside the payload: the message's type, its trace context, its source. A where reads them without reading the payload.
| Setting | Limit |
|---|---|
headers | up to 64 per message, 8 KiB in all |
names | a-z 0-9 . _ - (queen- is reserved) |
When to use it
Route and trace on a header, even on an encrypted queue. No secret in a header: it is never encrypted.
Other names
- Kafka record headers
- RabbitMQ message properties
- Amazon SQS message attributes
- Google Pub/Sub attributes
- Azure Service Bus application properties
await queen.queue('orders').partition('order-9137').push([{
transactionId: 'order-9137-paid',
data: { orderId: 9137, total: 120 },
headers: { type: 'order.paid', traceparent: '00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' },
}])A named set of partitions that share one configuration. A push creates it with the defaults; configure it when you want others.
When to use it
Always configure a queue before its first push: the defaults surprise. A queue a push creates leases for 60 s, one from create() for 300 s, and retention is off, so it keeps everything.
Other names
- Kafka topic
- Google Pub/Sub topic
- NATS JetStream stream
- Redis stream
import { Queen } from 'queen-mq'
const queen = new Queen({ url: 'http://localhost:6632' })
await queen.queue('orders')
.namespace('shop') // groups queues in the dashboard
.config({
leaseTime: 30, // seconds a worker holds a message before it is redelivered
retryLimit: 3, // failures before the message goes to the dead-letter queue
retentionEnabled: true, // retention runs only when this is on
retentionSeconds: 86400, // then messages older than a day are removed
})
.create()How long a worker holds the messages it took before they go to another worker. Long work renews the lease while it runs.
| Setting | Default |
|---|---|
leaseTime | 60 s (300 s from create()) |
.renewLease(true, ms) | off |
When to use it
Short enough that a crashed worker's messages move on quickly; renew it for slow but healthy work.
Other names
- Amazon SQS visibility timeout
- Azure Service Bus lock duration, peek-lock
- Google Pub/Sub ack deadline
- NATS JetStream AckWait
- RabbitMQ consumer timeout
const stop = new AbortController()
process.once('SIGTERM', () => stop.abort()) // a deploy stops the worker cleanly
await queen.queue('tasks')
.group('workers')
.subscriptionMode('all')
.concurrency(4) // four handlers at once, on four partitions
.renewLease(true, 10000) // long tasks keep their lease alive
.idleMillis(60000) // or stop after a minute with nothing to do
.each()
.consume(async (task) => runTask(task.data), { signal: stop.signal })Every new message waits this long after its push before anyone can take it.
| Setting | Default |
|---|---|
delayedProcessing | 0 s |
When to use it
The same delay for every message of a queue. For one message at a time, use a timer.
Other names
- Amazon SQS delay queue, DelaySeconds
await queen.queue('jobs')
.config({
leaseTime: 120, // 2 minutes per job, then it goes to another worker
retryLimit: 5, // after 5 failures, the dead-letter queue
retryDelay: 10000, // a failed job comes back after 10 s, not at once
delayedProcessing: 30, // a new job waits 30 s before anyone can take it
windowBuffer: 5, // a burst waits 5 s, then is handed over at once
dedupWindowSeconds: 86400, // a transactionId is remembered for a day
retentionEnabled: true, // retention runs only when this is on:
completedRetentionSeconds: 3600, // jobs every group has finished go after an hour
maxWaitTimeSeconds: 604800, // anything older than a week goes, finished or not
})
.create()New messages wait in their partition this long, so a burst is handed over as one batch.
| Setting | Default |
|---|---|
windowBuffer | 0 s |
When to use it
A guest sends three chat messages in a row, and the assistant answers all three at once.
Other names
- Also called debounce
// A burst waits 2 seconds, then goes to the handler as one batch.
await queen.queue('chat').config({ windowBuffer: 2 }).create()
// The guest sends three messages in a row.
for (const text of ['Hi!', 'Is breakfast included?', 'And is there parking?']) {
await queen.queue('chat').partition('conv-42').push([{ data: { text } }])
}
// The assistant answers the whole burst once.
await queen.queue('chat')
.group('assistant')
.subscriptionMode('all')
.batch(10) // the handler gets every message of the burst
.consume(async (messages) => {
question = messages.map((m) => m.data.text).join(' ')
console.log(question) // Hi! Is breakfast included? And is there parking?
})How many failed attempts a message gets, and how long it waits before each new one. Its partition waits with it, for that group.
| Setting | Default |
|---|---|
retryLimit | 3 |
retryDelay | 1000 ms |
When to use it
Temporary failures heal by themselves; the limit stops a message that never succeeds from blocking its partition.
Other names
- Amazon SQS maxReceiveCount, redrive policy
- RabbitMQ x-delivery-limit
- Azure Service Bus MaxDeliveryCount
- Google Pub/Sub max delivery attempts
- NATS JetStream MaxDeliver
await queen.queue('jobs')
.config({
leaseTime: 120, // 2 minutes per job, then it goes to another worker
retryLimit: 5, // after 5 failures, the dead-letter queue
retryDelay: 10000, // a failed job comes back after 10 s, not at once
delayedProcessing: 30, // a new job waits 30 s before anyone can take it
windowBuffer: 5, // a burst waits 5 s, then is handed over at once
dedupWindowSeconds: 86400, // a transactionId is remembered for a day
retentionEnabled: true, // retention runs only when this is on:
completedRetentionSeconds: 3600, // jobs every group has finished go after an hour
maxWaitTimeSeconds: 604800, // anything older than a week goes, finished or not
})
.create()Where a message goes after its last retry, with its error, so its group moves on. With both settings off, the message is dropped instead.
| Setting | Default |
|---|---|
deadLetterQueue | true |
dlqAfterMaxRetries | true |
When to use it
Keep both on: a message that fails for good is parked, not lost, and you can replay it once the cause is fixed.
Other names
- RabbitMQ dead letter exchange, DLX
- Amazon SQS dead-letter queue
- Google Pub/Sub dead-letter topic
- Kafka dead letter topic
- Also called dead letter channel
How long a partition remembers transactionIds: a second push with the same id in that time writes nothing and answers duplicate.
| Setting | Default |
|---|---|
dedupWindowSeconds | 3600 s; at most 7 days; 0 turns it off |
When to use it
Cover the time in which your producers may retry a push.
Other names
- Amazon SQS deduplication interval
- Azure Service Bus duplicate detection window
- NATS JetStream duplicate window
await queen.queue('jobs')
.config({
leaseTime: 120, // 2 minutes per job, then it goes to another worker
retryLimit: 5, // after 5 failures, the dead-letter queue
retryDelay: 10000, // a failed job comes back after 10 s, not at once
delayedProcessing: 30, // a new job waits 30 s before anyone can take it
windowBuffer: 5, // a burst waits 5 s, then is handed over at once
dedupWindowSeconds: 86400, // a transactionId is remembered for a day
retentionEnabled: true, // retention runs only when this is on:
completedRetentionSeconds: 3600, // jobs every group has finished go after an hour
maxWaitTimeSeconds: 604800, // anything older than a week goes, finished or not
})
.create()How long messages are kept. Consuming does not delete: a queue is a log, kept for its retention.
| Setting | Default |
|---|---|
retentionEnabled | false |
retentionSeconds | 0: older than this goes, consumed or not |
completedRetentionSeconds | 0: older than this goes, once every group acked it |
maxWaitTimeSeconds | 0: works without retentionEnabled |
When to use it
Keep the history long enough to read it again, and no longer than the disk allows.
Other names
- Kafka retention.ms
- Amazon SQS MessageRetentionPeriod
- Google Pub/Sub message retention duration
- NATS JetStream MaxAge
- RabbitMQ max-age
await queen.queue('jobs')
.config({
leaseTime: 120, // 2 minutes per job, then it goes to another worker
retryLimit: 5, // after 5 failures, the dead-letter queue
retryDelay: 10000, // a failed job comes back after 10 s, not at once
delayedProcessing: 30, // a new job waits 30 s before anyone can take it
windowBuffer: 5, // a burst waits 5 s, then is handed over at once
dedupWindowSeconds: 86400, // a transactionId is remembered for a day
retentionEnabled: true, // retention runs only when this is on:
completedRetentionSeconds: 3600, // jobs every group has finished go after an hour
maxWaitTimeSeconds: 604800, // anything older than a week goes, finished or not
})
.create()Payloads are encrypted on disk. Headers stay in clear, so routing still reads them.
| Setting | Default |
|---|---|
encryptionEnabled | false |
QUEEN_ENCRYPTION_KEY | required on every node |
When to use it
Queues that carry personal data.
Other names
- Amazon SQS server-side encryption, SSE
- Also called encryption at rest
namespace and task group queues in the dashboard and in a discovery pop, one pop over every queue of a namespace or a task.
| Setting | Default |
|---|---|
.namespace(name) | from the queue's name |
.task(name) | from the queue's name |
When to use it
Find a team's queues at a glance, and pop from all of them in one call.
import { Queen } from 'queen-mq'
const queen = new Queen({ url: 'http://localhost:6632' })
await queen.queue('orders')
.namespace('shop') // groups queues in the dashboard
.config({
leaseTime: 30, // seconds a worker holds a message before it is redelivered
retryLimit: 3, // failures before the message goes to the dead-letter queue
retentionEnabled: true, // retention runs only when this is on
retentionSeconds: 86400, // then messages older than a day are removed
})
.create()A queue whose messages come from other queues, with a where and partitions of its own. It holds links of about 20 bytes, not copies.
| Setting | Default |
|---|---|
from | none: a queue, or a list of them |
where | every message |
partitionBy | same |
start | now |
When to use it
A reader that needs settings of its own, partitions of its own or several sources.
Other names
- RabbitMQ exchange, binding, topic exchange
- Amazon SNS filter policy
- Azure Service Bus auto-forwarding, subscription filter
- Also called router, content-based router
await queen.queue('orders-it').config({
from: 'orders',
where: "data.country = 'IT'",
partitionBy: 'same',
leaseTime: 30,
retryLimit: 5,
}).create()The broker copies data between QueenMQ and the systems around it: database tables into queues (sources), queues into tables and buckets (sinks), and Kafka clients through the Kafka port. No other process runs.
| Setting | What it needs |
|---|---|
PUT /api/v1/connectors/:name | one call per source or sink |
QUEEN_ENCRYPTION_KEY | required: a database password is stored sealed with it |
When to use it
Get a table's changes out without Debezium and Kafka Connect, keep a table or a bucket in step with a queue, keep your Kafka clients while you move.
Other names
- Kafka Kafka Connect, connector
- Also called CDC
Reads committed changes through logical replication: every insert, update and delete becomes a message, with exactly-once effects.
| Setting | What it needs |
|---|---|
PostgreSQL | 17 or later |
wal_level | logical |
When to use it
Keep a cache, a search index or another service in step with a table.
Other names
- Kafka Debezium, source connector
- Also called CDC, change data capture
curl -s -X PUT localhost:6632/api/v1/connectors/orders-src \
-H 'content-type: application/json' -d '{
"kind": "source",
"connection": {"host": "pg", "database": "postgres", "user": "postgres",
"password": "secret", "sslMode": "disable"},
"source": {"tables": [{"table": "public.orders", "queue": "orders"}]}
}'Reads the binlog the way a replica does: every committed row change becomes a message, with exactly-once effects.
| Setting | What it needs |
|---|---|
MySQL | 8.0 or later, not MariaDB |
binlog_row_metadata | FULL |
gtid_mode | ON |
When to use it
The same as the PostgreSQL source, for a MySQL table.
Other names
- Kafka Debezium, source connector
- Also called CDC, change data capture
A consumer group the broker runs for you, writing every message into a table.
| Setting | How |
|---|---|
PUT /api/v1/connectors/:name | one call per sink |
When to use it
Replace the consumer you would write to keep a table in step with a queue.
Other names
- Kafka JDBC sink connector, sink connector
curl -s -X PUT localhost:6632/api/v1/connectors/orders-copy \
-H 'content-type: application/json' -d '{
"kind": "sink",
"connection": {"host": "pg", "database": "postgres", "user": "postgres",
"password": "secret", "sslMode": "disable"},
"sink": {"queue": "orders", "table": "public.orders_copy", "mode": "cdc"}
}'Copies chosen queues into a bucket as JSONL or Parquet files that DuckDB, Spark or ClickHouse read. Exactly once: a retried upload rewrites the same file.
| Setting | Default |
|---|---|
QUEEN_S3_EMBEDDED | off until set to true |
QUEEN_S3_QUEUES | the queues to copy |
retentionSinkHold (on the queue) | "default": retention waits for the sink, up to 7 days |
When to use it
History for analytics, audits and replays, on storage that costs less than the brokers' disks.
Other names
- Kafka S3 sink connector
- AWS Firehose
Kafka producers, consumers and admin tools talk to QueenMQ through the Kafka port: a topic is a queue, a key is a partition, a Kafka consumer group is a QueenMQ group.
| Setting | Default |
|---|---|
QUEEN_KAFKA_EMBEDDED | off until set to true |
QUEEN_KAFKA_ADVERTISED_ADDR | required, no default |
When to use it
Move from Kafka one service at a time: change bootstrap.servers and nothing else.
Other names
- Also called Kafka protocol, Kafka API, Kafka-compatible
Take messages. Without a consumer group each message goes to one worker; with groups, every group gets every message.
When to use it
Workers that compete for jobs, and services that each want every event.
Other names
- Kafka poll, consumer
- RabbitMQ basic.consume, basic.get
- Amazon SQS ReceiveMessage
- Google Pub/Sub pull
- Redis XREADGROUP
- Also called competing consumers
await queen.queue('orders')
.each() // the handler gets one message at a time
.limit(1) // stop after one message; a worker leaves this out
.consume(async (msg) => { // acked when the handler returns, retried if it throws
console.log(msg.data)
})Each group reads the whole queue at its own pace, and the workers of one group share it. A group works a partition with one worker at a time.
| Setting | Default |
|---|---|
.group(name) | none: queue mode, each message to one worker |
When to use it
Billing, emails and analytics each want every order: one group each, and none slows the others.
Other names
- Kafka consumer group
- RabbitMQ fanout exchange
- Amazon SNS subscription
- Google Pub/Sub subscription
- Azure Service Bus topic subscription
- NATS JetStream durable consumer
- Redis XGROUP
await queen.queue('orders')
.group('billing') // every group gets every message
.subscriptionMode('all') // a new group starts at the tail unless told otherwise
.batch(100) // up to 100 messages per pop...
.partitions(10) // ...from up to 10 partitions
.renewLease(true, 2000) // heartbeat every 2 s while the handler runs
.autoAck(false) // the handler decides the outcome
.each()
.limit(1)
.consume(async (msg) => {
const ok = msg.data.total > 0
await queen.ack(msg, ok ? 'completed' : 'failed', { group: 'billing' })
})Fixed at the group's first pop: new reads what comes after it, all everything still kept, subscriptionFrom from a time. A seek moves a group later.
| Setting | Default |
|---|---|
subscriptionMode | new |
subscriptionFrom | none |
When to use it
A new projection wants the history, a bug fix replays the last day. A new group skips everything pushed before its first pop unless it asks for all.
Other names
- Kafka auto.offset.reset, earliest, latest
- NATS JetStream DeliverPolicy
- Google Pub/Sub seek
await queen.queue('orders')
.group('audit') // a new group, with its own cursor
.subscriptionFrom('2026-10-22T00:00:00Z') // everything pushed from this time on
.each()
.limit(1)
.consume(async (msg) => {
console.log('again:', msg.data)
})By default the broker sizes every pop: 100 to 1,000 messages, from the partitions that have work. A value you set yourself is never changed.
| Setting | Default |
|---|---|
.batch(n) | autopilot, 100 to 1,000 |
.partitions(n) | autopilot, 1 to 64 |
.wait(bool) | true: waits for messages, up to the timeout |
.timeoutMillis(ms) | 30000, at most 60 s |
.autopilot(false) | one message from one partition per pop |
When to use it
Leave it to the broker, unless the handler needs a batch of a fixed size.
Other names
- Kafka max.poll.records
- Amazon SQS MaxNumberOfMessages, WaitTimeSeconds, long polling
- RabbitMQ prefetch, basic.qos
await queen.queue('events')
.group('archiver')
.subscriptionMode('all')
.batch(100) // up to 100 messages per call
.limit(1000)
.consume(async (messages) => { // no .each(): the handler gets the whole batch
console.log('archiving', messages.length)
}) // all acked if it returns, all nacked if it throwsWith .each() the handler gets one message at a time; without it, the whole batch. When it returns the client acks, when it throws it acks failed; .autoAck(false) leaves the ack to the handler.
| Setting | Default |
|---|---|
.each() | off: the handler gets the batch |
.autoAck(bool) | true |
When to use it
each() for one-at-a-time work, a batch for bulk writes, autoAck(false) when the handler acks in its own transaction.
Other names
- Spring Kafka batch listener
await queen.queue('events')
.group('archiver')
.subscriptionMode('all')
.batch(100) // up to 100 messages per call
.limit(1000)
.consume(async (messages) => { // no .each(): the handler gets the whole batch
console.log('archiving', messages.length)
}) // all acked if it returns, all nacked if it throwsconcurrency(n) runs n handlers, each on its own partitions; renewLease keeps long work leased; limit and idleMillis end the loop; a signal stops it at once and gives back what it had not started.
| Setting | Default |
|---|---|
.concurrency(n) | 1 |
.renewLease(true, ms) | off |
.idleMillis(ms) | none: run until stopped |
{ signal } | none |
When to use it
A deployed worker: several handlers, renewal for long tasks, a clean stop on SIGTERM.
Other names
- Also called consumer threads, graceful shutdown
const stop = new AbortController()
process.once('SIGTERM', () => stop.abort()) // a deploy stops the worker cleanly
await queen.queue('tasks')
.group('workers')
.subscriptionMode('all')
.concurrency(4) // four handlers at once, on four partitions
.renewLease(true, 10000) // long tasks keep their lease alive
.idleMillis(60000) // or stop after a minute with nothing to do
.each()
.consume(async (task) => runTask(task.data), { signal: stop.signal })A group with .conflation() gets only the newest message of each partition, and its cursor moves past the older ones.
| Setting | Default |
|---|---|
.conflation() | off; needs a group, set at its first pop |
When to use it
Commands where only the latest counts, like "recompute the price of A-1": a worker that is behind does the work once.
Other names
- Kafka log compaction, compacted topic
- RabbitMQ last value queue
- Also called LVQ
const [latest] = await queen.queue('prices')
.group('repricer')
.subscriptionMode('all')
.conflation() // only the newest message of each partition
.wait(false)
.pop()
console.log(latest.data) // { sku: 'A-1', price: 12 }: 10 and 11 are skipped
await queen.ack(latest, 'completed', { group: 'repricer' })With .commitOnDelivery() the cursor moves as the messages are handed out: no lease, nothing to ack. Delivery is then at most once.
| Setting | Default |
|---|---|
.commitOnDelivery() | off; pop() only |
When to use it
A live chart that can lose a reading now and then.
Other names
- RabbitMQ autoAck, auto-ack
- Kafka enable.auto.commit
- Also called at most once
const readings = await queen.queue('metrics')
.group('live-chart')
.subscriptionMode('all')
.commitOnDelivery() // the cursor moves as they are handed out: no ack
.batch(100)
.wait(false)
.pop()A group with a where gets only the messages it matches, and its lag counts only those. The where is set at the group's first pop.
| Setting | Default |
|---|---|
.where(field, op, value) | every message |
When to use it
"Only some of these", in one line in the consumer, with no queue to create.
Other names
- RabbitMQ exchange, routing key, binding, topic exchange, headers exchange
- Amazon SNS filter policy
- Azure Service Bus subscription filter, SQL filter
- Google Pub/Sub subscription filter
- NATS JetStream filter subject
- Also called router, content-based router, message filter
await queen.queue('order-events')
.group('billing')
.where('data.kind', 'in', ['order.paid', 'order.refunded'])
.consume(handler)fetch reads a partition by offset: no group, no lease, no ack.
When to use it
An audit, rebuilding one entity, or a sink that keeps its own position.
Other names
- Kafka offset, seek, assign
- RabbitMQ stream offset
- Redis XRANGE
- Also called replay
const read = await queen.queue('orders').fetch(['customer-42']) // no group, lease or ack
for (const record of read.get('customer-42').records) {
console.log(record.offset, record.payload)
}completed moves the cursor. failed spends a retry. retry gives the message back at once, spending none. dlq files it in the dead-letter queue now. consume() acks for you: completed when the handler returns, failed when it throws.
| Setting | What happens |
|---|---|
completed | done, the cursor moves past it |
failed | comes back after retryDelay |
retry | comes back at once |
dlq | dead-lettered now |
When to use it
dlq for input that can never work, like a payload that will not parse; retry for a worker that is shutting down.
Other names
- RabbitMQ basic.ack, basic.nack, basic.reject, requeue
- Amazon SQS DeleteMessage, ChangeMessageVisibility
- Azure Service Bus complete, abandon, dead-letter
- NATS JetStream Ack, Nak, Term
- Kafka commitSync
await queen.queue('emails')
.group('mailer')
.subscriptionMode('all')
.each()
.limit(1)
.consume(async (msg) => sendEmail(msg.data))
.onSuccess((msg) => queen.ack(msg, 'completed', { group: 'mailer' }))
.onError((msg, error) => queen.ack(msg, error.permanent ? 'dlq' : 'failed',
{ group: 'mailer', error: error.message }))Read the dead-letter queue, newest first, with each letter's error. A replay puts a letter back at the end of its partition, once, and takes it out of the dead-letter queue.
| Setting | What it does |
|---|---|
.dlq(group).limit(n).get() | reads it, newest first |
admin.retryMessage(partitionId, transactionId) | replays one letter |
When to use it
Fix the cause, then send the parked messages through again, without anyone pushing them by hand.
Other names
- Amazon SQS DLQ redrive
- RabbitMQ shovel
const dead = await queen.queue('payments').dlq('charger').limit(20).get()
for (const letter of dead.messages) {
console.log(letter.transactionId, letter.errorMessage, letter.retryCount)
}A stream reads a queue or the tenant feed, keeps its state in the broker, and commits its results with the acks of its inputs in one transaction.
When to use it
Counting, summing and grouping over time in a normal process, with no separate stream processor. In the JavaScript, Python, Go and Rust clients.
Other names
- Kafka Kafka Streams, ksqlDB
- Also called stream processing, Flink
import { Stream } from 'queen-mq'
const revenue = await Stream
.from(queen.queue('orders')) // one window per partition: per customer
.windowTumbling({ seconds: 60 })
.aggregate({ count: () => 1, sum: (order) => order.total })
.to(queen.queue('revenue-per-minute')) // one result per customer per minute
.run({ queryId: 'revenue-per-minute', url: 'http://localhost:6632' })Four kinds: tumbling (fixed, one after the other), sliding (overlapping), session (closed after a quiet gap) and cron (aligned to the clock).
| Setting | Example |
|---|---|
windowTumbling | { seconds: 60 } |
windowSliding | { size: 600, slide: 60 } |
windowSession | { gap: 1800 } |
windowCron | { every: 'hour' } |
When to use it
Revenue per minute, logins in the last ten minutes, one window per visit, hourly reports.
Other names
- Kafka tumbling window, hopping window, session window
- Also called windowed aggregation
const visits = await Stream
.from(queen.queue('clicks')) // one partition per visitor
.windowSession({ gap: 1800 }) // a visit ends after 30 minutes of quiet
.aggregate({ count: () => 1 })
.foreach(async (visit) => console.log('visit ended', visit)) // acked after this returns
.run({ queryId: 'visits', url: 'http://localhost:6632' })With eventTime, a window holds the events that happened during it, not the ones that arrived; allowedLateness says how late an event may still count.
| Setting | Default |
|---|---|
eventTime | the arrival time |
allowedLateness | 0 |
onLate | 'drop' |
When to use it
A phone that was offline, a batch upload: count a sale in the hour it happened.
Other names
- Kafka TimestampExtractor
- Also called watermark, late events
const hourly = await Stream
.from(queen.queue('sales'))
.windowCron({
every: 'hour', // on the clock: 10:00 to 11:00, 11:00 to 12:00
eventTime: (msg) => msg.data.soldAt, // when the sale happened, not when it arrived
allowedLateness: 120, // a sale up to 2 minutes late still counts
})
.aggregate({ count: () => 1, sum: (sale) => sale.amount, max: (sale) => sale.amount })
.to(queen.queue('sales-per-hour'))
.run({ queryId: 'sales-per-hour', url: 'http://localhost:6632', subscriptionMode: 'all' })map, filter and flatMap change, drop or split messages; aggregate folds a window into named numbers (count, sum, min, max, avg), reduce is the general fold; keyBy groups by something other than the partition.
When to use it
Most stream jobs are the same few steps: drop noise, reshape, group, fold.
Other names
- Kafka KStream, mapValues, aggregate
const perMinute = await Stream
.fromFeed(queen, { queues: ['orders', 'payments'] })
.windowTumbling({ seconds: 60 })
.aggregate({ count: () => 1, sum: (e) => e.total })
.map((totals, w) => ({ ...totals, customer: w.key, minute: w.windowKey }))
.to(queen.queue('per-minute'))
.run({ queryId: 'per-minute' })gate(fn) decides, message by message, whether a message goes on now, with state kept per partition. A refused message stops its partition and comes back, in order, when its lease runs out.
| Setting | Gates |
|---|---|
tokenBucketGate({ capacity, refillPerSec }) | ready-made |
slidingWindowGate | ready-made |
When to use it
An SMS provider that accepts five messages a second per customer.
Other names
- Also called rate limit, token bucket, throttle
import { tokenBucketGate } from 'queen-mq'
const limiter = await Stream
.from(queen.queue('outgoing-sms')) // one partition per customer
.gate(tokenBucketGate({ capacity: 10, refillPerSec: 1 })) // 10 at once, then 1 a second
.foreach(async (sms) => sendSms(sms))
.run({ queryId: 'sms-limiter', url: 'http://localhost:6632' }).to(queue) pushes each result in the same commit as the state and the acks. .foreach(fn) calls your code and acks after it returns, so a crash calls it again. .foreachBatch hands results over in batches.
| Setting | Delivery |
|---|---|
.to(queue) | exactly once, in the commit |
.foreach(fn) | at least once |
.foreachBatch(fn, { maxItems, maxWaitMillis }) | 1000 items, 0 ms |
When to use it
Results that stay in QueenMQ go to a queue; a call to the outside world goes in foreach, with an idempotency key.
Other names
- Kafka to(), exactly-once, EOS
const toWarehouse = await Stream
.fromFeed(queen, { queues: ['orders'] })
.foreachBatch(async (items) => {
// One write for the batch. An item's id is the same each time it is delivered.
await warehouse.upsert(items.map((it) => ({ key: it.id, ...it.value })))
}, { maxItems: 500, maxWaitMillis: 1000 })
.run({ queryId: 'orders-to-warehouse' })Stream.fromFeed reads every push to the queues you name, in commit order, from the followers. Its where runs in the broker, so a message that fails it never crosses the network.
| Setting | Default |
|---|---|
queues | every queue of the tenant |
from | 'now' |
When to use it
Route, filter or window across several queues at once.
const ordersIt = await Stream
.fromFeed(queen, { queues: ['orders'] })
.where('data.country', '=', 'IT')
.filter((m) => m.data.total >= 10)
.to(queen.queue('orders-it'))
.run({ queryId: 'orders-it' })Wait for one message after another (pattern), for several kinds (join), for n parts (gather), or join a stream with a table (joinTable), each within a time.
| Setting | Example |
|---|---|
within | '15m', '2d', ... |
When to use it
An order not paid within 15 minutes; an order paid and shipped; a batch whose parts all arrived.
Other names
- Kafka KStream join
- Also called CEP, complex event processing
const followups = await Stream
.fromFeed(queen, { queues: ['orders', 'payments'] })
.pattern({
start: (m) => m.queue === 'orders',
then: (m) => m.queue === 'payments',
within: '15m',
onMatch: (order, payment) => ({ order: order.data.id, paid: payment.data.amount }),
onTimeout: (order) => ({ order: order.data.id, remind: true })
})
.to(queen.queue('order-followups'))
.run({ queryId: 'order-followups' })run({ queryId }) names the stream: its position, snapshots and lock. One process runs it at a time; shards spreads it over several; Stream.view reads its state while it runs; onPoison says what a message that keeps failing does.
| Setting | Default |
|---|---|
queryId | required |
exclusive | true: one process at a time |
shards | 1 |
onPoison, retries | 'stop', 3 |
serveState | true |
When to use it
A stream that restarts where it stopped, runs in one place, and can be read while it runs.
Other names
- Kafka application.id, state store
const perTenant = await Stream
.fromFeed(queen, { queues: ['clicks'] })
.keyBy((m) => m.data.tenant)
.windowTumbling({ seconds: 60 }).aggregate({ clicks: () => 1 })
.to(queen.queue('clicks-per-tenant'))
.run({ queryId: 'clicks-per-tenant', shards: 4, shardBy: (m) => m.data.tenant })One call that commits acks, pushes, KV writes and timers together: one entry of the log, all or nothing.
| Part | Limit |
|---|---|
KV operations | at most 64, over 256 keys, one write per key |
Timer operations | one per queue and timer key |
Request body | at most 64 MiB |
When to use it
Every step of a workflow: ack the input, push the next events, save the state, schedule the reminder.
Other names
- Kafka transactions, exactly-once
- Also called outbox, transactional outbox
await queen.queue('orders')
.group('fulfillment')
.subscriptionMode('all')
.autoAck(false)
.each()
.limit(1)
.consume(async (msg) => {
const { orderId, customer } = msg.data
await queen.transaction()
.ack(msg, 'completed', { consumerGroup: 'fulfillment' }) // the input is done
.kv.put('orders', `order-${orderId}`, { status: 'paid' }, { ttl: '7d' })
.queue('shipments').partition(msg.partition) // the next event
.push([{ transactionId: `ship-${orderId}`, data: { orderId } }])
.timer('reminders').key(`review-${orderId}`).delay('24h') // a message tomorrow
.payload({ orderId, customer }).schedule()
.commit()
})A transaction cannot read and then decide. Put a condition on a write instead: expect names the version a key must have, and required: true rolls the whole call back when it misses.
| Setting | Default |
|---|---|
expect | none |
required | false |
When to use it
Two workers read the same version and both decide to write: one wins, the other rolls back with nothing written.
Other names
- Also called compare-and-set, CAS, optimistic locking
await queen.queue('order-events')
.group('workflow')
.subscriptionMode('all')
.autoAck(false)
.each()
.consume(async (msg) => {
const { orderId } = msg.data
const order = await queen.kv.get('orders', orderId) // { status: 'paid' }
if (order.value.status !== 'paid') { // already moved: just ack
return queen.ack(msg, 'completed', { group: 'workflow' })
}
// Do the work of this step here: charge the card, book the courier, ...
const result = await queen.transaction()
.ack(msg, 'completed', { consumerGroup: 'workflow' })
.kv.put('orders', orderId, { status: 'shipping' },
{ ttl: '30d', expect: order.version, required: true }) // only from what we read
.queue('shipping').partition(orderId)
.push([{ transactionId: `ship-${orderId}`, data: { orderId } }])
.commit()
if (result.success === false) return // moved by someone else: nothing written
})A marker written with putIfAbsent and required: true, keyed on the business fact. A second delivery of the same fact rolls back whole, receipts and timers included.
| Setting | Where |
|---|---|
.once(ns, key, { ttl }) | JavaScript and Python; elsewhere putIfAbsent + required |
When to use it
Send the receipt once per payment: key the marker on the paymentId.
Other names
- Also called idempotency key, idempotent consumer, inbox
Forward the message you hold into another queue without its payload: about 20 bytes, written in the same commit.
| Setting | Where |
|---|---|
.link([message], { headers }) | inside a transaction |
When to use it
Relays and pipelines: a hop costs about 20 bytes, whatever the payload weighs.
Other names
- Azure Service Bus auto-forwarding
- Also called forward
await queen.transaction()
.ack(message)
.queue('manual-review').partition(message.partition)
.link([message], { headers: { 'review-reason': 'amount' } })
.commit()A rolled-back call answers HTTP 200 with success: false and a reason: rejected_ack, kv_precondition, duplicate or reserved. The SDKs return it; they do not throw.
| Setting | In the answer |
|---|---|
success | false on a rollback |
reason | why it rolled back |
When to use it
Read success after every commit. duplicate means the step had already committed.
Other names
- Kafka abortTransaction
- Also called abort
Keys and values in the same log as the messages, and the locks built on them. Every write names its lifetime and gives the key a new version; a lock is a lease with one holder, a semaphore one with n.
When to use it
An entity's state, the markers that make a step safe to repeat, counters, and the work no message triggers: a nightly job, a singleton, a pool of n. KV has no queries: the records you search stay in your database.
Other names
- Redis GET, SET
- NATS JetStream KV bucket
- Also called key-value store
// Every write names its lifetime: a ttl, or forever.
await queen.kv.put('carts', 'customer-42', { items: 3 }, { ttl: '7d' })
const cart = await queen.kv.get('carts', 'customer-42') // { found, value, version, ... }
// Write only if nobody wrote since we read it: the version is the condition.
const saved = await queen.kv.put('carts', 'customer-42', { items: 4 },
{ ttl: '7d', expect: cart.version })
console.log(saved.applied) // false if the version moved
// A counter with a ceiling: past max, the increment does not apply.
const hit = await queen.kv.incr('limits', 'api:customer-42', 1, { max: 100, ttl: '1m' })
if (!hit.applied) console.log('over the limit') // reason: 'limit'get, getMany, put and delete. A write names a TTL or forever; a read returns the value, its version and when it expires.
| Setting | Rule |
|---|---|
ttl | every write names one, or forever |
When to use it
The state a step reads before it decides, written back in the step's transaction.
Other names
- Redis GET, SET, EXPIRE
- NATS JetStream KV get, KV put
// Every write names its lifetime: a ttl, or forever.
await queen.kv.put('carts', 'customer-42', { items: 3 }, { ttl: '7d' })
const cart = await queen.kv.get('carts', 'customer-42') // { found, value, version, ... }
// Write only if nobody wrote since we read it: the version is the condition.
const saved = await queen.kv.put('carts', 'customer-42', { items: 4 },
{ ttl: '7d', expect: cart.version })
console.log(saved.applied) // false if the version moved
// A counter with a ceiling: past max, the increment does not apply.
const hit = await queen.kv.incr('limits', 'api:customer-42', 1, { max: 100, ttl: '1m' })
if (!hit.applied) console.log('over the limit') // reason: 'limit'putIfAbsent writes only when the key does not exist: of several callers, one wins.
When to use it
A claim: the first worker to arrive does the work, the others see who has it.
Other names
- Redis SETNX, SET NX
- NATS JetStream KV create
expect makes a write depend on the version you read; check asserts a version and writes nothing. A missed condition answers applied: false with the current value.
| Setting | Default |
|---|---|
expect | none |
check | asserts, writes nothing |
When to use it
Two workers read the same cart and both write it: the second is refused instead of overwriting what it never saw.
Other names
- NATS JetStream revision
- Also called ETag, compare-and-set
// Every write names its lifetime: a ttl, or forever.
await queen.kv.put('carts', 'customer-42', { items: 3 }, { ttl: '7d' })
const cart = await queen.kv.get('carts', 'customer-42') // { found, value, version, ... }
// Write only if nobody wrote since we read it: the version is the condition.
const saved = await queen.kv.put('carts', 'customer-42', { items: 4 },
{ ttl: '7d', expect: cart.version })
console.log(saved.applied) // false if the version moved
// A counter with a ceiling: past max, the increment does not apply.
const hit = await queen.kv.incr('limits', 'api:customer-42', 1, { max: 100, ttl: '1m' })
if (!hit.applied) console.log('over the limit') // reason: 'limit'incr adds to a counter with no read first. With max or min, an increment past the bound does not apply.
| Setting | Default |
|---|---|
incr(ns, key, by, { max, min, ttl }) | no bound |
When to use it
A rate limit in one call: at most 100 calls a minute per customer.
Other names
- Redis INCR, INCRBY
// Every write names its lifetime: a ttl, or forever.
await queen.kv.put('carts', 'customer-42', { items: 3 }, { ttl: '7d' })
const cart = await queen.kv.get('carts', 'customer-42') // { found, value, version, ... }
// Write only if nobody wrote since we read it: the version is the condition.
const saved = await queen.kv.put('carts', 'customer-42', { items: 4 },
{ ttl: '7d', expect: cart.version })
console.log(saved.applied) // false if the version moved
// A counter with a ceiling: past max, the increment does not apply.
const hit = await queen.kv.incr('limits', 'api:customer-42', 1, { max: 100, ttl: '1m' })
if (!hit.applied) console.log('over the limit') // reason: 'limit'getPrefix reads one page of keys under a prefix, with a cursor for the next. Not inside a transaction.
When to use it
List the keys of one customer, or of one job.
Other names
- Redis SCAN MATCH
- Also called list keys
queen.lock(name, { ttl }): acquire() answers false while someone else holds it. The JavaScript, Python, Go, Rust and C++ handles renew it in the background.
| Setting | Default |
|---|---|
ttl | you choose; renewed every third of it |
When to use it
One worker at a time for a nightly report.
Other names
- Redis Redlock
- Also called distributed lock, leader election, mutex
const lock = queen.lock('nightly-report', { ttl: '30s' })
if (!(await lock.acquire({ wait: '1s' }))) return 'skipped' // another worker has it
try {
const report = { day: '2026-10-24', bookings: 412 }
const result = await queen.transaction()
.guard(lock) // commits only while the lock is ours
.queue('daily-reports').push([{ transactionId: `report-${report.day}`, data: report }])
.commit()
if (result.success === false) return 'lost the lock'
return 'done'
} finally {
await lock.release()
}queen.semaphore(name, n): at most n holders at once. One handle is one permit.
When to use it
At most four jobs at a time on a shared resource, across every worker.
Other names
- Also called distributed semaphore, concurrency limit
.guard(lock) on a transaction: it commits only while the lock is still yours. A lock expires without telling its holder; the guard stops a paused process from doing the work twice.
When to use it
Fence the work of a lock holder inside QueenMQ.
Other names
- Also called fencing token
const lock = queen.lock('daily-report', { ttl: '30s' }) // a lease with a fencing token
if (await lock.acquire()) { // false: another worker has it
try {
await queen.transaction()
.guard(lock) // commits only while the lock is ours
.queue('reports').push([{ transactionId: 'report-2026-10-22', data: { day: '2026-10-22' } }])
.commit()
} finally {
await lock.release()
}
}A message scheduled for later, named by its queue and its key. When it fires, the broker pushes it through the log like any push.
When to use it
Reminders, timeouts and follow-ups, with no cron job and no table to scan.
Other names
- Azure Service Bus scheduled messages
- Also called cron, scheduler, reminder
// When an order is delivered, ask for a review in 24 hours.
await queen.timer('review-reminders').key('review-2001').delay('24h')
.payload({ orderId: 2001 }).schedule() // status: 'scheduled'
// The delivery was late: schedule the same key again, and the reminder moves.
await queen.timer('review-reminders').key('review-2001').delay('48h')
.payload({ orderId: 2001 }).schedule() // status: 'rescheduled'
// The guest wrote a review already: no reminder at all.
await queen.timer('review-reminders').key('review-2001').cancel()Scheduling the same key again replaces the timer, so moving a reminder is the same call. A fire is never early, and a little late: the leader checks every 50 ms.
| Setting | How |
|---|---|
timer(queue).key(k).delay(d).schedule() | replaces a timer with the same key |
.headers({ ... }) | the message it fires carries them (2.9) |
When to use it
Ask for a review 24 hours after delivery, and move it when the delivery was late.
Other names
- Azure Service Bus ScheduledEnqueueTime
- RabbitMQ delayed message exchange, x-delay
- Amazon SQS message timer
- Pulsar deliverAt
// When an order is delivered, ask for a review in 24 hours.
await queen.timer('review-reminders').key('review-2001').delay('24h')
.payload({ orderId: 2001 }).schedule() // status: 'scheduled'
// The delivery was late: schedule the same key again, and the reminder moves.
await queen.timer('review-reminders').key('review-2001').delay('48h')
.payload({ orderId: 2001 }).schedule() // status: 'rescheduled'
// The guest wrote a review already: no reminder at all.
await queen.timer('review-reminders').key('review-2001').cancel()A cancel answers absent when there is no timer, which may mean it already fired.
When to use it
The guest already wrote a review: no reminder. A handler that compensates checks the entity's state first.
Other names
- Azure Service Bus CancelScheduledMessage
// When an order is delivered, ask for a review in 24 hours.
await queen.timer('review-reminders').key('review-2001').delay('24h')
.payload({ orderId: 2001 }).schedule() // status: 'scheduled'
// The delivery was late: schedule the same key again, and the reminder moves.
await queen.timer('review-reminders').key('review-2001').delay('48h')
.payload({ orderId: 2001 }).schedule() // status: 'rescheduled'
// The guest wrote a review already: no reminder at all.
await queen.timer('review-reminders').key('review-2001').cancel()Give a timer the entity's partition, and it arrives in order with the entity's other events.
| Setting | How |
|---|---|
.partition(name) | the entity's partition |
When to use it
A timeout that must stay in step with the events of the same order.
Queues in memory, outside the log, for request and reply. Every node serves every queue. No transactions and no replay, and a crash loses what is in flight, by design.
When to use it
A request that waits for its answer now (what is the price of A-1?): speed matters more than history.
Other names
- RabbitMQ Direct reply-to, transient queue
- NATS request-reply
- Also called RPC
const inbox = `inbox-${crypto.randomUUID()}` // a private reply queue
await queen.ephemeral.push('price-requests', [{ sku: 'A-1', replyTo: inbox }])
const reply = await queen.ephemeral.pop(inbox, { wait: true, timeout: 5000, commitOnDelivery: true })
console.log(reply.messages[0]?.payload) // { sku: 'A-1', price: 59.9 }The caller makes a private inbox, sends its name as replyTo, and waits on it with a timeout.
| Setting | What to give |
|---|---|
replyTo | the caller's inbox |
timeout | you choose |
When to use it
A quote with a fallback: wait at most half a second, then show "ask us".
Other names
- RabbitMQ Direct reply-to, reply_to
- NATS inbox
- Also called return address, correlation ID
// The pricing service answers quote requests, in memory.
async function pricing() {
const { messages } = await queen.ephemeral.pop('quotes', { group: 'pricing', wait: true })
for (const m of messages) {
await queen.ephemeral.push(m.payload.replyTo, [{ room: m.payload.room, price: 129 }])
}
await queen.ephemeral.ack('quotes', messages, { group: 'pricing' })
}
// The booking page asks, and waits at most half a second for the answer.
async function quote(room) {
const inbox = `inbox-${crypto.randomUUID()}`
await queen.ephemeral.push('quotes', [{ room, replyTo: inbox }])
const reply = await queen.ephemeral.pop(inbox,
{ wait: true, timeout: 500, commitOnDelivery: true })
return reply.messages[0]?.payload ?? { room, price: null } // too slow: show "ask us"
}
const [deluxe] = await Promise.all([quote('deluxe'), pricing()])
console.log(deluxe) // { room: 'deluxe', price: 129 }A worker pops requests and answers on whatever inbox each one names. Groups, leases and acks work as on a regular queue.
When to use it
A pricing service, a lookup, any answer computed on demand.
Other names
- Also called RPC server, responder
const { messages } = await queen.ephemeral.pop('price-requests', { group: 'pricers', wait: true })
for (const m of messages) {
await queen.ephemeral.push(m.payload.replyTo, [{ sku: m.payload.sku, price: 59.9 }])
}
await queen.ephemeral.ack('price-requests', messages, { group: 'pricers' })A partition keeps its order while one node owns it. When that node restarts, the partition moves, and a message pushed during the move can come out early. A crash loses that node's share.
When to use it
Anything that must survive a crash goes in a regular queue.
Other names
- RabbitMQ transient, non-durable
- NATS Core NATS
Many workers share one stream of jobs, and each job is done by one of them.
One event, read by every service at its own pace.
Each step of the workflow is one transaction.
A request with a deadline, answered through a private inbox in memory.
A timer is a message that waits, named by its queue and a key.
Every replica runs the same scheduler; one of them does the job.
A call that can fail is tried again, then kept aside for a person.
For a value that changes, only the last one matters.
A guest sends three messages in a row; the assistant answers once.
Each customer gets so many messages a second, all of them in order.
Events add up to one record per window: a trace, a session, a minute.
Each reader gets only the messages it wants, and each message is written once.
A table's changes arrive as messages, from inside the broker.
Kafka clients use QueenMQ unchanged; move one service at a time.