Skip to content

Acknowledge messages: completed, failed, retry

Every ack names an outcome: completed moves the cursor, failed spends a retry, retry redelivers for free, dlq files the message now. Batch acks and the lease that guards them.

Updated View as Markdown

An ack tells the broker what happened to a message. A completed ack moves the consumer group’s cursor past it, and that is all: the message stays in its partition until retention removes it.

One partition with messages at offsets 0 to 7, stored once. The billing group's cursor is after offset 5, and billing holds a lease on offsets 6 and 7. The analytics group's cursor is after offset 2. Acks moved the cursors; no message was deleted. Retention removes messages later, by age.One partition with messages at offsets 0 to 7, stored once. The billing group's cursor is after offset 5, and billing holds a lease on offsets 6 and 7. The analytics group's cursor is after offset 2. Acks moved the cursors; no message was deleted. Retention removes messages later, by age.
An ack moves the group's cursor; the message stays.
Why
the cursor is the group’s memory of where it is. Keeping the message after the ack is what lets other groups, and later replays, read it too.

Four answers

Status Spends a retry What happens, and when to use it
completed no Done. The cursor moves past it. true means the same.
failed yes The lease is released and the message comes back after retryDelay. After the last retry it goes to the dead-letter queue. Use it for failures that may pass. false means the same.
retry no The lease is released and the message comes back at once. Use it to give work back (a worker that is shutting down).
dlq no Filed in the dead-letter queue now. Use it when trying again cannot help (a payload that will never parse).

A failed or dlq ack can carry the reason (error: 'card declined' next to the group), and the dead-letter queue keeps it.

Why
“it did not work” has different meanings. Sometimes trying again in a while is right, sometimes now, and sometimes never. Each answer sends the message where it should go.

One message or a whole batch

queen.ack() takes one message or an array. With an array, one call acks the whole batch.

const messages = await queen.queue('events')
  .group('auditor')
  .subscriptionMode('all')
  .batch(50)
  .wait(true)                         // long poll: wait for messages, never return empty
  .pop()
console.log('auditing', messages.length)
await queen.ack(messages, 'completed', { group: 'auditor' })   // the whole batch at once

Each message of an array can carry its own answer:

await queen.ack([
  { ...first, _status: 'completed' },
  { ...second, _status: 'failed', _error: 'timeout' },
], 'completed', { group: 'moderation' })

In Go, Rust and C++ one batch ack carries one status for all its messages, so their code sends one call per status, or a transaction of acks, where each message has its own.

In a batch, the cursor stops at the first message that will come back: a completed message after it is delivered again too.

A batch of four messages acked with different statuses: offset 0 completed, offset 1 failed, offsets 2 and 3 completed. The cursor moves past offset 0 only, and stops before offset 1: offsets 1, 2 and 3 are delivered again, the completed ones included, so the partition's order holds.A batch of four messages acked with different statuses: offset 0 completed, offset 1 failed, offsets 2 and 3 completed. The cursor moves past offset 0 only, and stops before offset 1: offsets 1, 2 and 3 are delivered again, the completed ones included, so the partition's order holds.
In a batch, the cursor stops at the first message that comes back.
Why
one call for a hundred messages is much cheaper than a hundred calls. The cursor rule keeps order: a message is never passed while an earlier one of the same partition is still to be done.

Three things to know

  • Inside .each(), throw to fail a message (in JavaScript, Python and Go; see Pop and consume for the others). When the handler throws, the client nacks it and skips the rest of that partition’s batch. When the handler acks failed itself and returns, the client does not know: it goes on with the partition’s next messages, their acks are refused (that part of the batch is coming back anyway), and they are handled twice.
  • Every ack answers HTTP 200, even a refused one. Read success in the answer: a refused ack usually means the lease ran out and another worker has the message now.
  • An ack carries its consumer group and its lease. The client takes both from the message. Over HTTP you send them yourself: an ack without its lease is not checked, and can complete a batch another worker holds.

See also

Navigation

Type to search…

↑↓ navigate↵ selectEsc close