A failed message comes back after retryDelay, up to retryLimit times, then goes to the dead-letter queue and its partition moves on. Read the DLQ and replay what you fixed.

Every failed ack spends one retry. The message comes back after the queue’s retryDelay, and its partition waits with it, for that consumer group. After retryLimit failures, the message goes to the dead-letter queue and the group moves on. You set both on the queue (see Create and configure queues).

Why

retrying is how temporary failures heal by themselves, and the limit is how a message that can never succeed stops blocking everything behind it in its partition.

Two cases do not spend a retry: a retry ack, and a lease that runs out. So a message that crashes its worker’s process every time comes back forever and never reaches the dead-letter queue. If some input can crash your worker, check for it before the work starts and ack such a message with dlq.

A message is delivered, leased to a worker, and ends in one of four ways. Ack completed: the cursor moves past it. Ack failed with retries left: it comes back after retryDelay and one retry is spent. Ack retry, or a lease that ran out: it comes back at once and no retry is spent. The last try failed, or ack dlq: it goes to the dead-letter queue with its payload, error and retries. From the dead-letter queue, a replay pushes a copy to the end of the queue. — The life of a message: four endings, and the replay.

What a dead letter keeps

A dead letter keeps the payload, the error the worker sent, the number of retries, and the consumer group that gave up on it. It stays until you replay it or delete it. Each group has its own: a message that failed for billing can be fine for analytics.

Why

a failure you cannot see is a failure you cannot fix. The dead-letter queue is where they wait for a person, or for a fix in the code.

Some SDKs lose the error text on the way in 2.6.0: a Go or C++ transaction ack with dlq, and the automatic nack of PHP’s consume loop, file the message as “Retries exhausted”. Ack with failed or dlq and an error outside a transaction when the reason matters.

Read it

const dead = await queen.queue('payments').dlq('charger').limit(20).get()
for (const letter of dead.messages) {
  console.log(letter.transactionId, letter.errorMessage, letter.retryCount)
}

dead = await queen.queue("payments").dlq("charger").limit(20).get()
for letter in dead["messages"]:
    print(letter["transactionId"], letter["errorMessage"], letter["retryCount"])

dead, err := client.Queue("payments").DLQ("charger").Limit(20).Get(ctx)
if err != nil {
	log.Fatal(err)
}
for _, letter := range dead.Messages {
	fmt.Println(letter.TransactionID, letter.ErrorMessage, letter.RetryCount)
}

let dead = queen.queue("payments").group("charger").dlq(Some(20), None).await?;
for letter in &dead.messages {
    let id = letter.transaction_id.as_deref().unwrap_or_default();
    let error = letter.error.as_deref().unwrap_or_default();
    // retryCount has no field of its own in the Rust client: it is in `rest`
    let retries = letter.rest.get("retryCount").cloned().unwrap_or_default();
    println!("{id} {error} {retries}");
}

$dead = $queen->queue('payments')->dlq('charger')->limit(20)->get();
foreach ($dead['messages'] as $letter) {
    echo $letter['transactionId'], ' ', $letter['errorMessage'], ' ',
        $letter['retryCount'], "\n";
}

json dead = client.queue("payments").dlq("charger").limit(20).get();
for (const json& letter : dead["messages"]) {
    std::cout << letter["transactionId"] << " " << letter["errorMessage"] << " "
              << letter["retryCount"] << "\n";
}

limit() and offset() page through it, newest first. The JavaScript client also has from() and to(), but the 2.6.0 broker ignores them.

Replay it

Once the cause is fixed, put the messages back:

for (const letter of dead.messages) {
  // back into its partition, and out of the dead-letter queue
  await queen.admin.retryMessage(letter.partitionId, letter.transactionId)
}

for letter in dead["messages"]:
    # back into its partition, and out of the dead-letter queue
    await queen.admin.retry_message(letter["partitionId"], letter["transactionId"])

for _, letter := range dead.Messages {
	// back into its partition, and out of the dead-letter queue
	_, err := client.Admin().RetryMessage(ctx, letter.PartitionID, letter.TransactionID)
	if err != nil {
		log.Fatal(err)
	}
}

for letter in &dead.messages {
    // back into its partition, and out of the dead-letter queue
    if let (Some(partition_id), Some(transaction_id)) =
        (&letter.partition_id, &letter.transaction_id)
    {
        queen.admin().retry_message(partition_id, transaction_id).await?;
    }
}

foreach ($dead['messages'] as $letter) {
    // back into its partition, and out of the dead-letter queue
    $queen->admin()->retryMessage($letter['partitionId'], $letter['transactionId']);
}

for (const json& letter : dead["messages"]) {
    // back into its partition, and out of the dead-letter queue: the broker's
    // replay route, through the client's own HTTP layer (no call for it in C++)
    const std::string id = letter["id"];
    client.get_http_client()->post("/api/v1/dlq/" + id + "/replay");
}

A replay moves the dead letter back into its partition, at the end, as a new message with the transactionId dlq:<id>, and removes it from the dead-letter queue in the same step. Doing it twice cannot create a second copy. Every group that reads that partition sees the replayed message, not only the group that gave up on it. Over HTTP the same thing is POST /api/v1/dlq/:id/replay.

Why

the fix usually comes after the failure: a bug deployed, a provider back up. Replay lets the same messages go through again without anyone pushing them by hand.

Skip it, or turn it off

An ack with dlq files a message straight away, without spending its retries. Setting deadLetterQueue and dlqAfterMaxRetries to false on the queue drops a message that runs out of retries instead of keeping it.

Why

some messages are worth nothing once they fail (a typing indicator, a stale price), and keeping them would only fill a list nobody reads.

Retries and the dead-letter queue

What a dead letter keeps

Read it

Replay it

Skip it, or turn it off

See also