Idempotent producers

A producer that retries after a timeout does not know whether the first attempt landed. Queen answers that with an exact, server-side duplicate check on the transactionId, enforced inside the same lock that allocates the message’s offset, so a duplicate writes nothing at all rather than being written and later reconciled.

It is on by default, with a one-hour window.

The mechanism

const res1 = await client
.queue('test-queue-v2')
.push([{ transactionId: 'test-transaction-id', data: { message: 'Hello, world!' } }])

const res2 = await client
.queue('test-queue-v2')
.push([{ transactionId: 'test-transaction-id', data: { message: 'Hello, world!' } }])
// res1[0].status === 'queued', res2[0].status === 'duplicate'

Both pushes carry the same transactionId. The first returns queued; the second returns duplicate and stores nothing.

What happened between the two calls is the part worth knowing. On a queue with deduplication enabled, a push:

Because the probe happens under the write serializer, no concurrent producer can commit a colliding message between the probe and the allocation. The check is exact (a hash match, not a Bloom filter), and there is no false-positive path.

Choosing the transaction id

The transactionId is the idempotency key. If you leave it out, the SDK generates a fresh one per item, which makes every push unique and deduplication a no-op. To get idempotency, derive it deterministically from the thing you are publishing:

// Illustrative, not extracted from a test.
// Deterministic: the same order event always produces the same id.
await client.queue('orders').partition(order.id).push([
  { transactionId: `order:${order.id}:v${order.version}`, data: order }
])

Two properties to design around.

The scope is one partition. The probe is WHERE t.partition_id = <this partition>. The same transactionId pushed to two different partitions of the same queue is accepted twice: both return queued. That is verified behaviour, not an accident: partitions are independent lanes and the check lives inside one. Route an entity’s events to that entity’s partition and the key only has to be unique within it.

First wins inside one request too. Two items in the same push body sharing a transactionId, queue and partition produce one frame. The repeat becomes a follower of the leader and echoes the leader’s final message_id. This layer runs in the broker, before SQL sees the batch.

What a duplicate response looks like

The push response is one item per request item, in order, at HTTP 201:

[
  {"index":0,"message_id":"01991b2c-...","transaction_id":"order:8812:v3","queueName":"orders","status":"queued"},
  {"index":1,"message_id":"01991b2c-...","transaction_id":"order:8812:v3","queueName":"orders","status":"duplicate"}
]

The message_id on a duplicate item is the original message’s id, not a new one: the broker resolves it by fetching the covering segment of the original occurrence and unpacking that frame. When the original can no longer be read (its segment was already deleted by retention, for instance) the field degrades to the all-zero UUID 00000000-0000-0000-0000-000000000000, which is the “original unknown” sentinel. A duplicate verdict with a zero id still means “not written”; it only means the id of the message it collided with is no longer recoverable.

The window

The window is dedup_window_seconds on the queue, default 3600. It is measured from the stored creation time of the original message’s segment, so the guarantee is: a repeat of a transactionId is rejected for that many seconds after the original was committed to that partition.

Set it explicitly when you configure the queue:

{
  "queue": "orders",
  "options": { "dedupWindowSeconds": 86400 }
}

dedupWindowSeconds: 0 disables the check. The push then skips the probe entirely and takes a single UPDATE ... RETURNING allocator path. Explicit acknowledgement still works on a queue with deduplication off, because log_txns is written regardless and the ack path resolves transaction ids through it.

After the window closes

The hash sidecar is not kept forever. The retention loop purges queen.log_txns rows older than

GREATEST(dedup_window_seconds, completedRetentionSeconds, 900)

seconds, per partition, in bounded autocommitting steps. The floor of 900 seconds means the sidecar is always at least 15 minutes deep even on a queue with deduplication off, because the ack path needs it.

The consequence for a producer is simple and needs to be stated plainly: once the window has passed, a repeat of the same transactionId is accepted as a new message. There is no error, no warning and no marker on the message. Deduplication is a bounded protection against retry storms and at-least-once delivery from upstream. It is not a permanent uniqueness constraint. If your invariant is “this order is only ever processed once, forever”, that invariant belongs in your own datastore, keyed by your own id.

The same window bound applies to acknowledgement: an ack whose transaction id has aged out of the sidecar is reported as unresolvable rather than silently succeeding. See Errors, backpressure and retries.

The broker-side cache changes nothing you can observe

The broker keeps an in-memory cache of in-window hashes per partition (QUEEN_DEDUP_CACHE, on by default, sized by QUEEN_DEDUP_CACHE_MB, default 512). Its only output is a watermark it sends with the push, telling SQL “none of these hashes occur at or below offset X”, which lets the SQL probe scan a shorter span.

Whenever the cache cannot vouch for a span (unknown partition, a hydration gap, memory pressure, the kill switch), it answers with the sentinel that forces a full-window probe. So a cold, evicted, degraded or disabled cache produces exactly the same verdict as a warm one. The cache is a CPU and I/O optimization; the SQL probe is the authority. Turning it off costs throughput and changes no result.

What to expect

Next