← Use cases

Idempotent webhook deduplication, without Redis.

Stripe, Booking, Octorate, Twilio, GitHub all redeliver webhooks — sometimes within minutes, sometimes hours later. Your handler running twice means a double charge, a duplicate email, a duplicate row. The standard fix is a Redis SET with a TTL; the standard problem is that now you have Redis to operate, an eviction story to design, and an outage waiting when Redis fills up. Queen replaces all of that with a streaming window whose seen-set state lives in PostgreSQL and is garbage-collected automatically when the window closes.

Numbers are from a verified run with 1,000 events + 300 random duplicates over 5 providers, 10-second tumbling windows. Reproduce with examples/use-case-dedup.js.

1. The problem

External providers retry webhook deliveries on any non-2xx response, on read timeouts, and sometimes — especially during platform-side incidents — at random. The contract is "at-least-once". Real redelivery patterns we've seen:

• Stripe retries with exponential backoff for up to 3 days on non-2xx. idempotency-key is exposed but optional on the consumer side.
• Octorate / channel managers deliver the same inventory-update event twice within seconds when their multi-region pipeline fans out.
• Twilio retries 11 times over 24h on 5xx responses.
• GitHub redelivers any failed webhook for 8 hours, and the UI offers manual redelivery — which support engineers click multiple times in a panic.

The standard fix:

// The naïve handler — broken under redelivery.
app.post('/webhooks/stripe', async (req, res) => {
  await charge(req.body)             // ← runs twice on retry
  res.send('ok')
})

// The standard fix — adds Redis to your stack.
app.post('/webhooks/stripe', async (req, res) => {
  if (await redis.set(`seen:${req.body.id}`, 1, 'EX', 3600, 'NX') === null) {
    return res.send('ok')            // already processed
  }
  await charge(req.body)
  res.send('ok')
})

Now you operate Redis. You worry about its TTL policy when it approaches max-memory, about eviction kicking in mid-incident, about it being a SPOF for every webhook integration in your platform.

2. The pattern

A webhook ingestor pushes raw webhook bodies onto a Queen queue partitioned by provider. A streaming query tumbles into N-second windows per provider, holds a seen-set of eventIds in state, and emits each event downstream only on its first observation within that window. State lives in PostgreSQL; when the window closes, the row is deleted in the same transaction that emits the final accumulator.

Two key properties:

• State is durable — the seen-set is a JSONB accumulator inside queen_streams.state, replicated by your PG backup. Restart Node, kill the worker, redeploy the broker — state is untouched.
• State is garbage-collected automatically — when the window closes, the runner emits the accumulator to the sink and deletes the state row in the same PG transaction. No TTL job, no Redis eviction story, no operational surface area.

3. Quickstart

import { Queen, Stream } from 'queen-mq'

const url = 'http://localhost:6632'
const q   = new Queen({ url, handleSignals: false })

const Q_INCOMING = 'webhooks.incoming'
const Q_VERIFIED = 'webhooks.verified'

await q.queue(Q_INCOMING).config({
  leaseTime:        30,
  retryLimit:       3,
  retentionEnabled: true,
  retentionSeconds: 3600
}).create()
await q.queue(Q_VERIFIED).config({
  leaseTime:        30,
  retentionEnabled: true,
  retentionSeconds: 3600
}).create()

// 10-minute tumbling windows, per-provider seen-set, emit firsts to verified.
const stream = await Stream
  .from(q.queue(Q_INCOMING))
  // Default key = partition_id = providerId — what we want.
  .windowTumbling({ seconds: 600, idleFlushMs: 5000 })
  .reduce(
    (acc, msg) => {
      if (acc.seen.includes(msg.eventId)) return acc       // dup → drop
      return {
        seen:   [...acc.seen,   msg.eventId],
        firsts: [...acc.firsts, msg]
      }
    },
    { seen: [], firsts: [] }
  )
  .flatMap(window => window.firsts)                       // explode firsts
  .to(q.queue(Q_VERIFIED))
  .run({
    queryId:       'webhook-dedup',
    url,
    batchSize:     200,
    maxPartitions: 8,
    maxWaitMillis: 200,
    reset:         true
  })

// Downstream sees each unique webhook event exactly once per window.
await q.queue(Q_VERIFIED)
  .group('charge-processor')
  .batch(50)
  .each()
  .consume(async (msg) => {
    await chargeCustomer(msg.data)   // safe to run — deduplicated upstream
  })

// Verified run, 1000 unique + 300 duplicates injected over 5 providers:
//   events arriving at incoming:  1300
//   events through to verified:   1000   (expected 1000)
//   duplicates leaked downstream: 0      (expected 0)
//   dedup ratio:                  23.1%  (matches injected ratio)

4. Inspecting state via SQL

While a window is open, the seen-set for that provider is a JSONB field in queen_streams.state. You can monitor dedup effectiveness directly:

SELECT
    s.key                                              AS provider_or_partition,
    jsonb_array_length(s.value->'acc'->'seen')         AS distinct_events_in_window,
    jsonb_array_length(s.value->'acc'->'firsts')       AS forwarded_downstream,
    to_timestamp(((s.value->>'windowStart')::bigint / 1000.0)) AS window_start,
    to_timestamp(((s.value->>'windowEnd')::bigint / 1000.0))   AS window_end
FROM queen_streams.state s
JOIN queen_streams.queries q ON q.id = s.query_id
WHERE q.name = 'webhook-dedup'
ORDER BY distinct_events_in_window DESC;

Connect this to your existing dashboards as you would any PG table. When the window closes the row disappears; the closed-window aggregate is emitted to webhooks.verified as a regular Queen message that downstream consumers process.

5. Window sizing — how long to dedup for

The window size determines the maximum gap between an original delivery and a redelivery that the system will still deduplicate. Match it to the redelivery window of the upstream provider:

6. vs. Redis / outbox / hand-rolled

7. Common pitfalls

8. References

← Back to use cases