Skip to content

Publish/subscribe: one event, many consumer groups

One order-placed event read by billing, shipping and analytics: each consumer group gets every message, and a late group chooses where it starts.

Updated View as Markdown

One event, many readers: when an order is placed, billing charges it, shipping prepares it, and analytics counts it. Each reader must get every event, at its own pace, and a reader added later must be able to start from the beginning.

Each reader is a consumer group. A group gets every message of the queue and keeps its own position; the workers inside a group share its work, as in a work queue.

Orders 3001, 3002 and 3003 are published once to the queue order-placed. Three consumer groups subscribe: billing and shipping from the start, and analytics months later with subscriptionMode all. Each group gets all three orders and keeps its own position. A group created later without subscriptionMode all starts at the end and gets nothing of the past.Orders 3001, 3002 and 3003 are published once to the queue order-placed. Three consumer groups subscribe: billing and shipping from the start, and analytics months later with subscriptionMode all. Each group gets all three orders and keeps its own position. A group created later without subscriptionMode all starts at the end and gets nothing of the past.
Published once, read by every group, also by one that joins later.
const subscribe = (group) => queen.queue('order-placed')
  .group(group)                                   // each group gets every message
  .subscriptionMode('all')
  .each()
  .consume(async (event) => {
    console.log(group, event.data.orderId)
  })

for (const orderId of [3001, 3002, 3003]) {
  await queen.queue('order-placed').partition(`order-${orderId}`)
    .push([{ data: { orderId } }])
}
await Promise.all([subscribe('billing'), subscribe('shipping')])

// Months later, analytics subscribes too, and reads every order ever placed.
await subscribe('analytics')

Billing and shipping each got the three orders. Analytics, started later, read all three from the start.

Why
the queue is a log: reading does not delete. So a new subscriber costs nothing to the publisher, and a group that falls behind slows down only itself.

One thing to know: a new group starts at the end of the queue unless it asks for subscriptionMode('all'), or a point in time with subscriptionFrom. Our check proved it: a group created without it found nothing.

Built from: Pop and consume.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close