Skip to content

Workflow pattern: one step, one transaction

An order moves from paid to shipped: each step acks its input, writes the order's state with a version check and pushes the next event, in one transaction.

Updated View as Markdown

An order moves through states: paid, shipping, shipped. A redelivered event, two workers, or an event that arrives late must never move it twice or backwards, and a step must never be half done.

The order’s state lives in KV. Each step is one transaction: it acks the event, writes the new state only if the state is still the version the worker read (expect), and pushes the next event. If the version moved, the whole transaction rolls back, and nothing is written.

The state of order-2001 lives in KV: paid, at version 1. The workflow step reads it, does its work, and commits one transaction: the ack of the event, the KV write to shipping only if the version is still 1, and the push of the next event to shipping. The same step sent again from the old version finds version 2: the whole transaction rolls back and nothing is written, so there is still one shipping event.The state of order-2001 lives in KV: paid, at version 1. The workflow step reads it, does its work, and commits one transaction: the ack of the event, the KV write to shipping only if the version is still 1, and the push of the next event to shipping. The same step sent again from the old version finds version 2: the whole transaction rolls back and nothing is written, so there is still one shipping event.
A state machine whose steps cannot race.
await queen.queue('order-events')
  .group('workflow')
  .subscriptionMode('all')
  .autoAck(false)
  .each()
  .consume(async (msg) => {
    const { orderId } = msg.data
    const order = await queen.kv.get('orders', orderId)            // { status: 'paid' }
    if (order.value.status !== 'paid') {                   // already moved: just ack
      return queen.ack(msg, 'completed', { group: 'workflow' })
    }

    // Do the work of this step here: charge the card, book the courier, ...

    const result = await queen.transaction()
      .ack(msg, 'completed', { consumerGroup: 'workflow' })
      .kv.put('orders', orderId, { status: 'shipping' },
        { ttl: '30d', expect: order.version, required: true })  // only from what we read
      .queue('shipping').partition(orderId)
      .push([{ transactionId: `ship-${orderId}`, data: { orderId } }])
      .commit()
    if (result.success === false) return  // moved by someone else: nothing written
  })

The step moved order-2001 to shipping and pushed one shipping event. The same step sent again from the old version rolled back: still one event.

Why
the ack, the new state and the next event commit together or not at all. That replaces the outbox table, the idempotency table and the lock you would put around them.

One thing to know: a rollback is an answer, not an error. commit() returns success: false (in Go, IsKVPrecondition() on the result) and does not throw. Read it.

Built from: Transactions, KV, locks and timers.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close