Timer internals

A timer is a message that is not in the log yet. queen.log_timers is where it waits, and the log sees it only at the fire, inside one transaction that both pushes the frame and deletes the staging row.

Why a timer cannot live in the log

This is a proof rather than a preference, and it has three independent legs.

The pop is a contiguous offset scan from committed + 1. A frame with a future delivery time sitting in the middle of that scan either blocks the cursor for every consumer of the partition, or is skipped and never delivered.

A row inside a solid zstd blob is not deletable. queen.log_segments.blob is opaque to SQL and carries STORAGE EXTERNAL precisely because the broker has already compressed it, so there is no statement that removes one frame from a segment. A cancellation would have nothing to act on.

And a pending future frame would pin its own segment against log_start for the whole wait. A timer set ninety days out would keep ninety days of segments alive behind it.

So the frame is staged in an ordinary row, with ordinary indexes and an ordinary DELETE, and it enters the log at the moment it becomes deliverable.

The table

CREATE TABLE IF NOT EXISTS queen.log_timers (
    tenant_id     UUID NOT NULL DEFAULT '00000000-0000-0000-0000-000000000001',
    queue         TEXT COLLATE "C" NOT NULL,
    timer_key     TEXT COLLATE "C" NOT NULL,
    "partition"   TEXT NOT NULL DEFAULT 'Default',
    deliver_at    TIMESTAMPTZ NOT NULL,
    txn           TEXT NOT NULL,
    message_id    UUID NOT NULL,
    payload       BYTEA NOT NULL,
    payload_zstd  BOOLEAN NOT NULL DEFAULT FALSE,
    encrypted     BOOLEAN NOT NULL DEFAULT FALSE,
    producer_sub  TEXT,
    attempts      INT NOT NULL DEFAULT 0,
    last_error    TEXT,
    claimed_until TIMESTAMPTZ,
    claim_token   UUID,
    shard         SMALLINT NOT NULL
                  GENERATED ALWAYS AS ((hashtextextended(timer_key, 0) & 63)::smallint) STORED,
    visible_at    TIMESTAMPTZ NOT NULL
                  GENERATED ALWAYS AS (
                      CASE WHEN claimed_until IS NULL OR claimed_until < deliver_at
                           THEN deliver_at ELSE claimed_until END) STORED,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (tenant_id, queue, timer_key)
);

Keyed by names, and the second reason is the stronger one

The primary key is (tenant_id, queue, timer_key). There is no queue id and no partition id anywhere in this table.

The obvious reason is that at schedule time the destination queue may not exist: provisioning is lazy and happens at the fire, inside the same code path a push uses. The in-house precedent is queen.hotlist_repairs, keyed by names for the same motive.

The decisive reason is different. Because no column names a partition, this table adds no leg to queen.log_partition_dead_v1, so that per-partition scan does not get more expensive as timers accumulate. And the invariant has a free compile-time check: log_partition_dead_v1 is LANGUAGE sql, so it resolves its tables at creation time; it lives in 006 and this table lives in 025. Adding a leg that names queen.log_timers kills the boot rather than silently making maintenance quadratic.

A third consequence falls out of the same choice and is worth stating because it is what a reader of the schedule path notices first: the schedule never writes queen.queues. Validating the destination queue at schedule time would put two statements that write queen.queues with different row sets in one transaction wire, and two such transactions wait on each other on the unique index, which per-statement ordering does not prevent because they are different statements. So there is no validation at schedule, and the queue is born at the fire.

visible_at, and the price of indexing it

CREATE INDEX IF NOT EXISTS idx_log_timers_visible
    ON queen.log_timers (shard, visible_at);

visible_at is “not visible to the sweeper before this instant”: the delivery time, or the lease expiry when a lease is outstanding and reaches further. It is generated and stored rather than computed at query time, so the claim is a range scan per shard with zero rows filtered.

The price is paid at every claim and is stated rather than discovered: the claim’s UPDATE writes an indexed column, so it is never HOT. That is bought deliberately. The alternative, an index on (shard, deliver_at), keeps the claim HOT, and then every poisoned timer serving out its backoff sits at the head of the index and is re-read by every pass of every broker for the whole backoff. The cost would grow with the number of broken timers, which is the one direction that must not degrade. Timer rates are control-plane rates, orders of magnitude below message rates, so the write amplification is the cheaper side of that trade.

Also deliberately not created: an index on deliver_at, which has no reader because the sweeper orders by visible_at; one on (tenant_id, queue), because the leading columns of the primary key already serve peek and list; and one on claim_token, because the fire addresses rows by primary key.

The two lease columns

claim_token IS NOT NULL AND claimed_until > now() is the only definition of “in somebody’s hands”.

They are two columns rather than one because of what a single one would do to a broken timer. A row that is backing off after a failed fire has claimed_until in the future and claim_token set to NULL: it is not claimed, it is merely not due yet. Collapsing the two would make that row look claimed, and a claimed row cannot be cancelled, so a poisoned timer would be uncancellable for the whole of its backoff. The user would be unable to remove the broken thing, which is the one thing they certainly want to do.

The fire itself never waits in this space. It verifies its claimed rows with FOR UPDATE SKIP LOCKED, then provisions, pre-locks partitions in ascending id order and pushes, and deletes rows it already holds as its last step. A design note once prescribed a plain FOR UPDATE before that final delete; it is cancelled, and the header of 025_log_timers.sql says why in both of the ways it fails. The statement names an id column this table does not have, so the boot dies. And a FOR UPDATE without SKIP LOCKED would make the fire a waiter here, so one slow cancel inside a long transaction wire could stop the fire of the whole broker while holding the lease of every other timer in its batch.

Delete, not “mark done”, and the price of having no tombstone

The fire deletes. There is no status column and no completed-timer history, so the push and the disappearance of the staging row are the same commit, the fire is exactly-once with nothing to reconcile after a crash, and the table is bounded by pending work rather than by history. “Was it sent?” is answered by the log, by looking for the timer’s fixed txn in the destination queue.

The price is real and is declared instead of hidden: after the fire there is no tombstone, so a late cancel answers absent. That verdict means no longer pending. It never means “not delivered”, and a consumer of a compensation path has to check the state rather than infer it from a cancel result.

Numbers on this page that are not measured

The dated blocks in 001_log_schema.sql are the house formula: a storage setting carries the date it was measured, the symptom that produced it, and the number observed. The following carry no such block.

Each is to be replaced with a measured number after the first soak on a test rig, in the style of those dated blocks, and never with one taken from a live stack.