Skip to content

The write path: admission, batch, plan, apply

A write goes through four steps: a memory budget at admission, a batch of up to 4,096 commands, a planner that decides ids and offsets once, and a deterministic apply on every node.

Updated View as Markdown

A write goes through four steps.

The four steps of a write. Admission, on every node: a memory budget of 128 MiB; a push takes its share before its body is read, waits when the budget is full and gets 429 after about 15 seconds. Batch, on the leader: up to 4,096 commands or 4 MiB per entry, up to 8 entries in flight. Plan: eight lanes plan in parallel and decide ids, offsets and times once. Raft: the entry is committed when a majority has it on disk. Apply, on every node: in order, on four threads, with no clock and no random number, so every node ends with the same bytes.The four steps of a write. Admission, on every node: a memory budget of 128 MiB; a push takes its share before its body is read, waits when the budget is full and gets 429 after about 15 seconds. Batch, on the leader: up to 4,096 commands or 4 MiB per entry, up to 8 entries in flight. Plan: eight lanes plan in parallel and decide ids, offsets and times once. Raft: the entry is committed when a majority has it on disk. Apply, on every node: in order, on four threads, with no clock and no random number, so every node ends with the same bytes.
The four steps of a write.

Admission. Every node has a memory budget for the writes waiting to be planned (128 MiB by default). A push takes its share before its body is read. When the budget is full, the push waits, so a fast producer just slows down. After about 15 seconds it gets a 429, and the client tries again. Pops and acks never wait here. We added this after a broker filled 10 GB of memory with waiting requests and was killed.

Batch. The leader collects up to 4,096 commands (or 4 MiB) and plans them together, as one log entry. Up to eight entries are in flight at the same time. A retried request is found by its id and gets the answer it missed, so nothing is planned twice.

Plan. The planner decides what each command means right now, and writes the answer down as facts: “append these messages to partition 17, at offsets 4,210 to 4,309, at this microsecond”. Ids, offsets and times are chosen here, once. To use more cores, eight lanes plan in parallel: each lane owns a share of the partitions, and one last step plans what touches more than one lane.

One planning cycle on the leader. The commands of the batch are sorted by partition into eight lanes: a lane owns the partitions whose id modulo 8 is its number and plans the commands that touch only its own partitions, all eight at the same time. Then one control step plans everything else with the full view: new partitions, KV, timers, queue changes and commands that cross lanes. The result is still one entry, applied as if one thread had planned it.One planning cycle on the leader. The commands of the batch are sorted by partition into eight lanes: a lane owns the partitions whose id modulo 8 is its number and plans the commands that touch only its own partitions, all eight at the same time. Then one control step plans everything else with the full view: new partitions, KV, timers, queue changes and commands that cross lanes. The result is still one entry, applied as if one thread had planned it.
Eight lanes plan in parallel; one step plans what crosses lanes.

Apply. Every node executes the committed entries, in order. Apply reads no clock and no random number, so every node ends with the same bytes. Four threads apply the work of different partitions at the same time.

Edit this page
Navigation

Type to search…

↑↓ navigate↵ selectEsc close