Skip to content

The memory state of a Queen node

Each node keeps queues, partitions, positions, KV, timers and recent dedup records in memory. Messages stay on disk, so memory grows with partitions and keys, not with retained messages.

Updated View as Markdown

Each node keeps its whole state in memory: queues, partitions, consumer positions, KV, timers, request ids, and the dedup records of recent pushes. It is loaded once, at boot.

Messages are not in memory. They stay in the queue log files and are read from there. So memory grows with the number of partitions and keys, and with the pushes of the dedup window, not with how many messages a queue keeps.

In memory, on every node: the state, which is queues, partitions, consumer positions, KV, timers, request ids and the dedup records of recent pushes, loaded once at boot. On disk: the queue log files, which hold every message and are the log itself; store/data.mdb, the LMDB checkpoint of the state, written four times a second and read at boot; and traces/data.mdb, read from disk. Messages are never loaded into memory: they are read from the queue log files.In memory, on every node: the state, which is queues, partitions, consumer positions, KV, timers, request ids and the dedup records of recent pushes, loaded once at boot. On disk: the queue log files, which hold every message and are the log itself; store/data.mdb, the LMDB checkpoint of the state, written four times a second and read at boot; and traces/data.mdb, read from disk. Messages are never loaded into memory: they are read from the queue log files.
What lives in memory, and what lives on disk.

Message traces are kept apart, in their own file, and are read from disk.

Edit this page
Navigation

Type to search…

↑↓ navigate↵ selectEsc close