Surviving a PostgreSQL outage

When PostgreSQL is unreachable, a push does not have to fail. The broker appends it to an on-disk spool, reports it to the client as buffered, and replays it once the database comes back. The mechanism is small, and its limits are as important as its guarantees.

What the spool is

An append-only directory of event files at FILE_BUFFER_DIR, default /var/lib/queen/buffers. Each record is a little-endian u32 length followed by a JSON body carrying the queue, the partition, the resolved tenant, the original transactionId, the authenticated producer subject if there was one, an encrypted flag, and the payload verbatim.

An active file is *.tmp; a finalized, drainable file is *.buf. Names embed a 13-digit epoch-millisecond stamp and an 8-digit sequence, so a lexical sort is FIFO order. A file rotates after FILE_BUFFER_EVENTS_PER_FILE events (10000 by default).

If the directory cannot be created, the broker logs a spool warning and falls back to a per-user temporary directory, chosen so that a developer machine does not silently drop every buffered push. In production, set FILE_BUFFER_DIR explicitly and confirm it in the config: file_buffer boot line.

How a push becomes buffered

There are exactly two entry points.

flowchart TB
P["POST /api/v1/push"] --> MM{"maintenance mode"}
MM -->|"off"| T{"push transaction"}
MM -->|"on"| SP{"append to the spool<br/>FILE_BUFFER_DIR"}
T -->|"committed"| Q["status: queued"]
T -->|"PostgreSQL unreachable"| SP
SP -->|"write ok"| BUF["status: buffered"]
SP -->|"write fails"| FA["status: failed, the item is lost"]
BUF --> DR["drain cycle every 100 ms,<br/>oldest .buf file first"]
DR --> RP["replay keeps the original transactionId"]
RP --> DUP["stored, or duplicate if it already committed"]

Both entry points land in the same append-only file, and the per-item status is the only place a caller can tell them apart. The drain at the bottom only makes progress while the database is reachable and maintenance mode is off.

The first is failure. A push whose database transaction failed (a connection error, a statement timeout, a pool acquisition failure) comes back from the write path with per-item status error, meaning nothing was committed. The broker counts those items as database errors, then writes each one to the spool. If the spool write succeeds the item is relabelled buffered; if the spool write itself fails it becomes failed, and that item is genuinely lost. Items that were in-request duplicates of a spooled item inherit its status. The response is still 201, so a client that only inspects the status code learns nothing: the per-item status field is the contract.

[
  {"messageId":"0198…","transactionId":"order-4471","queue":"orders","status":"buffered"},
  {"messageId":"0198…","transactionId":"order-4472","queue":"orders","status":"failed"}
]

The second is deliberate: maintenance mode, below.

Two things are not covered. POST /api/v1/transaction has no spool path at all: a transaction is all-or-nothing against PostgreSQL, and during an outage it fails. And nothing on the consume side is buffered: pops and acks against an unreachable database return errors, which is safe, because an unacked lease redelivers.

Watching it

Three in-process Prometheus gauges are always present, even when the database-backed part of the scrape is missing:

GET /api/v1/status/buffers returns the same three values as JSON plus a live database ping, and GET /api/v1/system/maintenance includes bufferedMessages and a bufferStats block with the finalized-file count and total bytes on disk. In the log, the spool target reports the enter and leave transitions once each, and the periodic sizes block carries spool_pending and spool_healthy.

Replay, and why it is idempotent

A background loop runs one drain cycle every FILE_BUFFER_FLUSH_MS (100 ms). Each cycle finalizes the active file, lists .buf files oldest-first, and replays up to 10 of them when the database looks healthy or 1 when it does not. A file is replayed in batches of FILE_BUFFER_MAX_BATCH events (100 by default), each batch grouped by (tenant, queue, partition) and written as one multi-segment transaction, so a huge spool file never becomes one giant transaction.

Idempotence comes from deduplication, not from bookkeeping. Replay preserves each event’s original transactionId and mints only fresh message ids, and it explicitly declines to vouch for anything the broker’s in-memory dedup cache might believe, forcing the SQL probe to check the full dedup window. A batch that a previous attempt already committed therefore comes back as status: "duplicate" and writes nothing.

Failure handling inside the drain

Drain errors are classified, because the two kinds need opposite responses.

Transient covers a connection-level failure with no SQLSTATE, plus 40001 (serialization failure), 40P01 (deadlock detected) and SQLSTATE classes 08, 53, 57 and 58. The file is left exactly where it is, the broker marks the database unhealthy, and the cycle stops so the head of the FIFO is retried next tick. After 10 consecutive transient failures a circuit breaker holds the loop off for 5 seconds. Good spool is never discarded during an outage.

Permanent is any other server-side SQL error (bad data, a constraint, a custom RAISE), plus a spool file that cannot be read. The same data will fail every retry, and one such file at the head of the FIFO would block replay of every newer message behind it. So the broker counts consecutive permanent failures of that specific head file and, after 5 attempts, moves it into <FILE_BUFFER_DIR>/failed/ and carries on with the rest of the cycle. Each attempt logs an ERROR on the spool target with attempt and max_attempts. If the rename fails the file is deleted, on the reasoning that data which can never be stored must not be allowed to freeze the queue behind it.

The quarantine directory is invisible to the drain (every listing filters on the .buf and .tmp extensions and does not recurse), so a quarantined file stays put until someone looks at it. Nothing else looks at it. Treat a non-empty failed/ directory as an alert: those messages were accepted and will never be stored.

Startup recovery

Before the listener binds, the broker finalizes any .tmp file left behind by a crash and replays every .buf file in FIFO order. The pass is bounded at one hour; on the cap it logs a warning and hands the remainder to the background drain. If a file fails, recovery marks the database unhealthy, stops, and leaves the rest to the drain loop as well.

The consequence for operations is that an instance with a large spool has a slow start, and that a spool is only ever recovered by a process that boots with the same FILE_BUFFER_DIR. See Probes and restarts for how that shapes a rolling restart.

Maintenance mode is the deliberate diversion

POST /api/v1/system/maintenance with {"enabled": true} diverts every push to the spool before it touches PostgreSQL, and pauses the drain so the spool accumulates on purpose. Every item comes back buffered with 201; if any spool write fails the response is 500. Disabling it force-finalizes the active file and resumes the drain, which then replays everything.

This is the tool for a planned database operation (a failover, a VACUUM FULL, a version upgrade) where you want producers to keep succeeding while writes are impossible. What it does not do:

• It does not pause consumers. Pop maintenance is a separate flag, POST /api/v1/system/maintenance/pop, and pops keep serving already-stored messages while push maintenance is on.
• It does not divert transactions. POST /api/v1/transaction still writes straight to PostgreSQL, so it will fail if the database is actually down.
• It is not a per-tenant control. Both flags are cell-wide.

The flag is mirrored into queen.system_state, so it survives a restart and is re-read at boot; it is also broadcast to peer instances. Leaving maintenance mode on by accident is a silent way to accumulate an unbounded spool, which is why queen_maintenance_mode_enabled is worth an alert.

The limits

The spool is node-local, unreplicated, and unencrypted beyond what the queue already asked for. Read those three properties as one sentence: losing the node loses its spool.

• No other instance can drain another instance’s directory. There is no transfer and no cross-node visibility. Behind a load balancer, buffered messages are scattered across whichever instances accepted them.
• An ephemeral container filesystem makes the whole mechanism decorative. Give FILE_BUFFER_DIR storage that outlives the container, and keep the pod on the node that holds it.
• For a queue with encryptionEnabled, the spooled payload is the same AES-256-GCM envelope that would have been stored, and the flag is persisted so the replayed frame is still marked encrypted. For every other queue the payload sits on disk as plaintext JSON. If a malformed QUEEN_ENCRYPTION_KEY disabled encryption at boot (which is a warning, not a boot failure), then the spool holds plaintext for flagged queues too.
• The spool bounds nothing. There is no size cap and no maximum age. During a long outage it grows until the filesystem fills, at which point spool writes fail and pushes are reported failed. Monitor free space on that volume alongside queen_file_buffer_pending.

Related pages