---
title: "Surviving a PostgreSQL outage"
description: "The node-local disk spool: how a push becomes buffered, how replay stays idempotent, what maintenance mode diverts, and the limits of a spool that is not replicated."
---

> Documentation Index
> Fetch the complete documentation index at: https://queenmq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Surviving a PostgreSQL outage

When PostgreSQL is unreachable, a push does not have to fail. The broker appends
it to an on-disk spool, reports it to the client as `buffered`, and replays it
once the database comes back. The mechanism is small, and its limits are as
important as its guarantees.

## What the spool is

An append-only directory of event files at `FILE_BUFFER_DIR`, default
`/var/lib/queen/buffers`. Each record is a little-endian `u32` length followed by
a JSON body carrying the queue, the partition, the resolved tenant, the original
`transactionId`, the authenticated producer subject if there was one, an
`encrypted` flag, and the payload verbatim.

An active file is `*.tmp`; a finalized, drainable file is `*.buf`. Names embed a
13-digit epoch-millisecond stamp and an 8-digit sequence, so a lexical sort is
FIFO order. A file rotates after `FILE_BUFFER_EVENTS_PER_FILE` events (10000 by
default).

If the directory cannot be created, the broker logs a `spool` warning and falls
back to a per-user temporary directory, chosen so that a developer machine does
not silently drop every buffered push. In production, set `FILE_BUFFER_DIR`
explicitly and confirm it in the `config: file_buffer` boot line.

> **Caution**
>
> There is **no per-event fsync**. Records are plain buffered appends under a
> mutex. A process crash or a host power loss can therefore lose the tail of the
> active file. The reader is built for that case (it stops at the first incomplete
> record instead of failing the file), but "stops cleanly" is not "loses nothing".
> The spool trades a small crash window for not blocking the push path on disk.

## How a push becomes buffered

There are exactly two entry points.

<pre class="mermaid">{`
flowchart TB
  P["POST /api/v1/push"] --> MM{"maintenance mode"}
  MM -->|"off"| T{"push transaction"}
  MM -->|"on"| SP{"append to the spool<br/>FILE_BUFFER_DIR"}
  T -->|"committed"| Q["status: queued"]
  T -->|"PostgreSQL unreachable"| SP
  SP -->|"write ok"| BUF["status: buffered"]
  SP -->|"write fails"| FA["status: failed, the item is lost"]
  BUF --> DR["drain cycle every 100 ms,<br/>oldest .buf file first"]
  DR --> RP["replay keeps the original transactionId"]
  RP --> DUP["stored, or duplicate if it already committed"]
`}</pre>

Both entry points land in the same append-only file, and the per-item status is
the only place a caller can tell them apart. The drain at the bottom only makes
progress while the database is reachable and maintenance mode is off.

The first is failure. A push whose database transaction failed (a connection
error, a statement timeout, a pool acquisition failure) comes back from the write
path with per-item status `error`, meaning nothing was committed. The broker
counts those items as database errors, then writes each one to the spool. If the
spool write succeeds the item is relabelled `buffered`; if the spool write itself
fails it becomes `failed`, and that item is genuinely lost. Items that were
in-request duplicates of a spooled item inherit its status. The response is still
`201`, so a client that only inspects the status code learns nothing: the
per-item `status` field is the contract.

```json
[
  {"messageId":"0198…","transactionId":"order-4471","queue":"orders","status":"buffered"},
  {"messageId":"0198…","transactionId":"order-4472","queue":"orders","status":"failed"}
]
```

The second is deliberate: maintenance mode, below.

Two things are **not** covered. `POST /api/v1/transaction` has no spool path at
all: a transaction is all-or-nothing against PostgreSQL, and during an outage it
fails. And nothing on the consume side is buffered: pops and acks against an
unreachable database return errors, which is safe, because an unacked lease
redelivers.

## Watching it

Three in-process Prometheus gauges are always present, even when the
database-backed part of the scrape is missing:

| Gauge | Meaning |
| --- | --- |
| `queen_file_buffer_db_healthy` | 0 while the broker believes PostgreSQL is unreachable |
| `queen_file_buffer_pending` | events written and not yet replayed |
| `queen_file_buffer_failed` | spool *write* failures: accepted-then-lost work |

`GET /api/v1/status/buffers` returns the same three values as JSON plus a live
database ping, and `GET /api/v1/system/maintenance` includes `bufferedMessages`
and a `bufferStats` block with the finalized-file count and total bytes on disk.
In the log, the `spool` target reports the enter and leave transitions once each,
and the periodic `sizes` block carries `spool_pending` and `spool_healthy`.

## Replay, and why it is idempotent

A background loop runs one drain cycle every `FILE_BUFFER_FLUSH_MS` (100 ms). Each
cycle finalizes the active file, lists `.buf` files oldest-first, and replays up
to 10 of them when the database looks healthy or 1 when it does not. A file is
replayed in batches of `FILE_BUFFER_MAX_BATCH` events (100 by default), each batch
grouped by (tenant, queue, partition) and written as one multi-segment
transaction, so a huge spool file never becomes one giant transaction.

Idempotence comes from deduplication, not from bookkeeping. Replay preserves each
event's original `transactionId` and mints only fresh message ids, and it
explicitly declines to vouch for anything the broker's in-memory dedup cache
might believe, forcing the SQL probe to check the full dedup window. A batch that
a previous attempt already committed therefore comes back as
`status: "duplicate"` and writes nothing.

> **Caution**
>
> That guarantee is exactly as wide as the deduplication window. On a queue
> configured with `dedupWindowSeconds: 0` deduplication is off and replay is **not**
> idempotent: a partially committed drain attempt duplicates. And because the hash
> sidecar is purged on a cutoff of `GREATEST(dedupWindowSeconds,
> completedRetentionSeconds, 900s)`, an outage longer than that window can outlive
> the evidence a replay needs. Long outages plus short windows equal duplicates.

## Failure handling inside the drain

Drain errors are classified, because the two kinds need opposite responses.

**Transient** covers a connection-level failure with no SQLSTATE, plus
`40001` (serialization failure), `40P01` (deadlock detected) and SQLSTATE classes
`08`, `53`, `57` and `58`. The file is left exactly where it is, the broker marks
the database unhealthy, and the cycle stops so the head of the FIFO is retried
next tick. After 10 consecutive transient failures a circuit breaker holds the
loop off for 5 seconds. Good spool is never discarded during an outage.

**Permanent** is any other server-side SQL error (bad data, a constraint, a
custom `RAISE`), plus a spool file that cannot be read. The same data will fail
every retry, and one such file at the head of the FIFO would block replay of every
newer message behind it. So the broker counts consecutive permanent failures of
that specific head file and, after 5 attempts, moves it into
`<FILE_BUFFER_DIR>/failed/` and carries on with the rest of the cycle. Each
attempt logs an `ERROR` on the `spool` target with `attempt` and `max_attempts`.
If the rename fails the file is deleted, on the reasoning that data which can
never be stored must not be allowed to freeze the queue behind it.

The quarantine directory is invisible to the drain (every listing filters on the
`.buf` and `.tmp` extensions and does not recurse), so a quarantined file stays
put until someone looks at it. **Nothing else looks at it.** Treat a non-empty
`failed/` directory as an alert: those messages were accepted and will never be
stored.

## Startup recovery

Before the listener binds, the broker finalizes any `.tmp` file left behind by a
crash and replays every `.buf` file in FIFO order. The pass is bounded at one
hour; on the cap it logs a warning and hands the remainder to the background
drain. If a file fails, recovery marks the database unhealthy, stops, and leaves
the rest to the drain loop as well.

The consequence for operations is that an instance with a large spool has a slow
start, and that a spool is only ever recovered by a process that boots with the
same `FILE_BUFFER_DIR`. See
[Probes and restarts](/selfhost/probes-restarts) for how that shapes a rolling
restart.

## Maintenance mode is the deliberate diversion

`POST /api/v1/system/maintenance` with `{"enabled": true}` diverts **every** push
to the spool before it touches PostgreSQL, and pauses the drain so the spool
accumulates on purpose. Every item comes back `buffered` with `201`; if any spool
write fails the response is `500`. Disabling it force-finalizes the active file
and resumes the drain, which then replays everything.

This is the tool for a planned database operation (a failover, a `VACUUM FULL`,
a version upgrade) where you want producers to keep succeeding while writes are
impossible. What it does not do:

- It does not pause consumers. Pop maintenance is a separate flag,
  `POST /api/v1/system/maintenance/pop`, and pops keep serving already-stored
  messages while push maintenance is on.
- It does not divert transactions. `POST /api/v1/transaction` still writes
  straight to PostgreSQL, so it will fail if the database is actually down.
- It is not a per-tenant control. Both flags are cell-wide.

The flag is mirrored into `queen.system_state`, so it **survives a restart** and
is re-read at boot; it is also broadcast to peer instances. Leaving maintenance
mode on by accident is a silent way to accumulate an unbounded spool, which is
why `queen_maintenance_mode_enabled` is worth an alert.

## The limits

The spool is node-local, unreplicated, and unencrypted beyond what the queue
already asked for. Read those three properties as one sentence: **losing the node
loses its spool.**

- No other instance can drain another instance's directory. There is no transfer
  and no cross-node visibility. Behind a load balancer, buffered messages are
  scattered across whichever instances accepted them.
- An ephemeral container filesystem makes the whole mechanism decorative. Give
  `FILE_BUFFER_DIR` storage that outlives the container, and keep the pod on the
  node that holds it.
- For a queue with `encryptionEnabled`, the spooled payload is the same
  AES-256-GCM envelope that would have been stored, and the flag is persisted so
  the replayed frame is still marked encrypted. For every other queue the payload
  sits on disk as plaintext JSON. If a malformed `QUEEN_ENCRYPTION_KEY` disabled
  encryption at boot (which is a warning, not a boot failure), then the spool
  holds plaintext for flagged queues too.
- The spool bounds nothing. There is no size cap and no maximum age. During a
  long outage it grows until the filesystem fills, at which point spool writes
  fail and pushes are reported `failed`. Monitor free space on that volume
  alongside `queen_file_buffer_pending`.

## Related pages

- [Observability](/selfhost/observability) — The spool gauges, the buffered-mode transitions, and what to alert on.
- [Backup and restore](/selfhost/backup) — What a pg_dump covers, and what it cannot cover.
- [Operating retention](/selfhost/retention) — What reclaims disk in PostgreSQL, and the two surfaces nothing reclaims.

Source: https://queenmq.com/selfhost/durability/index.mdx