---
title: "Operations"
description: "Running Queen day to day: the log blocks and metrics worth watching, turning retention on, dumping both schemas, and how a version change goes."
---

> Queen MQ documentation, for AI agents
> Complete self-contained summary of Queen MQ: https://queenmq.com/llms-brief.txt
> Fetch that first when the question is about the product rather than about this page.
> Index of all pages: https://queenmq.com/llms.txt

# Operations

The broker prints its rates and cache sizes on a timer, and exposes database-backed cluster totals
to Prometheus. Retention is off until you turn it on. A backup is a `pg_dump` of two schemas. A
version change is a cutover, not a migration.

## What to watch

Nothing is logged per message. One task emits two aggregate blocks every `QUEEN_LOG_RATES_MS`,
default 10 seconds: `rates` carries `push_s`, `pop_s`, `ack_s`, latency percentiles, the number of
parked long-polls and `pool_waiting`, and `sizes` carries cache occupancy, `spool_pending` and
`rss_gb`. A handful of per-queue lines follow, ranked by traffic. `QUEEN_LOG_JSON=1` turns each
line into one JSON object for a shipper.

Two lines fire on transition rather than on the timer, and they are the ones to page on:

```text
WARN spool  entering buffered mode (degraded durability)
INFO spool  DB recovered
```

Buffered means pushes are landing on the broker's local disk instead of PostgreSQL. Alert on the
same condition: `queen_file_buffer_db_healthy == 0`, `queen_file_buffer_pending` not returning to
zero once the database recovers, and `queen_file_buffer_failed > 0`, which counts accepted-then-lost
pushes and should be zero forever.

The scrape target is `GET /metrics/prometheus`. `GET /metrics` is a small JSON object with several
fields wired to zero: read it by hand, do not build a dashboard on it. Per-queue series sum across
tenants and the endpoint sits outside broker authentication, so it is an operator surface, never a
tenant one. The full family list is in [Prometheus](/reference/prometheus).

Do not alert on `/health` answering 503 as "the broker is down": it is a database check, and
[the probe wiring](/deploy/kubernetes#the-probes) is where that matters.

## Maintenance mode does not hold the timer fire

> **Danger**
>
> While push maintenance mode is on, pushes are diverted to the broker's disk spool, and **the timer
> fire is not on that path**. The fire is a `DELETE` of the staging row and a push of the frame in the
> *same* transaction, and that shared commit is the whole of its exactly-once guarantee; putting the
> push half on the spool would break it. So the fire is not spooled, and it is not held back either:
> the sweeper reads only `timers_fire_enabled`, never the maintenance flag, and no stored procedure
> reads that flag on its behalf.
>
> A maintenance window therefore does not make the log static. Ordinary pushes wait on disk and land
> when the window closes, while every timer that comes due inside it commits into `queen.log_segments`
> throughout, which also reverses their order: a push accepted early in the window is appended after a
> timer that fired later in it. If the window is open because the database is going away, the fires
> fail transiently instead, spend no delivery attempt, and drain oldest first when it returns.
>
> To hold delivery for the length of a window, pause it deliberately with the `timersFire` kill switch
> and turn it back on when the window closes. Nothing is lost while it is off, but promised messages
> are not being delivered, so it is a lever to leave a clock on.

The sweeper is also the component an operator kill switch acts on, separately from maintenance mode
and of exactly the same class: those switches pause something that is running, they do not turn a
feature on. [KV and timers](/deploy/state) has the three rungs of refusal, what each one answers, and
the order in which the broker sheds work before anybody has to pull a lever.

## Retention

A queue keeps everything forever, and consuming deletes nothing. Turning retention on is one call:

```bash
curl -s -X POST http://localhost:6632/api/v1/configure \
  -H 'Content-Type: application/json' \
  -d '{"queue":"orders","options":{"retentionEnabled":true,"retentionSeconds":604800,"completedRetentionSeconds":86400}}'
```

`retentionSeconds` deletes on age alone, read or not. `completedRetentionSeconds` never passes the
slowest group's cursor, so one abandoned group pins the history for every other. Deletion is
whole-segment, so a window is a floor on what is kept, not a ceiling. `/configure` is a full
replace: a later call that omits these keys turns retention back off, so send the whole set from
[queue options](/reference/queue-options).

The sweep runs every `RETENTION_INTERVAL` ms, default 5000, on the one replica per cycle that wins
the `queen.maintenance_leases` claim, and logs only when it actually deleted something:

```text
INFO retention  swept queues=12 due=27 backed_off=15 segments_deleted=340 txns_purged=340 max_wait_evicted=0 partitions_deleted=0 metrics_purge=… elapsed_ms=47
```

Silence on a growing deployment means nothing is being deleted. `due` and `backed_off` explain a
delete rate that looks too slow: `due` pinned at its cap every cycle means the cycle is capped
rather than idle, and a large `backed_off` means the due partitions are held back by a group's
cursor rather than deletable. Two things the sweep never
touches: dead letters, which leave one at a time through
`DELETE /api/v1/messages/:partitionId/:transactionId`, and streaming state in
`queen_streams.state`. The loop step by step is in [retention internals](/internals/retention).

## Backup and restore

Everything the broker persists lives in two schemas, plus the spool on local disk that no database
backup captures. Dump both schemas: omitting `queen_streams` loses every registered streaming query
and its per-key state, silently, because the broker boots and recreates an empty one.

```bash
pg_dump -Fc -n queen -n queen_streams -f queen-2026-08-15.dump "postgresql://user@host:5432/queen"
```

A dump can be taken against a running broker, since `pg_dump` reads one snapshot. Restore into an
empty database with every instance stopped, including the ones behind the load balancer.

```bash
createdb -h host -U postgres queen_restored
pg_restore --no-owner --no-privileges -d "postgresql://postgres@host:5432/queen_restored" queen-2026-08-15.dump
```

`--no-owner --no-privileges` because the broker re-issues its own grants at boot, which is also
why a schema-only dump is redundant. Restore into the version that produced the dump.

`queen.kv`, `queen.log_timers` and the two quota tables live in the `queen` schema, so the command
above already carries them. One of them is not data but configuration: `queen.kv_quota` holds the
per-tenant grants and limits an operator wrote by hand, and it is the one table in this feature that a
rollback must never drop. A restore also brings back timers that were pending at the dump instant,
which then fire again, so the same idempotency the rest of a restore demands applies to their
consumers.

A restore rolls consumption back to the dump instant, so every message acked since redelivers: an
hour-old dump means an hour of reprocessing, and it is a recovery only if handlers are idempotent.
Leases are columns on a row rather than sessions, so restored ones expire and re-lease on their
own. One thing to check by hand: the maintenance flags live in `queen.system_state`, and a dump
taken during a maintenance window restores with push maintenance on, sending every push straight
to the spool. Read `GET /api/v1/system/maintenance` immediately after.

## Upgrades

Same-version restarts and replacements need no coordination. Start the replacement, wait for a 200
from `/health`, take the old instance out of the load balancer, SIGTERM, repeat. Shutdown drains
in-flight requests, and a leased batch nobody acked redelivers once its lease expires.

A version change is different, because the boot-time apply is not a migration engine: every boot
re-applies the same idempotent corpus a fresh database gets, and there is no upgrade path, so
pointing a new build at an old database is unsupported.

1. **Stop producing** to the old deployment.

2. **Let consumers drain it** to empty.

3. **Start the new version against a fresh database.** It creates its own schema at first boot.

4. **Repoint the clients**, and re-apply your `/configure` options: they lived in the old database.

Rollback is the same move reversed, so keep the old database until you trust the new one. If you
cannot drain, copy at the application level, reading the backlog through the old API and pushing
it with a deterministic `transactionId` per message so a retried copy loop cannot duplicate.

Watch the spool, turn retention on before the disk decides for you, dump both schemas, and cut
over rather than migrate.

Source: https://queenmq.com/deploy/operations/index.mdx