---
title: "Probes and restarts"
description: "What /health really tests, why it is the wrong liveness probe, how SIGTERM drains, and what happens to leased batches when an instance goes away."
---

> Documentation Index
> Fetch the complete documentation index at: https://queenmq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Probes and restarts

The broker is one stateless process. Every piece of durable state is in
PostgreSQL, with one exception: the on-disk push spool, which is node-local.
That single exception is what makes probe configuration matter more than it
usually does.

## What /health tests

`GET /health` takes a connection from the pool and runs a trivial query against
PostgreSQL. It returns 200 with

```json
{"status":"healthy","database":"connected","engine":"segments-rust","version":"1.0.0"}
```

and 503 with `"status":"unhealthy","database":"disconnected"` when either the pool
cannot hand out a connection or the query fails.

So `/health` is a **dependency** check, not a process check. It answers "can this
instance reach the database right now", which is exactly the question a readiness
probe asks and exactly the wrong question for a liveness probe.

The endpoint resolves to access level `public` in the route table, and the
authentication middleware short-circuits on `public` before it looks for a token.
Turning on `JWT_ENABLED` does not close it.

## The liveness-probe trap

> **Danger**
>
> Wiring `/health` to a Kubernetes liveness probe defeats the disk spool it exists
> to complement. During a PostgreSQL outage the broker is *working as designed*: it
> returns 503 on `/health` while it accepts pushes and writes them to the spool.
> A liveness probe reads that 503 as a dead container and kills the pod, in a loop,
> for the whole outage.

The damage is concrete, and it compounds:

- Each kill discards in-flight requests, so pushes that would have been spooled
  are refused at the socket instead.
- Nothing drains while the container is restarting, so the spool only grows.
- If the spool directory is on the container filesystem rather than a volume that
  survives the restart, every kill destroys the buffered messages outright. This
  is the case that turns a survivable outage into data loss.

Use `/health` for **readiness**: an instance that cannot reach PostgreSQL should
leave the load balancer, because it can serve nothing but spooled pushes.

For **liveness**, probe the process, not its dependency. A TCP connect on `PORT`
is enough, and it is the honest test: the listener exists only while the process
is alive, and the process aborts on a panic instead of limping. `GET /status`
answers `{"status":"ok","engine":"segments-rust"}` with no database access at
all, but it resolves to read-only access rather than public, so with
`JWT_ENABLED=true` it needs a token, which makes it awkward as a probe target.

## SIGTERM drains, background loops do not

The HTTP server is wrapped in a graceful shutdown driven by SIGTERM or Ctrl-C.
On the signal the broker logs

```text
INFO shutdown  signal received, draining in-flight requests
```

stops accepting new connections, and waits for in-flight requests to finish.
Then it reports the spool depth and exits:

```text
WARN  shutdown  spool has undrained events at shutdown pending=<n>
INFO  shutdown  shutdown complete
```

Two consequences follow from *what* is being drained.

First, a parked long-poll pop is an in-flight request. A consumer waiting on an
empty queue holds its request open until its own deadline
(`POP_DEFAULT_TIMEOUT_MS` is 30000 ms by default, and `DEFAULT_TIMEOUT` takes
precedence over it), so shutdown can take that long even on an idle broker. Size
the orchestrator's termination grace period above your largest pop timeout, or
the orchestrator will SIGKILL a process that was shutting down correctly. Parked
pops release their pooled database connection *before* parking, so they cost a
connection slot only while actually querying.

Second, the background loops (retention, the stats reconciler, the metrics
collector, the spool drain) are detached tasks. They are not drained; they stop
with the runtime. In particular **the spool is not flushed on the way out**. The
`pending=<n>` warning is telling you that those events are still on disk and will
only be replayed by a process that starts with the same `FILE_BUFFER_DIR`. If you
are draining a node permanently, that warning is the signal to stop and get the
spool drained before the volume goes away.

## What happens to in-flight leases

Nothing is handed back. This is the single most important thing to understand
about restarting a broker.

Consumption state is one cursor per (partition, consumer group):
`queen.log_consumers.committed`. A pop takes a lease on the span
`(committed, batch_end]` by writing `worker_id`, `batch_end`,
`lease_acquired_at` and `lease_expires_at` onto that row. The lease is a row in
PostgreSQL, not a session, so it does not notice that the broker holding it
disappeared, and there is no release-on-disconnect path anywhere in the engine.

The sequence after a restart is therefore:

1. The instance dies. The consumer's HTTP request fails; the leased batch was
   never acked, so `committed` is unchanged.

2. Until `lease_expires_at` passes, every pop for that (partition, group) is
   filtered out by the lease predicate. The batch is invisible, and the partition
   looks idle to that group. Lag grows.

3. Once the lease expires, the next pop re-leases from the same `committed`, so
   the whole span is redelivered, to whichever consumer asks first, on any
   instance.

The practical consequence is that the *lease time* is your restart recovery
window for already-leased work: a queue with a 300-second lease can stall for up
to 300 seconds on the partitions that had leases open. A short lease redelivers
faster; too short a lease redelivers work that is still being processed. That
trade-off, not the restart itself, is what to tune.

Redelivery after a lease expiry does not consume the retry budget. The engine
tracks two different counters on the consumer row: `attempt_count`, which
increments when a batch is re-leased from the same start offset (redelivery
telemetry), and `batch_retry_count`, which is the retry budget and is charged
**only** by an explicit `failed` ack. A restart therefore cannot push a message
toward the dead-letter queue, no matter how many times it happens.

`autoAck=true` pops behave differently, because they commit the cursor inside the
pop transaction and take no lease. A restart cannot redeliver them: anything
delivered and not yet processed by the client is gone. That is the
at-most-once trade you accepted by asking for it.

## Rolling restarts

Every boot re-applies the embedded schema (`schema.sql` plus every stored
procedure file, in lexical order) under a session advisory lock, so concurrent
boots serialize instead of racing. Every statement is idempotent, so re-applying
is a no-op on an already-current database. A DDL error fails the boot: the process
logs `FATAL: schema apply failed` and exits 1 rather than serving against a
half-applied schema.

That makes a rolling restart safe by construction, with three things to get right:

- **Drain before terminating.** Remove the instance from the load balancer first
  (readiness), then SIGTERM, then wait for the grace period discussed above.
  A SIGKILL mid-request is survivable (an unacked batch redelivers), but it
  strands the spool.
- **Restart one instance at a time when the spool may be non-empty.** Startup
  recovery replays the spool *before* the listener binds, oldest file first, with
  a one-hour cap; anything left over is handed to the background drain loop. An
  instance therefore has a slow start proportional to its spool, and taking two
  down together makes that worse.
- **Keep `FILE_BUFFER_DIR` on storage that outlives the container.** A
  node-local spool is only a durability mechanism if the next process on that
  node can see it.

If you pre-apply the DDL with a privileged role and run the broker as a
low-privilege user, set `QUEEN_APPLY_SCHEMA=0`. The boot then logs
`apply skipped` and touches no catalog, which also means the schema in the
database is entirely yours: the definitions serving requests are the ones you
applied, not the ones the binary carries.

## Related pages

- [Surviving a PostgreSQL outage](/selfhost/durability) — The spool, how a push becomes buffered, and why replay is idempotent.
- [Observability](/selfhost/observability) — The shutdown, spool and boot log lines this page refers to.
- [Reference](/reference) — Route access levels, timeouts and every environment variable.

Source: https://queenmq.com/selfhost/probes-restarts/index.mdx