Probes and restarts

The broker is one stateless process. Every piece of durable state is in PostgreSQL, with one exception: the on-disk push spool, which is node-local. That single exception is what makes probe configuration matter more than it usually does.

What /health tests

GET /health takes a connection from the pool and runs a trivial query against PostgreSQL. It returns 200 with

{"status":"healthy","database":"connected","engine":"segments-rust","version":"1.0.0"}

and 503 with "status":"unhealthy","database":"disconnected" when either the pool cannot hand out a connection or the query fails.

So /health is a dependency check, not a process check. It answers “can this instance reach the database right now”, which is exactly the question a readiness probe asks and exactly the wrong question for a liveness probe.

The endpoint resolves to access level public in the route table, and the authentication middleware short-circuits on public before it looks for a token. Turning on JWT_ENABLED does not close it.

The liveness-probe trap

The damage is concrete, and it compounds:

• Each kill discards in-flight requests, so pushes that would have been spooled are refused at the socket instead.
• Nothing drains while the container is restarting, so the spool only grows.
• If the spool directory is on the container filesystem rather than a volume that survives the restart, every kill destroys the buffered messages outright. This is the case that turns a survivable outage into data loss.

Use /health for readiness: an instance that cannot reach PostgreSQL should leave the load balancer, because it can serve nothing but spooled pushes.

For liveness, probe the process, not its dependency. A TCP connect on PORT is enough, and it is the honest test: the listener exists only while the process is alive, and the process aborts on a panic instead of limping. GET /status answers {"status":"ok","engine":"segments-rust"} with no database access at all, but it resolves to read-only access rather than public, so with JWT_ENABLED=true it needs a token, which makes it awkward as a probe target.

SIGTERM drains, background loops do not

The HTTP server is wrapped in a graceful shutdown driven by SIGTERM or Ctrl-C. On the signal the broker logs

INFO shutdown  signal received, draining in-flight requests

stops accepting new connections, and waits for in-flight requests to finish. Then it reports the spool depth and exits:

WARN  shutdown  spool has undrained events at shutdown pending=<n>
INFO  shutdown  shutdown complete

Two consequences follow from what is being drained.

First, a parked long-poll pop is an in-flight request. A consumer waiting on an empty queue holds its request open until its own deadline (POP_DEFAULT_TIMEOUT_MS is 30000 ms by default, and DEFAULT_TIMEOUT takes precedence over it), so shutdown can take that long even on an idle broker. Size the orchestrator’s termination grace period above your largest pop timeout, or the orchestrator will SIGKILL a process that was shutting down correctly. Parked pops release their pooled database connection before parking, so they cost a connection slot only while actually querying.

Second, the background loops (retention, the stats reconciler, the metrics collector, the spool drain) are detached tasks. They are not drained; they stop with the runtime. In particular the spool is not flushed on the way out. The pending=<n> warning is telling you that those events are still on disk and will only be replayed by a process that starts with the same FILE_BUFFER_DIR. If you are draining a node permanently, that warning is the signal to stop and get the spool drained before the volume goes away.

What happens to in-flight leases

Nothing is handed back. This is the single most important thing to understand about restarting a broker.

Consumption state is one cursor per (partition, consumer group): queen.log_consumers.committed. A pop takes a lease on the span (committed, batch_end] by writing worker_id, batch_end, lease_acquired_at and lease_expires_at onto that row. The lease is a row in PostgreSQL, not a session, so it does not notice that the broker holding it disappeared, and there is no release-on-disconnect path anywhere in the engine.

The sequence after a restart is therefore:

The practical consequence is that the lease time is your restart recovery window for already-leased work: a queue with a 300-second lease can stall for up to 300 seconds on the partitions that had leases open. A short lease redelivers faster; too short a lease redelivers work that is still being processed. That trade-off, not the restart itself, is what to tune.

Redelivery after a lease expiry does not consume the retry budget. The engine tracks two different counters on the consumer row: attempt_count, which increments when a batch is re-leased from the same start offset (redelivery telemetry), and batch_retry_count, which is the retry budget and is charged only by an explicit failed ack. A restart therefore cannot push a message toward the dead-letter queue, no matter how many times it happens.

autoAck=true pops behave differently, because they commit the cursor inside the pop transaction and take no lease. A restart cannot redeliver them: anything delivered and not yet processed by the client is gone. That is the at-most-once trade you accepted by asking for it.

Rolling restarts

Every boot re-applies the embedded schema (schema.sql plus every stored procedure file, in lexical order) under a session advisory lock, so concurrent boots serialize instead of racing. Every statement is idempotent, so re-applying is a no-op on an already-current database. A DDL error fails the boot: the process logs FATAL: schema apply failed and exits 1 rather than serving against a half-applied schema.

That makes a rolling restart safe by construction, with three things to get right:

• Drain before terminating. Remove the instance from the load balancer first (readiness), then SIGTERM, then wait for the grace period discussed above. A SIGKILL mid-request is survivable (an unacked batch redelivers), but it strands the spool.
• Restart one instance at a time when the spool may be non-empty. Startup recovery replays the spool before the listener binds, oldest file first, with a one-hour cap; anything left over is handed to the background drain loop. An instance therefore has a slow start proportional to its spool, and taking two down together makes that worse.
• Keep FILE_BUFFER_DIR on storage that outlives the container. A node-local spool is only a durability mechanism if the next process on that node can see it.

If you pre-apply the DDL with a privileged role and run the broker as a low-privilege user, set QUEEN_APPLY_SCHEMA=0. The boot then logs apply skipped and touches no catalog, which also means the schema in the database is entirely yours: the definitions serving requests are the ones you applied, not the ones the binary carries.

Related pages