System and internal routes

Everything on this page is an operator surface, not part of the client API. No SDK calls these routes, and a tenant on a shared cell must never reach them.

route_access_level maps every path starting with /api/v1/system/ or /internal/ to admin, plus /api/v1/stats/refresh explicitly. That mapping only protects anything when JWT is on.

None of these routes are tenant-scoped, with one exception: POST /internal/api/notify resolves a tenant, because the wake gate it triggers is keyed by (tenant, queue). Maintenance mode is a property of the process and the cell, not of a tenant.

Maintenance mode

Two independent flags, each with a GET and a POST. Both are stored twice: as an in-process AtomicBool (the source of truth for hot-path checks) and as a row in queen.system_state keyed maintenance_mode / pop_maintenance_mode with the value {"enabled": bool} (a best-effort mirror for restart and cluster propagation).

GET /api/v1/system/maintenance

Reads both flags fresh from queen.system_state, so a change made on another replica is reflected immediately, and updates the local atomics to match. If the pool or the query fails it falls back to the in-process atomics rather than returning an error: this route never 500s.

{
  "maintenanceMode": false,
  "popMaintenanceMode": false,
  "bufferedMessages": 0,
  "bufferHealthy": true,
  "bufferStats": {
    "pendingCount": 0,
    "failedCount": 0,
    "dbHealthy": true,
    "failedFiles": { "count": 0, "totalBytes": 0, "totalMB": 0.0, "failoverCount": 0, "qos0Count": 0 }
  }
}

bufferStats.failedFiles is read off the spool directory: count and totalBytes are the finalized .buf files still waiting to drain. failoverCount mirrors count; qos0Count is always 0.

POST /api/v1/system/maintenance

Body: enabled (boolean, required). A missing field is a 400 (enabled (boolean) is required).

{ "enabled": true }

What enabling it does, in order: sets the in-process flag, mirrors it to queen.system_state, pauses the background spool drain so buffered pushes accumulate, and broadcasts a MAINTENANCE_MODE_SET frame to mesh peers. Disabling it force-finalizes the active spool file and resumes the drain, so the spool replays into PostgreSQL.

{
  "maintenanceMode": true,
  "bufferedMessages": 0,
  "bufferHealthy": true,
  "message": "Maintenance mode ENABLED. All PUSHes routing to file buffer."
}

The message strings are exact and stable (Maintenance mode DISABLED. Background processor will drain buffer to DB. on disable) because tooling greps for them.

Effect on pushes. While the flag is on, POST /api/v1/push never touches PostgreSQL. Every item is written to the disk spool and reported with status: "buffered" at HTTP 201; an item whose spool write failed reports status: "failed" and the whole response becomes 500. A push that omitted transactionId gets one minted at buffer time, so the buffered result and the replay dedup key are both well defined.

Effect on a dead-letter replay. The two replay routes are refused instead, with 503 and result: "maintenance". A replay is a move, and a move cannot be spooled: the spool carries frames, not the deletion of the dead-letter row, so a spooled one would land in exactly the pushed-but-still-dead-lettered state the move primitive exists to make impossible. Nothing is lost by refusing, because the dead-letter row stays where it is and the same replay works once the switch is off.

GET /api/v1/system/maintenance/pop

{
  "popMaintenanceMode": false,
  "message": "Pop maintenance mode is OFF. Normal operation."
}

Unlike the push-maintenance GET, this one reads only the in-process flag. On a multi-replica deployment it therefore answers for the replica you happened to reach; GET /api/v1/system/maintenance is the route that re-reads the database.

POST /api/v1/system/maintenance/pop

Body: enabled (boolean, required).

{
  "popMaintenanceMode": true,
  "message": "Pop maintenance mode ENABLED. All POP operations will return empty arrays."
}

Effect on pops. Every pop route returns HTTP 204 immediately, before touching the database. The 204 carries no body at all (not even the paused flag the handler constructs), because announcing a content length on a body that hyper then elides poisoned strict HTTP/1.1 clients under empty-poll load. Clients treat 204 as “no messages” and retry, so a paused broker looks like an idle one to a consumer. That is deliberate, and it is why the flag has an explicit GET.

The flip is mirrored to queen.system_state and broadcast to mesh peers as POP_MAINTENANCE_MODE_SET.

The kv and timer kill switches

Access level admin, like every route under /api/v1/system/.

KV and timers have no boot flag. There is no variable that decides whether those routes exist, for the same reason there is none for push and for pop, so nothing here is a way to opt into a feature. What these three switches are is the other instrument entirely: “stop the bleeding, now”, for an incident at three in the morning that a rollout does not resolve. They are on the same pattern as maintenance mode above, an in-process atomic that is authoritative on the hot path, mirrored into queen.system_state for propagation and restart.

That is why a paused surface answers 503 with Retry-After and never 404: an operator pulled a lever, it is temporary, and the client should come back. A 404 on a KV or timer path means the path is wrong. Inside the transaction wire the same pause answers 403, because a bundle that retries a paused surface forever is a client spinning on the hot path with messages in hand.

GET /api/v1/system/kv-timers

{
  "kvEnabled": true,
  "timersScheduleEnabled": true,
  "timersFireEnabled": true,
  "quotaTenants": 12,
  "quotaAgeMs": 8412,
  "quotaHot": 1
}

The three …Enabled fields are the switches, and they are what POST writes. There is no boot half to report beside them any more. The GET re-reads the mirror rows before answering and falls back to the in-process atomics, so it never fails on a busy pool.

quotaAgeMs is the number to look at when limits behave oddly. It is how stale the measurement the quota gate is enforcing against has become, and a value that keeps growing means the refresh is failing. Enforcement still works, because each broker adds its own local delta to whatever it last measured, but nothing is being released: a tenant that has come back under its limit stays blocked. quotaHot counts the tenants above the hot threshold, whose measurement is refreshed on the faster cadence.

POST /api/v1/system/kv-timers

{ "timersSchedule": false }

Each field is independent and an omitted one is left alone. A body that reset the unmentioned ones would turn “pause new timers” into “and also stop delivering the ones already promised”. A body with none of the three is a 400.

An absent mirror row means on. That is the difference between a kill switch and a feature flag, and getting it backwards would mean every fresh cell booting with the feature dead and no row to explain why.

Every flip of any of the three writes a single sweeper log line at the instant it happens, naming the switch and what the operator has just done to the cell. It is on-change only, so a switch that has been off for an hour does not write an hour of lines.

The ephemeral kill switch

Same instrument, one switch, for the ephemeral route family. It is on the same pattern as the three above: an in-process atomic mirrored into queen.system_state, an absent row meaning on, and a paused surface answering 503 with Retry-After rather than 404.

GET /api/v1/system/ephemeral

{
  "ephemeralEnabled": true,
  "cellBytes": 8402944,
  "cellMaxBytes": 268435456,
  "queues": 314,
  "grantedTenants": 12,
  "requireGrant": true,
  "epoch": "9f3a1c"
}

Every number here is an in-process gauge: this route reads no table and counts nothing, which is what makes it safe to poll during the incident it exists for. cellBytes against cellMaxBytes is the pair a 503 ephemeral_unavailable is decided on. grantedTenants at 0 with requireGrant true means every tenant is being refused, which is the single most confusing state to diagnose from the outside. epoch is this broker’s incarnation id, the one carried inside every ephemeral message id, so a support ticket about an ack that answered stale can be resolved against it.

POST /api/v1/system/ephemeral

{ "enabled": false }

With enabled: false every /api/v1/ephemeral route answers 503. Consumers already parked stay parked until their own timeout expires, the rings keep their memory until the idle collector reaches them, and nothing stored durably is affected, because nothing of this class is stored durably except the declarations. Turning it back on resumes service on rings that are, by then, whatever the age and bound limits have left in them.

GET /api/v1/system/shared-state

Returns a hard-coded single-node summary:

{
  "enabled": false,
  "reason": "single_node_segments_broker",
  "maintenance_mode": false,
  "pop_maintenance_mode": false
}

POST /api/v1/stats/refresh

Access level admin. Runs queen.log_refresh_all_stats_v1() (the same reconciler the broker’s own loop runs every STATS_INTERVAL_MS, scheduled through the stats_refresh claim row in queen.maintenance_leases) and returns its summary JSON verbatim, labelled engine: "segments" with a segPartitions count. The route does not touch the schedule: forcing a refresh never delays or doubles the background cadence.

Use it to force a refresh instead of waiting out the interval. If the dashboard’s numbers are stale rather than merely late, check whether that loop is running at all (a large statsAge on the status routes). It refreshes the counters only: retainedBytes is owned by the retained-bytes lane (RETAINED_BYTES_INTERVAL_MS, advisory lock 737003) and keeps its last value here.

Internal broker-to-broker routes

Three routes exist for the mesh layer. They are admin-gated and are not a public contract: treat their shapes as version-coupled to the broker.

POST /internal/api/notify

Body: queue (required, non-empty), partition (optional). A missing queue is a 400.

{ "queue": "orders", "partition": "eu-1" }

Emits exactly the signal a local push emits: wakes this replica’s parked long-poll pops for that (tenant, queue) and fans the signal out to mesh peers. Returns {"status": "ok"}. This is the HTTP fallback for a peer that has no mesh reachability, and the hook for an external system that wrote to the database out of band and wants parked consumers woken now instead of at the next poll interval.

The tenant comes from the same x-queen-tenant header every other route uses; absent, it is the default tenant. The header is unauthenticated by design, and the trust boundary for it is the proxy, which is another reason this path must not be reachable from outside the cell.

GET /internal/api/shared-state/stats

GET /internal/api/inter-instance/stats

Identical bodies; the second is a legacy alias. With a mesh transport running:

{
  "server_id": "queen-0",
  "transport": "tcp-mesh",
  "port": 6633,
  "peer_count": 1,
  "peers": [
    { "host": "queen-1", "port": 6633, "connected": true, "resolved": true, "resolved_ip": "10.0.0.12" }
  ],
  "servers_alive": 1,
  "servers_dead": 0,
  "eph_members": 1,
  "messages_sent": 41022,
  "messages_received": 40911,
  "messages_dropped": 0,
  "signature_failures": 0,
  "handshake_failures": 0,
  "sequence_rejections": 0,
  "enabled": true,
  "running": true,
  "maintenance_mode": false,
  "pop_maintenance_mode": false
}

With no peers configured, or no transport:

{
  "enabled": false,
  "reason": "no_peers",
  "maintenance_mode": false,
  "pop_maintenance_mode": false
}

signature_failures and handshake_failures are the same counter under two names. sequence_rejections is always 0: there is no sequence tracking. eph_members counts the live peers that advertised the v2 handshake and therefore participate in the ephemeral rendezvous ring; during a rolling deploy it sits below servers_alive, which is the expected reading, not an error.

Firewall the mesh port

The mesh is framed TCP on QUEEN_MESH_PORT (default 6633), dialled from QUEEN_MESH_PEERS (the older QUEEN_UDP_* variable names survive as aliases, which is why the default port comes from QUEEN_UDP_NOTIFY_PORT when the new name is unset). The listener binds QUEEN_MESH_BIND_ADDR, defaulting to the HTTP listener’s QUEEN_BIND_ADDR. Its HELLO handshake is HMAC’d, but the nonce is generated by the dialer and never tracked, so a captured handshake is replayable. After the handshake, frames are unauthenticated JSON. The frame set includes MAINTENANCE_MODE_SET and POP_MAINTENANCE_MODE_SET.

Anyone who can open a TCP connection to that port can therefore stop the cell from accepting pushes or delivering messages, without any token. Restricting the mesh port to the broker replicas themselves is a requirement, not a hardening tip.

No migration routes

There is no /api/v1/migration/* surface: those routes were removed. Backing up Queen is a plain pg_dump of the database, and the offline rows-to-segments conversion is a CLI subcommand of the binary, not an HTTP endpoint.

Access levels

Deployment guidance for these surfaces (what to expose, where to terminate TLS, how the proxy classifies routes) is in the self-hosting section.