Deploying queen_proxy

queen_proxy is a separate Rust binary. One instance fronts one cell, and it needs its own PostgreSQL database: the control-plane store, called pxdb throughout the code. It is not the database the broker writes messages to.

Keeping those two apart is structural. The cell’s PostgreSQL holds one tenant’s worth of messages per cluster and is sized for message throughput; pxdb holds the tenant, cluster, key, plan and usage tables for the whole fleet and is read almost entirely out of a 30-second in-process cache. A pxdb outage must not stop the data plane, and the cache is what makes that true.

Boot, in order

Every step below can end the process, and each does so with a log line naming the cause. Nothing is retried at boot except the JWKS-style lazy work.

On shutdown the listener stops accepting, live connections are given their in-flight request to finish (10 seconds on the TLS path), and then the metering accumulators are flushed, bounded by QUEEN_PROXY_SHUTDOWN_DRAIN_MS, falling back to the disk spool, so a dead pxdb cannot hold a deploy open.

Migrations are applied at boot

Six SQL files are embedded in the binary with include_str! and applied in order, each recorded by name in queen_proxy.schema_migrations and skipped if already recorded:

Every statement is idempotent (IF NOT EXISTS / OR REPLACE), so the files are also safe to apply by hand with psql against an empty database. 001_init runs CREATE EXTENSION IF NOT EXISTS pgcrypto WITH SCHEMA public, so the role applying it needs that privilege. This is the one place the proxy differs from the broker, which requires no extensions at all.

TLS termination

The listener is HTTPS only when both variables are set:

QUEEN_PROXY_TLS_CERT=/etc/queen/origin.pem

QUEEN_PROXY_TLS_KEY=/etc/queen/origin.key

One without the other is a misconfiguration the process refuses to start with, rather than silently serving the plaintext port. Details worth knowing before you generate the material:

• The certificate file is read as a full chain, leaf first: every CERTIFICATE block in the file is used, not just the first. A file with no CERTIFICATE block is a fatal error.
• The key may be PRIVATE KEY (PKCS#8), RSA PRIVATE KEY (PKCS#1) or EC PRIVATE KEY (SEC1). The first key block found wins.
• ALPN advertises http/1.1 only. The TLS listener is hyper’s HTTP/1 server, so a client that negotiates h2 would fail; the proxy avoids that by never offering it. If you need HTTP/2 at the edge, terminate it upstream and speak HTTP/1.1 to the proxy.
• rustls on the ring provider, the same crypto stack the outbound PostgreSQL connector uses, so TLS adds no second backend.

With both variables unset, the listener is plaintext and something in front of it is expected to terminate TLS.

The cell secret

The proxy always strips the client’s Authorization header before forwarding. The client’s credential is spent at the proxy; the broker never sees it. In its place, if the cell row carries a cell_secret, the proxy sends Authorization: Bearer <cell_secret>.

That is the whole of the broker-facing credential. Two consequences:

• Whatever the broker requires, this one value must satisfy. If the cell’s broker has JWT_ENABLED=true, cell_secret has to be a token that broker’s validator accepts. And because the proxy forwards routes the broker classifies as admin (a queue delete, a consumer-group delete), the token needs the broker’s admin role. A key with a narrower role will pass pushes and fail queue administration.
• cell_secret is stored in plaintext in queen_proxy.cells. Anyone with read access to pxdb holds the cell’s broker credential. Treat pxdb as a secret store, and the broker as reachable only from the proxy. Isolation explains why the second half of that is not optional.

The cell’s base_url must be an http:// URL. The data path would parse either, but the registry reconciler refuses anything else outright: a deliberate assertion that broker traffic is cell-internal plaintext, with TLS terminated at the proxy edge.

The cache, and what happens when pxdb is down

Two caches back every request: Host slug → cluster context, and API-key sha256 → (cluster, key id, scopes).

The short negative TTL is deliberate: a loop of garbage keys re-checks the database every five seconds rather than being answered from memory, because the database is the only authority on whether a hash exists, while still capping how hard that loop can hit pxdb.

Invalidation is push-based. Every mutating control-plane function calls record_operation, which appends the audit row and fires pg_notify('queen_proxy_inval', <cluster_id>). The proxy holds a dedicated PostgreSQL connection for LISTEN (deliberately not a pooled one, because the pool driver discards notifications) and drops every cached entry for that cluster on arrival. The connection reconnects with backoff from 1 s to 30 s, and the backoff resets to its floor once a session has stayed up at least 10 seconds.

When pxdb cannot answer, the rule distinguishes two failures that look alike:

The grace window is QUEEN_PROXY_STALE_GRACE_MS, default 600000 (10 minutes). It covers a managed-PostgreSQL failover, a restart or a migration. It is also the worst-case delay before a control-plane decision taken during the outage (a suspension, a deletion, a key revocation) takes effect, because nothing refreshes while pxdb is down. Setting it to 0 disables stale serving entirely.

api_keys.last_used_at is bumped only on a cache miss, so it is accurate to within the 30-second positive TTL and costs nothing per request.

Configuration

Only the literal string true is truthy, matching the proxy’s own parser. Empty strings are treated as unset for the Option values.

Listener and control-plane database

Enforcement and upstream

Identity

Metering

Development only

QUEEN_PROXY_DEV_INSECURE=true hands every request a full-scope API-key principal before any credential is read. It also makes the cluster console permanently 403, since that surface requires a human session and the bypass produces an API-key principal. Nothing about this mode belongs on a shared deployment.

Endpoints the proxy owns

Everything else is forwarded to the broker, so these are the paths that never leave the proxy:

/healthz reporting enforce is there for a reason: enforcement used to be visible only in the boot log, which survives a restart, so a check that grepped the log could pass against a proxy that had since been replaced by a shadow-mode one.

Per cell, then