Tenants, clusters and cells

Everything the proxy needs to route, authorize, limit and bill a request lives in the queen_proxy schema of its own PostgreSQL. The proxy reads those tables through a 30-second cache and never writes the lifecycle ones: mutations go through SQL functions, which is what keeps the audit trail and the cache invalidation from being optional.

The tables

queues deserves the emphasis on cache. The broker owns queue existence; this table exists so the proxy can answer “how many queues does this cluster have” without a round trip per push. It stores a partition count, never partition names (neither this table nor the broker’s own listing exposes them), which is why the proxy keeps the exact pairs it admitted in memory and treats the stored count as a floor after a restart.

Status, and how two statuses combine

Both tenants.status and clusters.status are TEXT with a CHECK, not an enum, so adding a state is a one-line migration.

The two are merged worst-wins: a tenant in grace puts every one of its clusters into the push-blocked state without touching their rows. An unrecognised status value is treated as at least as restrictive as suspended, so a future state cannot fail open.

grace mapping to push-blocked rather than a full suspend is a deliberate choice: a payment problem should stop new data arriving, not strand data the tenant has already paid to store.

Plans and per-cluster overrides

Four plans are seeded by 002_functions. Every value is a hard cap the proxy enforces, NULL meaning unlimited:

These are the seeds shipped in the migration, not a recommendation. plans.code is an open catalog key with only a format check, so add your own rows rather than reshaping these.

monthly_msgs_quota is NULL on every seeded plan. Nothing monthly is enforced until you set one. The mechanism exists and is described in Usage metering; it simply has no value to compare against out of the box.

Per-cluster deltas live in clusters.limit_overrides, a jsonb object whose keys match the plan column names one-to-one. The merge has exactly three cases:

SELECT queen_proxy.set_limit_override(
  (SELECT id FROM queen_proxy.clusters WHERE slug = 'acme-prod'),
  '{"max_queues": 500, "max_retained_bytes": null}'::jsonb
);

(acme-prod is the cluster provisioned in the next section; the function takes the cluster uuid, fetched here by slug.)

Passing NULL or '{}' clears every override. Support raising one limit for one customer is therefore a row update, not a forked plan.

Provisioning

The control-plane contract is: write only through queen_proxy.* functions. Each one validates its input, writes its tables, then calls record_operation, which appends the audit row and, when a cluster is involved, fires pg_notify('queen_proxy_inval', <cluster_id>) so every proxy drops that cluster’s cached entries immediately.

One call to onboard

The cell id is the one argument you cannot type from memory. Fetch it from the slug of the cells row you inserted:

SELECT id FROM queen_proxy.cells WHERE slug = 'eu1-shared-a';

or inline the lookup, which makes the whole call copy-paste runnable:

SELECT queen_proxy.bootstrap_tenant(
  'acme',                    -- tenant slug
  'Acme Inc',                -- tenant name
  'acme-prod',               -- cluster slug (becomes the subdomain)
  'free',                    -- plan code
  (SELECT id FROM queen_proxy.cells WHERE slug = 'eu1-shared-a'),
  'ops@acme.example',        -- admin email
  'initial-password',        -- NULL for an OAuth-only admin
  'default'                  -- API key name
);

It returns {tenant_id, cluster_id, user_id, api_key} and composes the individual functions, so a RAISE anywhere unwinds the whole thing: there is no half-created tenant. The api_key field is the plaintext key, shown exactly once; only its sha256 hash is stored.

It is idempotent on the slugs: a re-run returns the existing ids with api_key set to null, because the plaintext cannot be recovered. It still re-asserts the admin role. Two collisions are refusals rather than repeats: a cluster slug owned by another tenant, and an email that already belongs to another tenant. Both columns are globally unique.

A password passed here is bcrypt-hashed in the database with pgcrypto, at cost 10, in a form the login path accepts.

The individual functions

set_operator has no HTTP path on purpose. The routes an operator may open are cell-wide and not tenant-scopable, so if any console endpoint could set that bit, a cluster admin could escalate to reading (and pausing) every other tenant on the cell. Granting it requires a psql session on pxdb.

API keys

The format is qk_<env>_<43 base64url characters>, generated from 32 bytes of the OS CSPRNG. Only sha256(key) in lowercase hex is stored, in api_keys.key_hash, which carries a CHECK for that shape. The plaintext exists once, in the response of the call that created it.

The <env> segment (live, test, whatever you choose) is informational. Authentication routes on the qk_ prefix and then matches on the hash alone; nothing parses the environment out.

Scopes are a TEXT[] constrained to a subset of produce, consume, admin, read, and at least one is required. What each scope opens is in Credentials and authorization.

A key is bound to the cluster it was issued on, for its whole life. Presenting it on another cluster’s hostname is 403, and the x-queen-act-cluster header is refused for API keys outright.

Cells

Cells are fleet infrastructure, so no SQL function creates one. cell_secret is the bearer token the proxy presents upstream: a JWT carrying the broker’s admin role, in the stock setup HS256-signed with that cell broker’s JWT_SECRET. Mint it with the recipe in Deploying queen_proxy, which leaves it in $CELL_SECRET, then hand it to psql as a variable. The -v plus :'cell_secret' pair is psql interpolation; it works from stdin or -f, not from -c:

psql -h 10.0.3.20 -U postgres -d queen_proxy -v cell_secret="$CELL_SECRET" <<'SQL'
INSERT INTO queen_proxy.cells (slug, region, base_url, class, cell_secret)
VALUES ('eu1-shared-a', 'eu1', 'http://10.0.3.11:6632', 'shared', :'cell_secret');
SQL

A minted value looks like this (an example signed with a throwaway secret; its claims decode to {"sub":"queen-proxy","role":"admin"}, no exp):

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJxdWVlbi1wcm94eSIsInJvbGUiOiJhZG1pbiJ9.CECMnGKvXYLGZtvQuCcCCKwG2M0xYeq4vlatq1l5gDM

Every other column has a working default: id and created_at generate themselves, status starts active, and capacity_slots, used_slots and broker_version are cosmetic today (see below). On a cell whose broker runs JWT_ENABLED=false behind pure network isolation, insert NULL as the secret and the proxy forwards no Authorization header at all.

The new row’s id is the value create_cluster and bootstrap_tenant take as their cell argument:

SELECT id FROM queen_proxy.cells WHERE slug = 'eu1-shared-a';

base_url must be http://: the registry reconciler refuses anything else, as an assertion that broker traffic is cell-internal. The secret is stored in plaintext; Deploying queen_proxy covers what it must satisfy and why pxdb has to be treated as a secret store.

Placement is manual, and draining does not exist

Two limitations to plan around, both verifiable by what the code does not read:

• create_cluster takes the cell id as an argument. Nothing chooses a cell for you: no capacity check, no balancing, no affinity rule. Whoever calls it decides, and getting it wrong means a cluster on a cell that cannot carry it.
• cells.status, cells.capacity_slots and cells.used_slots are never read by any code in the tree. Setting a cell to draining changes nothing: new clusters can still be created on it, the reconciler still polls it, and traffic still routes to it. used_slots is never incremented, so it is not a count of anything.

There is also no cluster migration path. Moving a cluster between cells means moving its data, and nothing here does that.

Until those gaps are filled, treat cell placement as an operational procedure with a written record outside the database, and size cells with enough headroom that you are never forced to move a cluster in a hurry.

The audit trail

operations is append-only, indexed by (tenant, at DESC) and by (cluster, at DESC). Every function above writes exactly one row, with an actor of user, api_key, control_plane or system. The higher-level functions hardcode control_plane; record_operation itself takes an explicit actor, and the self-serve console is the caller that uses it today. Both the console handlers and the OAuth sign-in path call queen_proxy.record_operation(…, 'user', …) with the signed-in user’s id, so a console action that also runs a higher-level function leaves two rows: the function’s own control_plane row and a second, user-attributed one. That is deliberate, and it is why a user actor in the audit trail means a human acted rather than an automation did. The second row is best-effort: a failed record_operation logs a warning and does not fail the action it was recording.

cluster_id on that table is ON DELETE SET NULL rather than cascading, so a hard-deleted cluster’s history outlives the row. tenant_id is NOT NULL and restricts, which is another way of saying: this design expresses tenant lifecycle as a status column, and DELETE FROM tenants is not part of it.

Related

• The self-serve console: the caller that drives these functions today, and what it writes to the audit trail.
• Credentials and authorization: what a key or a session can actually do with these rows.
• Quotas and rate limits: how each plan column is enforced, and which ones ignore shadow mode.
• Usage metering: how usage_minutes is filled and rolled up.