Run Queen as a multi-tenant service

The broker and the gateway in front of it are both Apache-2.0. Nothing in this repository is a hosted-only component, which means anyone can stand up the same multi-tenant service in-house: for internal teams, for customers, or for a product built on top. This section documents how that system is put together, what it enforces, and where its edges are.

It is not a description of a commercial offering. It is the design of proxy/, read out of the code.

Three nouns

Users belong to a tenant; permissions do not. A user’s role is granted per cluster (queen_proxy.cluster_roles), so one person can be admin on one cluster and a viewer on another inside the same organisation.

A tenant may own many clusters: production, staging, one per region. Each cluster is a separate namespace on the broker and a separate quota bundle.

Cluster to cell

flowchart LR
T1["tenant acme"] --> C1["cluster acme-prod"]
T1 --> C2["cluster acme-staging"]
T2["tenant globex"] --> C3["cluster globex"]
C1 --> S1["cell eu1-shared-a<br/>one proxy, one broker, one PostgreSQL"]
C2 --> S1
C3 --> S2["cell eu1-ded-7<br/>one proxy, one broker, one PostgreSQL<br/>dedicated: nothing else on it"]

Each cell is a whole stack in its own right, which is what makes the arrows go one way only: a cluster names a single cell and never spans two.

The relationship is N:1 on shared cells and 1:1 on dedicated ones. Plans carry a cell_class of shared or dedicated and cells carry the same field, but the pairing is a placement decision made by whoever runs the fleet. There is no database constraint that stops a shared cluster landing on a dedicated cell. See Tenants, clusters and cells for what that means in practice, including the fact that placement is manual today.

Each cluster also carries a broker_tenant_uuid, generated when the cluster is created. That UUID, not the tenant id, is what the proxy injects as x-queen-tenant upstream, and it is what scopes queue identity inside the broker.

One cluster lives on exactly one cell

This is the structural decision the whole enforcement model rests on. A cluster’s traffic can only arrive at the one proxy that fronts its cell. Therefore:

• Rate limits are exact. The request and message token buckets are plain in-process state, sharded 16 ways by cluster id. There is no Redis, no gossip, no approximation of a shared counter, and no coordination latency in the request path.
• Parked-consumer slots are exact. A long-poll pop holds an RAII guard across the upstream await; the gauge is a per-cluster atomic plus a cell-wide one.
• Queue and partition caps are exact. The proxy remembers every (queue, partition) pair it has admitted for the cluster, so the common case is a set lookup and no database round trip.
• Usage is aggregated locally and flushed per minute, rather than counted twice and reconciled.

The cost of that design is honest and worth stating: the counters are process-local, so a proxy restart resets them. Token buckets start full, and the in-memory push-block flags are empty until the pumps that own them run again: the storage flag within about ten seconds of the next successful reconcile, the monthly-quota flag on the rollup task’s first tick, which fires immediately at startup for exactly this reason.

Running two proxies in front of one cell would break the exactness, because each would enforce its own share of the limits. One proxy per cell is not a deployment convention, it is the assumption the code is written against.

Routing is a DNS label

The proxy resolves the cluster from the first label of the Host header:

Host: acme-prod.eu1.example.com  →  slug "acme-prod"
Host: Acme-Prod.eu1.example.com:6711  →  slug "acme-prod"

The port is stripped when what follows the colon is all digits, and the label is lowercased because clusters.slug is stored lowercase and DNS is case-insensitive. A slug that resolves to no cluster answers 421 with {"code":"cluster_unknown"}, not 404, because the request reached the wrong endpoint rather than asking for a missing thing.

clusters.slug is checked against ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$. It is literally used as a DNS label, so it is validated as one, and it is globally unique across all tenants.

One hostname per cluster is the normal shape. The dashboard is the exception: it is served from a single hostname for every cluster, so a browser has no label to route by. That case uses an explicit x-queen-act-cluster header, which is honoured for a human session and refused for an API key. Credentials and authorization has the matrix.

What a request goes through

The order is load-bearing, and it is the order in the gateway:

What the proxy is not

• Not a control plane. Every mutation of tenants, clusters, users, keys, plans and roles happens through SQL functions in the queen_proxy schema. The proxy reads that data; it does not own its lifecycle. There is no HTTP endpoint that creates a tenant.
• Not a billing console. It has a cluster console (usage, keys, members) and nothing about invoices or payment.
• Not a second broker. It parses request bodies to count and cap them and it never rewrites one. A refused /configure is an error the caller sees, not a silently clamped configuration.
• Not an isolation layer bolted onto an open broker. The broker does its own tenant scoping natively when the flag is on. The proxy is what makes the header that drives it trustworthy. Isolation draws the line precisely.

The pieces

Read next