Tenants, API keys and quotas on one cluster

The broker on its own has no users: anything that reaches its port can use it. The proxy built into the same binary adds them. With it on, every request presents an API key or a console session, and the proxy maps it to one tenant’s cluster, checks it against the plan’s limits, meters it, and hands it to the broker inside the same process. The proxy keeps its own state (tenants, users, key hashes, plans, usage) in the broker’s replicated KV, so every node runs the same proxy and there is no database beside it.

For more write capacity across tenants, give them their own raft groups: see Many tenants on one cluster.

Turn it on

Generate the secrets once, keep them in your secret store, and give every node the same values:

SESSION_SECRET=$(openssl rand -hex 32)   # signs console sessions
CP_TOKEN=$(openssl rand -hex 32)         # opens the control plane, /api/cp/*
QUEEN_KEY="qk_live_$(openssl rand -base64 32 | tr '+/' '-_' | tr -d '=')"

docker run -d --name queen -p 6711:6711 -p 127.0.0.1:6632:6632 \
  -v queen-data:/var/lib/queen/raft \
  -e QUEEN_PROXY_EMBEDDED=true \
  -e QUEEN_PROXY_PORT=6711 \
  -e QUEEN_PROXY_JWT_SECRET="$SESSION_SECRET" \
  -e QUEEN_PROXY_CP_TOKEN="$CP_TOKEN" \
  -e QUEEN_PROXY_SPOOL_DIR=/var/lib/queen/raft/proxy-spool \
  -e QUEEN_PROXY_BOOTSTRAP_TENANT=acme \
  -e QUEEN_PROXY_BOOTSTRAP_API_KEY="$QUEEN_KEY" \
  -e QUEEN_PROXY_DEFAULT_CLUSTER=acme \
  ghcr.io/queen-mq/queen:2.6.0

A push through the proxy’s port now needs the key, which clients send as a bearer token:

curl -s -X POST localhost:6711/api/v1/push -H "authorization: Bearer $QUEEN_KEY" \
  -H 'content-type: application/json' \
  -d '{"items":[{"queue":"orders","partition":"customer-123","payload":{"orderId":8891}}]}'

[{"index":0,"message_id":"01a0fcca-5436-7000-ab29-ef2bea95314f","transaction_id":"01a0fcca-5436-7000-ab29-ef2bea95314f","queueName":"orders","status":"queued","offset":0}]

Without it the answer is 401 unauthorized. The bootstrap runs at every boot on every node and changes nothing once its rows exist: it creates the tenant acme, a cluster of the same name on the dev plan, an admin user, and the key you gave, which must start with qk_. Keep QUEEN_PROXY_SPOOL_DIR on the data volume.

Two ports

Run a cluster with the proxy on its own port: the nodes talk to each other over PORT, and the membership routes are blocked through the proxy. Port 6632 is then for trusted clients, probes and scrapers inside the network, and never faces the internet.

Tenants, clusters and keys

A tenant is an organisation, and its users belong to it. A tenant owns clusters, and a cluster is what clients talk to: a slug, a plan, and a broker tenant of its own, so the queues, KV and timers of one cluster are invisible to every other. The proxy finds the cluster from the first label of the Host (acme.queen.example.com is acme), or from QUEEN_PROXY_DEFAULT_CLUSTER. A key is bound to its cluster for life.

A key is qk_<env>_ and 43 random characters, and only its sha256 is stored. A key carries scopes; a person holds one role per cluster:

A revoked key stops working on every node within about a second.

Plans and quotas

Four plans are seeded at the first boot, and per-cluster overrides change single limits (PUT /api/cp/clusters/:id/overrides, where null is unlimited):

The byte rate (max_bytes_per_sec) counts uncompressed bytes, and no plan sets one. The proxy starts in shadow mode (QUEEN_PROXY_ENFORCE=false): limits are computed and logged with would_block=true, and the request goes through, so you can watch a plan against real traffic before it bites. With QUEEN_PROXY_ENFORCE=true the same decisions answer 429 or 403. The body cap, the storage quota and the monthly quota apply in both modes.

Every request is metered per cluster, minute and class (push, delivery, transaction, configure, read), kept for 90 days, and read back with GET /api/cp/clusters/:id/usage.

The console and sign-in

/console on the proxy’s port is for people: the plan, limits and usage, the queues, the keys and the members; the dashboard is at / behind the same login. Sign-in takes a local password, Google or GitHub. Operators, the people who run the deployment (QUEEN_PROXY_OPERATOR_ENABLED=true, QUEEN_PROXY_OPERATORS=ops@example.com), are admin on every cluster and see the node and raft pages; the list is read at every boot and is the whole truth.

The control plane

Everything after the bootstrap can be driven over /api/cp/* on the proxy’s port, with x-queen-cp-token on every call. Every route is safe to repeat: a retry of a call that already took effect answers success with the same ids.

provision is the call to build on. You mint the key and send only its sha256, so the plaintext never reaches the proxy:

KEY="qk_live_$(openssl rand -base64 32 | tr '+/' '-_' | tr -d '=')"
HASH=$(printf %s "$KEY" | sha256sum | cut -d' ' -f1)
curl -s -X POST localhost:6711/api/cp/provision -H "x-queen-cp-token: $CP_TOKEN" \
  -H 'content-type: application/json' -d '{
    "tenant_slug": "globex", "cluster_slug": "globex", "plan": "pro",
    "user_id": "308d3f3d-2306-41de-95f1-e32b12ec7470", "email": "ops@globex.example",
    "role": "admin", "key_name": "default", "key_hash": "'"$HASH"'",
    "scopes": ["produce", "consume", "read"], "overrides": {"max_queues": 500}}'

Taking a tenant down is three steps, refused out of order: set its status to deleting, purge until the answer says done: true, then delete it.

Limits

• Rate buckets live in each node’s memory: behind a load balancer over three nodes, a cluster can reach three times its plan’s rate. The storage quota is shared.
• Ephemeral requests forwarded between nodes keep their tenant only when every node has the same QUEEN_RAFT_TOKEN.