--- title: "Kubernetes" description: "A StatefulSet, two Services and the probe wiring, written against this broker: /health is the readiness question and never the liveness one." --- > Queen MQ documentation, for AI agents > Complete self-contained summary of Queen MQ: https://queenmq.com/llms-brief.txt > Fetch that first when the question is about the product rather than about this page. > Index of all pages: https://queenmq.com/llms.txt # Kubernetes Three replicas, a headless Service so they can find each other, and a client Service carrying only the HTTP port. The manifest below is the deployment in full: apply it with `-n queen`, against a PostgreSQL at `queen-pg`. Every value in it that is not obvious is explained underneath, including the three that decide whether the broker performs. ```bash kubectl -n queen create secret generic queen-secrets \ --from-literal=pg-password=change-me \ --from-literal=sync-secret=change-me-too ``` `sync-secret` is the HMAC secret for the mesh handshake and must be byte-identical on every replica. Empty means open mode: any well-formed handshake from anyone reaching the port is accepted. ```yaml apiVersion: v1 kind: Service metadata: name: queen spec: selector: app.kubernetes.io/name: queen ports: - name: http port: 6632 targetPort: http protocol: TCP --- apiVersion: v1 kind: Service metadata: name: queen-mesh spec: clusterIP: None publishNotReadyAddresses: true selector: app.kubernetes.io/name: queen ports: - name: mesh port: 6633 targetPort: mesh protocol: TCP --- apiVersion: apps/v1 kind: StatefulSet metadata: name: queen spec: serviceName: queen-mesh replicas: 3 podManagementPolicy: Parallel selector: matchLabels: app.kubernetes.io/name: queen template: metadata: labels: app.kubernetes.io/name: queen spec: terminationGracePeriodSeconds: 60 securityContext: runAsNonRoot: true runAsUser: 65532 runAsGroup: 65532 fsGroup: 65532 containers: - name: queen image: ghcr.io/queen-mq/queen:latest command: ["/app/bin/queen"] ports: - name: http containerPort: 6632 - name: mesh containerPort: 6633 env: - name: PG_HOST value: queen-pg - name: PG_PORT value: "5432" - name: PG_USER value: queen - name: PG_DATABASE value: queen - name: PG_PASSWORD valueFrom: secretKeyRef: { name: queen-secrets, key: pg-password } - name: DB_POOL_SIZE value: "60" - name: PORT value: "6632" - name: QUEEN_MESH_PORT value: "6633" - name: QUEEN_MESH_PEERS value: "queen-0.queen-mesh:6633,queen-1.queen-mesh:6633,queen-2.queen-mesh:6633" - name: QUEEN_SYNC_SECRET valueFrom: secretKeyRef: { name: queen-secrets, key: sync-secret } - name: QUEEN_SERVER_ID valueFrom: fieldRef: { fieldPath: metadata.name } - name: FILE_BUFFER_DIR value: /var/lib/queen/buffers - name: QUEEN_LOG_JSON value: "true" startupProbe: httpGet: { path: /health, port: http } periodSeconds: 10 failureThreshold: 30 readinessProbe: httpGet: { path: /health, port: http } periodSeconds: 10 failureThreshold: 3 livenessProbe: tcpSocket: { port: http } periodSeconds: 20 failureThreshold: 3 resources: requests: { cpu: "1", memory: 512Mi } limits: { memory: 2Gi } securityContext: readOnlyRootFilesystem: true volumeMounts: - name: spool mountPath: /var/lib/queen/buffers - name: tmp mountPath: /tmp volumes: - name: tmp emptyDir: {} volumeClaimTemplates: - metadata: name: spool spec: accessModes: ["ReadWriteOnce"] resources: requests: { storage: 5Gi } ``` ## The probes `/health` takes a pooled connection and runs a real query, which makes it the readiness answer: a broker that cannot reach PostgreSQL should leave the Service endpoints. Wire it to `livenessProbe` and a PostgreSQL outage becomes a kill loop on every replica: the pushes the disk spool exists to absorb are refused at the socket instead, and the spool goes with the container if it sits on the container filesystem. Liveness is a TCP connect on the HTTP port, or nothing: a panic aborts the process, so there is no half-alive state to detect. The startup probe gives the first boot five minutes: the broker applies its embedded schema before it serves. ## Why a StatefulSet - `QUEEN_MESH_PEERS` is a list of hostnames parsed once at boot, and only a StatefulSet behind a headless Service hands out stable names to put in it. - The spool is node-local and replayed only by a broker starting against the same `FILE_BUFFER_DIR`, which is what one claim per ordinal buys. For one replica that accepts losing an unreplayed spool: drop the claim template, mount an `emptyDir` there, remove `QUEEN_MESH_PEERS`, and a Deployment is equivalent. ## The fields that are not obvious - `replicas` times `DB_POOL_SIZE` must fit PostgreSQL's `max_connections`: 180 here. - `terminationGracePeriodSeconds` must exceed your longest pop timeout, 30 seconds by default, because a parked long-poll pop is an in-flight request that SIGTERM drains. - Port 6633 is TCP despite the legacy `QUEEN_UDP_*` alias names, and a Service or NetworkPolicy declaring UDP blocks the mesh silently: brokers keep serving, with pop wakes arriving late. - The peer list is uniform and names the pod itself, which is safe because a frame is applied to local state and never re-broadcast. - No CPU limit, but not for the reason usually given: tokio sizes its worker pool from `available_parallelism()`, which honours the cgroup quota on the toolchain this builds with, so a limit would size the pool correctly rather than oversubscribe it. The cost is CFS throttling. The commit path is bursty, and a quota turns a burst into scheduler latency at exactly the percentile you are watching. Memory has no such caveat and is set. - `podManagementPolicy: Parallel` is safe: the schema apply serialises cluster-wide on a PostgreSQL advisory lock. - Nothing but the other brokers belongs on 6633: keep it off the client Service and close it with a NetworkPolicy ([the mesh port](/deploy/security#the-mesh-port)). ## The three sizing rules None of these shows up in a `kubectl get`, and each one is the difference between a broker that runs and a broker that performs. - **`PG_PORT` points at PostgreSQL directly, not at a pooler.** The broker holds session-scoped advisory locks across the boot schema apply and across every retention cycle, plus the statements the hot paths prepare once per connection, and all of that lives on one specific backend session. PgBouncer in session mode is the supported way to pool, and it pins a server connection per client, so it multiplexes nothing for a long-lived application pool anyway: 28 client connections were measured against 28 backends. It can still recycle the server underneath and drop a held lock, which is what `pg_advisory_unlock reported not-held` in the retention logs was. Full reasoning in [PostgreSQL](/deploy/postgres#give-it-a-direct-connection). - **`DB_POOL_SIZE` has to comfortably exceed `QUEEN_ADMISSION_POOL_RESERVE`, default 16.** The admission arbiter derives its band from `DB_POOL_SIZE` minus that reserve, so with the default reserve a pool of 24 or less collapses the band to `[8,8]`: the broker then runs at a fixed eight concurrent write transactions whatever the disk can take, and the AIMD adapter stops adapting. The 60 in the manifest gives a band of `[29,44]`. If PostgreSQL cannot take `replicas × DB_POOL_SIZE`, lower `QUEEN_ADMISSION_POOL_RESERVE` alongside rather than shrinking the pool on its own ([flow control](/internals/flow-control)). - **Leave `QUEEN_STMT_TIMEOUT_MS` at its 30000 default unless you have measured a reason.** An admission slot is held across the timed call, so a longer deadline means a wedged statement occupies a write slot for that much longer, and the budget the arbiter is trying to hold shrinks by one for the duration. ## Verify ```bash kubectl -n queen rollout status statefulset/queen kubectl -n queen logs queen-0 | grep -E 'schema|config:|listening|mesh' ``` The `mesh` line reports `peers` and `auth`: `auth=hmac` means the secret arrived, `auth=off` is open mode. Every variable above, and the ones left at their defaults, is in [configuration](/reference/config). Source: https://queenmq.com/deploy/kubernetes/index.mdx