--- title: "Kubernetes" description: "A complete reference manifest written for this broker: the real image and command, the mesh port as TCP, peers for replicas, a Secret for the mesh secret, and probes that are not coupled to the database." --- > Documentation Index > Fetch the complete documentation index at: https://queenmq.com/llms.txt > Use this file to discover all available pages before exploring further. # Kubernetes There is **no official Helm chart** for Queen today. What follows is a complete reference manifest you can apply as-is, written against this broker's actual behaviour rather than adapted from a generic message-queue template. Every non-obvious field is explained below the YAML, and the two fields most commonly got wrong (the liveness probe and the mesh port) are explained at length, because getting either wrong produces failures that look like something else. Assumptions: PostgreSQL already exists and is reachable at `queen-pg` in the same namespace, and you have a namespace called `queen`. ## Why a StatefulSet A single-replica broker works perfectly well as a `Deployment`. Once you want more than one, a `StatefulSet` earns its keep for two reasons that are specific to this broker: - **The mesh peer list is static hostnames.** `QUEEN_MESH_PEERS` is parsed once at boot into `(host, port)` pairs. A `StatefulSet` with a headless Service gives every pod a stable DNS name (`queen-0.queen-mesh`, `queen-1.queen-mesh`) that you can write into that variable. Pods behind a `Deployment` have neither stable names nor stable addresses, so there is nothing to put there. - **The disk spool wants to follow the pod.** `FILE_BUFFER_DIR` holds pushes the broker accepted while PostgreSQL was unreachable, and they are only replayed by a broker that starts against the same directory. A `volumeClaimTemplate` binds one PersistentVolumeClaim per ordinal, so a rescheduled pod finds its own spool again. An `emptyDir` survives a container restart but not a pod replacement. If you run one replica and accept that an unreplayed spool dies with the pod, drop the `volumeClaimTemplate`, mount an `emptyDir` at the spool path, remove `QUEEN_MESH_PEERS`, and a `Deployment` is equivalent. Nothing else changes. ## The manifest ### Secret ```yaml apiVersion: v1 kind: Secret metadata: name: queen-secrets namespace: queen type: Opaque stringData: # The PostgreSQL password for the broker's role. pg-password: change-me # HMAC secret for the mesh HELLO handshake. Must be byte-identical on every # replica. An empty value means "open mode": any well-formed handshake is # accepted from anyone that can reach the port. sync-secret: change-me-too ``` ### Services Two Services, with different jobs. The client-facing one carries **only** the HTTP port; the headless one exists so the pods can resolve each other for the mesh. ```yaml apiVersion: v1 kind: Service metadata: name: queen namespace: queen spec: type: ClusterIP selector: app.kubernetes.io/name: queen ports: - name: http port: 6632 targetPort: http protocol: TCP --- apiVersion: v1 kind: Service metadata: name: queen-mesh namespace: queen spec: clusterIP: None # Peers must be resolvable before they are Ready: a broker's readiness depends # on PostgreSQL, and the mesh should come up regardless of that ordering. publishNotReadyAddresses: true selector: app.kubernetes.io/name: queen ports: - name: mesh port: 6633 targetPort: mesh protocol: TCP ``` ### StatefulSet ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: queen namespace: queen spec: serviceName: queen-mesh replicas: 3 podManagementPolicy: Parallel selector: matchLabels: app.kubernetes.io/name: queen template: metadata: labels: app.kubernetes.io/name: queen spec: # A parked long-poll pop is an in-flight request. SIGTERM drains in-flight # requests before the process exits, and the default pop timeout is 30 s, # so give the drain more than that. terminationGracePeriodSeconds: 60 securityContext: # The published image declares no USER, so it runs as root unless you say # otherwise. fsGroup is what makes the spool volume writable by a non-root # UID. runAsNonRoot: true runAsUser: 65532 runAsGroup: 65532 fsGroup: 65532 containers: - name: queen image: ghcr.io/queen-mq/queen:latest # The image's own default command, spelled out. WORKDIR is /app. command: ["/app/bin/queen"] ports: - name: http containerPort: 6632 protocol: TCP - name: mesh containerPort: 6633 protocol: TCP env: - name: PG_HOST value: queen-pg - name: PG_PORT value: "5432" - name: PG_USER value: queen - name: PG_DATABASE value: queen - name: PG_PASSWORD valueFrom: secretKeyRef: name: queen-secrets key: pg-password # replicas x DB_POOL_SIZE must fit inside PostgreSQL's # max_connections: 3 x 60 = 180 here. - name: DB_POOL_SIZE value: "60" - name: PORT value: "6632" # Mesh. The list is uniform across pods and includes this pod itself; # see the note below. - name: QUEEN_MESH_PORT value: "6633" - name: QUEEN_MESH_PEERS value: "queen-0.queen-mesh:6633,queen-1.queen-mesh:6633,queen-2.queen-mesh:6633" - name: QUEEN_SYNC_SECRET valueFrom: secretKeyRef: name: queen-secrets key: sync-secret # Names this replica in mesh heartbeats and peer stats. Taken from the # pod name so it is stable and readable in the logs. - name: QUEEN_SERVER_ID valueFrom: fieldRef: fieldPath: metadata.name - name: FILE_BUFFER_DIR value: /var/lib/queen/buffers - name: LOG_LEVEL value: info # One JSON object per line, for a structured log shipper. - name: QUEEN_LOG_JSON value: "true" # Do not start counting failures until the broker has applied its # schema and reached PostgreSQL once. 30 x 10 s = up to 5 minutes. startupProbe: httpGet: path: /health port: http periodSeconds: 10 failureThreshold: 30 # /health does a real database round-trip, so this correctly removes a # broker that cannot serve from the Service endpoints. readinessProbe: httpGet: path: /health port: http periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3 # DELIBERATELY not /health. See "Do not couple liveness to the # database" below. This checks only that the listener still accepts. livenessProbe: tcpSocket: port: http periodSeconds: 20 timeoutSeconds: 5 failureThreshold: 3 resources: requests: cpu: "1" memory: 512Mi limits: # Read the note on CPU limits below before setting this. memory: 2Gi securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: ["ALL"] volumeMounts: - name: spool mountPath: /var/lib/queen/buffers # If the spool directory cannot be created the broker falls back to # the OS temp directory, which a read-only root filesystem would # deny. Mount one so the fallback has somewhere to go. - name: tmp mountPath: /tmp volumes: - name: tmp emptyDir: {} volumeClaimTemplates: - metadata: name: spool spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 5Gi ``` ### NetworkPolicy The mesh port must not be reachable from anywhere except the other brokers. This is a requirement, not hardening. See [High availability](/selfhost/ha) for why. ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: queen namespace: queen spec: podSelector: matchLabels: app.kubernetes.io/name: queen policyTypes: ["Ingress"] ingress: # HTTP: from whatever should be allowed to call the API. Narrow this. - from: - podSelector: matchLabels: queen-client: "true" ports: - port: 6632 protocol: TCP # Mesh: brokers only. - from: - podSelector: matchLabels: app.kubernetes.io/name: queen ports: - port: 6633 protocol: TCP ``` Add a `PodDisruptionBudget` with `minAvailable: 2` if you run three replicas and your nodes get drained; brokers are interchangeable, so the only thing a disruption budget protects is your request capacity. ## Do not couple liveness to the database This is the single most consequential choice in the manifest. `GET /health` is not a static answer. The handler takes a connection from the pool, issues a real query, and returns **503** with `{"status":"unhealthy","database":"disconnected",…}` when that fails. That makes it a correct *readiness* signal: a broker that cannot reach PostgreSQL cannot serve a push or a pop, and should be pulled out of the Service. It makes it a destructive *liveness* signal. Wire `/health` to `livenessProbe` and a PostgreSQL blip becomes a restart of every broker in the cluster, simultaneously. That is worse than doing nothing, for three reasons: 1. **It destroys the mechanism built for exactly this failure.** While PostgreSQL is unreachable, pushes are appended to the on-disk spool and replayed when it comes back. Killing the process mid-outage interrupts that; the spool survives only because it is on disk, and the pod has to come back to the same volume to drain it. 2. **Restarting does not fix an unreachable database.** The new process fails the same probe. You get a `CrashLoopBackOff` across the fleet whose actual cause is one line in the PostgreSQL log. 3. **It makes recovery slower.** When PostgreSQL returns, a broker that stayed up resumes immediately. A broker in backoff waits for its next attempt, then re-applies the schema, then drains its spool before it binds the port. Use `tcpSocket` on the HTTP port for liveness, or omit `livenessProbe` entirely. Either is safe, because the broker's release profile sets `panic = "abort"`: a panic in any of its background loops aborts the whole process after logging a structured error on the `panic` target, so the kubelet restarts the container on its own. There is no half-alive state for a liveness probe to detect. > **Note** > > Probes keep working with authentication enabled. `JWT_SKIP_PATHS` defaults to > `/health,/metrics/prometheus,/metrics,/`, so `/health` is unauthenticated even > with `JWT_ENABLED=true`. If you override `JWT_SKIP_PATHS`, keep `/health` in it or > your probes will start collecting 401s. ## Notes on the rest of the manifest **The uniform peer list includes the pod itself.** A `StatefulSet` template cannot give each ordinal a different value, so the simplest correct configuration lists all three names everywhere. This is safe: an inbound frame's effect is applied to local state and **never re-broadcast**, so a self-connection cannot amplify anything. The cost is one extra connection pair per pod. The alternative, an init script that removes the pod's own name from the list, buys you two TCP connections. **The mesh port is TCP.** It looks like it should be UDP because the environment variables `QUEEN_UDP_PEERS` and `QUEEN_UDP_NOTIFY_PORT` still work as aliases. They are the only survivors of the retired UDP transport; the wire is length-prefixed frames over TCP. A Service or NetworkPolicy declaring `protocol: UDP` for 6633 silently blocks all mesh traffic, and the failure mode is subtle: brokers keep serving correctly, only with pop wakes arriving one backoff interval late. **`podManagementPolicy: Parallel`** is safe here and much faster to roll. Brokers have no startup ordering requirement: the schema apply is serialised cluster-wide by a PostgreSQL advisory lock, so three pods booting simultaneously queue behind that lock rather than racing through the DDL. **`QUEEN_APPLY_SCHEMA` is left at its default** so each broker applies the schema at boot. If your policy forbids DDL from an application role, set it to `"0"` and pre-apply the DDL (see [PostgreSQL](/selfhost/postgres)), and read the mixed-version note in [Upgrades](/selfhost/upgrade) before you roll a version change that way. **No CPU limit is set, deliberately.** The broker's async runtime sizes its worker thread pool from the core count it can see, and it does **not** read the cgroup `cpu.max` quota. A `limits.cpu` of `2` on a 16-core node is a quota, not a narrower CPU set, so the broker still spawns 16 worker threads and oversubscribes its quota 8×, which shows up as scheduler contention, not as a clean throttle. If your platform requires a CPU limit, set it close to the node's core count, or schedule brokers onto nodes sized for them. A memory limit has no such caveat and is set above. **The client Service does not expose 6633.** Keep it that way. There is no reason for a client to reach the mesh port, and every reason for it not to. ## Verify the deployment ```bash kubectl -n queen rollout status statefulset/queen ``` ```bash kubectl -n queen logs queen-0 | grep -E 'schema|config:|listening|mesh' ``` The mesh started when the log carries a `mesh` line reporting `peers` and `auth`. `auth=hmac` means `QUEEN_SYNC_SECRET` arrived; `auth=off` means it did not, and the mesh is in open mode. Then run the push-and-pop check from [Deploy a broker](/selfhost/deploy) through a port-forward: ```bash kubectl -n queen port-forward svc/queen 6632:6632 ``` ## What a chart would need to add Stated so you can judge the gap if you write your own, and so nothing here reads as a claim that a chart exists: - Rendering `QUEEN_MESH_PEERS` from the replica count instead of hard-coding it. - A checksum annotation on the pod template so a Secret change triggers a roll. - Optional Prometheus scrape configuration for `/metrics/prometheus`, which must never be exposed to a tenant on a multi-tenant deployment: its per-queue series sum across tenants. - The JWT variables, and the PostgreSQL TLS variables, as values. - The ingress or gateway resource that terminates TLS, since the broker has no HTTPS listener of its own. ## Related - [High availability](/selfhost/ha) — What the mesh carries, what it does not, and the full reasoning behind firewalling its port. - [Upgrades](/selfhost/upgrade) — Rolling replicas, in-flight leases, and the mixed-version window. - [PostgreSQL](/selfhost/postgres) — Pool sizing across replicas and the privileges the schema apply needs. Source: https://queenmq.com/selfhost/kubernetes/index.mdx