Kubernetes

Three replicas, a headless Service so they can find each other, and a client Service carrying only the HTTP port. The manifest below is the deployment in full: apply it with -n queen, against a PostgreSQL at queen-pg. Every value in it that is not obvious is explained underneath, including the three that decide whether the broker performs.

kubectl -n queen create secret generic queen-secrets \
  --from-literal=pg-password=change-me \
  --from-literal=sync-secret=change-me-too

sync-secret is the HMAC secret for the mesh handshake and must be byte-identical on every replica. Empty means open mode: any well-formed handshake from anyone reaching the port is accepted.

apiVersion: v1
kind: Service
metadata:
  name: queen
spec:
  selector:
    app.kubernetes.io/name: queen
  ports:
    - name: http
      port: 6632
      targetPort: http
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: queen-mesh
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/name: queen
  ports:
    - name: mesh
      port: 6633
      targetPort: mesh
      protocol: TCP
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: queen
spec:
  serviceName: queen-mesh
  replicas: 3
  podManagementPolicy: Parallel
  selector:
    matchLabels:
      app.kubernetes.io/name: queen
  template:
    metadata:
      labels:
        app.kubernetes.io/name: queen
    spec:
      terminationGracePeriodSeconds: 60
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        runAsGroup: 65532
        fsGroup: 65532
      containers:
        - name: queen
          image: ghcr.io/queen-mq/queen:latest
          command: ["/app/bin/queen"]
          ports:
            - name: http
              containerPort: 6632
            - name: mesh
              containerPort: 6633
          env:
            - name: PG_HOST
              value: queen-pg
            - name: PG_PORT
              value: "5432"
            - name: PG_USER
              value: queen
            - name: PG_DATABASE
              value: queen
            - name: PG_PASSWORD
              valueFrom:
                secretKeyRef: { name: queen-secrets, key: pg-password }
            - name: DB_POOL_SIZE
              value: "60"
            - name: PORT
              value: "6632"
            - name: QUEEN_MESH_PORT
              value: "6633"
            - name: QUEEN_MESH_PEERS
              value: "queen-0.queen-mesh:6633,queen-1.queen-mesh:6633,queen-2.queen-mesh:6633"
            - name: QUEEN_SYNC_SECRET
              valueFrom:
                secretKeyRef: { name: queen-secrets, key: sync-secret }
            - name: QUEEN_SERVER_ID
              valueFrom:
                fieldRef: { fieldPath: metadata.name }
            - name: FILE_BUFFER_DIR
              value: /var/lib/queen/buffers
            - name: QUEEN_LOG_JSON
              value: "true"
          startupProbe:
            httpGet: { path: /health, port: http }
            periodSeconds: 10
            failureThreshold: 30
          readinessProbe:
            httpGet: { path: /health, port: http }
            periodSeconds: 10
            failureThreshold: 3
          livenessProbe:
            tcpSocket: { port: http }
            periodSeconds: 20
            failureThreshold: 3
          resources:
            requests: { cpu: "1", memory: 512Mi }
            limits: { memory: 2Gi }
          securityContext:
            readOnlyRootFilesystem: true
          volumeMounts:
            - name: spool
              mountPath: /var/lib/queen/buffers
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}
  volumeClaimTemplates:
    - metadata:
        name: spool
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests: { storage: 5Gi }

The probes

/health takes a pooled connection and runs a real query, which makes it the readiness answer: a broker that cannot reach PostgreSQL should leave the Service endpoints. Wire it to livenessProbe and a PostgreSQL outage becomes a kill loop on every replica: the pushes the disk spool exists to absorb are refused at the socket instead, and the spool goes with the container if it sits on the container filesystem. Liveness is a TCP connect on the HTTP port, or nothing: a panic aborts the process, so there is no half-alive state to detect.

The startup probe gives the first boot five minutes: the broker applies its embedded schema before it serves.

Why a StatefulSet

• QUEEN_MESH_PEERS is a list of hostnames parsed once at boot, and only a StatefulSet behind a headless Service hands out stable names to put in it.
• The spool is node-local and replayed only by a broker starting against the same FILE_BUFFER_DIR, which is what one claim per ordinal buys.

For one replica that accepts losing an unreplayed spool: drop the claim template, mount an emptyDir there, remove QUEEN_MESH_PEERS, and a Deployment is equivalent.

The fields that are not obvious

• replicas times DB_POOL_SIZE must fit PostgreSQL’s max_connections: 180 here.
• terminationGracePeriodSeconds must exceed your longest pop timeout, 30 seconds by default, because a parked long-poll pop is an in-flight request that SIGTERM drains.
• Port 6633 is TCP despite the legacy QUEEN_UDP_* alias names, and a Service or NetworkPolicy declaring UDP blocks the mesh silently: brokers keep serving, with pop wakes arriving late.
• The peer list is uniform and names the pod itself, which is safe because a frame is applied to local state and never re-broadcast.
• No CPU limit, but not for the reason usually given: tokio sizes its worker pool from available_parallelism(), which honours the cgroup quota on the toolchain this builds with, so a limit would size the pool correctly rather than oversubscribe it. The cost is CFS throttling. The commit path is bursty, and a quota turns a burst into scheduler latency at exactly the percentile you are watching. Memory has no such caveat and is set.
• podManagementPolicy: Parallel is safe: the schema apply serialises cluster-wide on a PostgreSQL advisory lock.
• Nothing but the other brokers belongs on 6633: keep it off the client Service and close it with a NetworkPolicy (the mesh port).

The three sizing rules

None of these shows up in a kubectl get, and each one is the difference between a broker that runs and a broker that performs.

• PG_PORT points at PostgreSQL directly, not at a pooler. The broker holds session-scoped advisory locks across the boot schema apply and across every retention cycle, plus the statements the hot paths prepare once per connection, and all of that lives on one specific backend session. PgBouncer in session mode is the supported way to pool, and it pins a server connection per client, so it multiplexes nothing for a long-lived application pool anyway: 28 client connections were measured against 28 backends. It can still recycle the server underneath and drop a held lock, which is what pg_advisory_unlock reported not-held in the retention logs was. Full reasoning in PostgreSQL.
• DB_POOL_SIZE has to comfortably exceed QUEEN_ADMISSION_POOL_RESERVE, default 16. The admission arbiter derives its band from DB_POOL_SIZE minus that reserve, so with the default reserve a pool of 24 or less collapses the band to [8,8]: the broker then runs at a fixed eight concurrent write transactions whatever the disk can take, and the AIMD adapter stops adapting. The 60 in the manifest gives a band of [29,44]. If PostgreSQL cannot take replicas × DB_POOL_SIZE, lower QUEEN_ADMISSION_POOL_RESERVE alongside rather than shrinking the pool on its own (flow control).
• Leave QUEEN_STMT_TIMEOUT_MS at its 30000 default unless you have measured a reason. An admission slot is held across the timed call, so a longer deadline means a wedged statement occupies a write slot for that much longer, and the budget the arbiter is trying to hold shrinks by one for the duration.

Verify

kubectl -n queen rollout status statefulset/queen
kubectl -n queen logs queen-0 | grep -E 'schema|config:|listening|mesh'

The mesh line reports peers and auth: auth=hmac means the secret arrived, auth=off is open mode. Every variable above, and the ones left at their defaults, is in configuration.