Kubernetes

There is no official Helm chart for Queen today. What follows is a complete reference manifest you can apply as-is, written against this broker’s actual behaviour rather than adapted from a generic message-queue template. Every non-obvious field is explained below the YAML, and the two fields most commonly got wrong (the liveness probe and the mesh port) are explained at length, because getting either wrong produces failures that look like something else.

Assumptions: PostgreSQL already exists and is reachable at queen-pg in the same namespace, and you have a namespace called queen.

Why a StatefulSet

A single-replica broker works perfectly well as a Deployment. Once you want more than one, a StatefulSet earns its keep for two reasons that are specific to this broker:

• The mesh peer list is static hostnames. QUEEN_MESH_PEERS is parsed once at boot into (host, port) pairs. A StatefulSet with a headless Service gives every pod a stable DNS name (queen-0.queen-mesh, queen-1.queen-mesh) that you can write into that variable. Pods behind a Deployment have neither stable names nor stable addresses, so there is nothing to put there.
• The disk spool wants to follow the pod. FILE_BUFFER_DIR holds pushes the broker accepted while PostgreSQL was unreachable, and they are only replayed by a broker that starts against the same directory. A volumeClaimTemplate binds one PersistentVolumeClaim per ordinal, so a rescheduled pod finds its own spool again. An emptyDir survives a container restart but not a pod replacement.

If you run one replica and accept that an unreplayed spool dies with the pod, drop the volumeClaimTemplate, mount an emptyDir at the spool path, remove QUEEN_MESH_PEERS, and a Deployment is equivalent. Nothing else changes.

The manifest

Secret

apiVersion: v1
kind: Secret
metadata:
  name: queen-secrets
  namespace: queen
type: Opaque
stringData:
  # The PostgreSQL password for the broker's role.
  pg-password: change-me
  # HMAC secret for the mesh HELLO handshake. Must be byte-identical on every
  # replica. An empty value means "open mode": any well-formed handshake is
  # accepted from anyone that can reach the port.
  sync-secret: change-me-too

Services

Two Services, with different jobs. The client-facing one carries only the HTTP port; the headless one exists so the pods can resolve each other for the mesh.

apiVersion: v1
kind: Service
metadata:
  name: queen
  namespace: queen
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: queen
  ports:
    - name: http
      port: 6632
      targetPort: http
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: queen-mesh
  namespace: queen
spec:
  clusterIP: None
  # Peers must be resolvable before they are Ready: a broker's readiness depends
  # on PostgreSQL, and the mesh should come up regardless of that ordering.
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/name: queen
  ports:
    - name: mesh
      port: 6633
      targetPort: mesh
      protocol: TCP

StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: queen
  namespace: queen
spec:
  serviceName: queen-mesh
  replicas: 3
  podManagementPolicy: Parallel
  selector:
    matchLabels:
      app.kubernetes.io/name: queen
  template:
    metadata:
      labels:
        app.kubernetes.io/name: queen
    spec:
      # A parked long-poll pop is an in-flight request. SIGTERM drains in-flight
      # requests before the process exits, and the default pop timeout is 30 s,
      # so give the drain more than that.
      terminationGracePeriodSeconds: 60
      securityContext:
        # The published image declares no USER, so it runs as root unless you say
        # otherwise. fsGroup is what makes the spool volume writable by a non-root
        # UID.
        runAsNonRoot: true
        runAsUser: 65532
        runAsGroup: 65532
        fsGroup: 65532
      containers:
        - name: queen
          image: ghcr.io/queen-mq/queen:latest
          # The image's own default command, spelled out. WORKDIR is /app.
          command: ["/app/bin/queen"]
          ports:
            - name: http
              containerPort: 6632
              protocol: TCP
            - name: mesh
              containerPort: 6633
              protocol: TCP
          env:
            - name: PG_HOST
              value: queen-pg
            - name: PG_PORT
              value: "5432"
            - name: PG_USER
              value: queen
            - name: PG_DATABASE
              value: queen
            - name: PG_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: queen-secrets
                  key: pg-password
            # replicas x DB_POOL_SIZE must fit inside PostgreSQL's
            # max_connections: 3 x 60 = 180 here.
            - name: DB_POOL_SIZE
              value: "60"
            - name: PORT
              value: "6632"
            # Mesh. The list is uniform across pods and includes this pod itself;
            # see the note below.
            - name: QUEEN_MESH_PORT
              value: "6633"
            - name: QUEEN_MESH_PEERS
              value: "queen-0.queen-mesh:6633,queen-1.queen-mesh:6633,queen-2.queen-mesh:6633"
            - name: QUEEN_SYNC_SECRET
              valueFrom:
                secretKeyRef:
                  name: queen-secrets
                  key: sync-secret
            # Names this replica in mesh heartbeats and peer stats. Taken from the
            # pod name so it is stable and readable in the logs.
            - name: QUEEN_SERVER_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: FILE_BUFFER_DIR
              value: /var/lib/queen/buffers
            - name: LOG_LEVEL
              value: info
            # One JSON object per line, for a structured log shipper.
            - name: QUEEN_LOG_JSON
              value: "true"
          # Do not start counting failures until the broker has applied its
          # schema and reached PostgreSQL once. 30 x 10 s = up to 5 minutes.
          startupProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            failureThreshold: 30
          # /health does a real database round-trip, so this correctly removes a
          # broker that cannot serve from the Service endpoints.
          readinessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          # DELIBERATELY not /health. See "Do not couple liveness to the
          # database" below. This checks only that the listener still accepts.
          livenessProbe:
            tcpSocket:
              port: http
            periodSeconds: 20
            timeoutSeconds: 5
            failureThreshold: 3
          resources:
            requests:
              cpu: "1"
              memory: 512Mi
            limits:
              # Read the note on CPU limits below before setting this.
              memory: 2Gi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: spool
              mountPath: /var/lib/queen/buffers
            # If the spool directory cannot be created the broker falls back to
            # the OS temp directory, which a read-only root filesystem would
            # deny. Mount one so the fallback has somewhere to go.
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}
  volumeClaimTemplates:
    - metadata:
        name: spool
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 5Gi

NetworkPolicy

The mesh port must not be reachable from anywhere except the other brokers. This is a requirement, not hardening. See High availability for why.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: queen
  namespace: queen
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: queen
  policyTypes: ["Ingress"]
  ingress:
    # HTTP: from whatever should be allowed to call the API. Narrow this.
    - from:
        - podSelector:
            matchLabels:
              queen-client: "true"
      ports:
        - port: 6632
          protocol: TCP
    # Mesh: brokers only.
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: queen
      ports:
        - port: 6633
          protocol: TCP

Add a PodDisruptionBudget with minAvailable: 2 if you run three replicas and your nodes get drained; brokers are interchangeable, so the only thing a disruption budget protects is your request capacity.

Do not couple liveness to the database

This is the single most consequential choice in the manifest.

GET /health is not a static answer. The handler takes a connection from the pool, issues a real query, and returns 503 with {"status":"unhealthy","database":"disconnected",…} when that fails. That makes it a correct readiness signal: a broker that cannot reach PostgreSQL cannot serve a push or a pop, and should be pulled out of the Service.

It makes it a destructive liveness signal. Wire /health to livenessProbe and a PostgreSQL blip becomes a restart of every broker in the cluster, simultaneously. That is worse than doing nothing, for three reasons:

1. It destroys the mechanism built for exactly this failure. While PostgreSQL is unreachable, pushes are appended to the on-disk spool and replayed when it comes back. Killing the process mid-outage interrupts that; the spool survives only because it is on disk, and the pod has to come back to the same volume to drain it.
2. Restarting does not fix an unreachable database. The new process fails the same probe. You get a CrashLoopBackOff across the fleet whose actual cause is one line in the PostgreSQL log.
3. It makes recovery slower. When PostgreSQL returns, a broker that stayed up resumes immediately. A broker in backoff waits for its next attempt, then re-applies the schema, then drains its spool before it binds the port.

Use tcpSocket on the HTTP port for liveness, or omit livenessProbe entirely. Either is safe, because the broker’s release profile sets panic = "abort": a panic in any of its background loops aborts the whole process after logging a structured error on the panic target, so the kubelet restarts the container on its own. There is no half-alive state for a liveness probe to detect.

Notes on the rest of the manifest

The uniform peer list includes the pod itself. A StatefulSet template cannot give each ordinal a different value, so the simplest correct configuration lists all three names everywhere. This is safe: an inbound frame’s effect is applied to local state and never re-broadcast, so a self-connection cannot amplify anything. The cost is one extra connection pair per pod. The alternative, an init script that removes the pod’s own name from the list, buys you two TCP connections.

The mesh port is TCP. It looks like it should be UDP because the environment variables QUEEN_UDP_PEERS and QUEEN_UDP_NOTIFY_PORT still work as aliases. They are the only survivors of the retired UDP transport; the wire is length-prefixed frames over TCP. A Service or NetworkPolicy declaring protocol: UDP for 6633 silently blocks all mesh traffic, and the failure mode is subtle: brokers keep serving correctly, only with pop wakes arriving one backoff interval late.

podManagementPolicy: Parallel is safe here and much faster to roll. Brokers have no startup ordering requirement: the schema apply is serialised cluster-wide by a PostgreSQL advisory lock, so three pods booting simultaneously queue behind that lock rather than racing through the DDL.

QUEEN_APPLY_SCHEMA is left at its default so each broker applies the schema at boot. If your policy forbids DDL from an application role, set it to "0" and pre-apply the DDL (see PostgreSQL), and read the mixed-version note in Upgrades before you roll a version change that way.

No CPU limit is set, deliberately. The broker’s async runtime sizes its worker thread pool from the core count it can see, and it does not read the cgroup cpu.max quota. A limits.cpu of 2 on a 16-core node is a quota, not a narrower CPU set, so the broker still spawns 16 worker threads and oversubscribes its quota 8×, which shows up as scheduler contention, not as a clean throttle. If your platform requires a CPU limit, set it close to the node’s core count, or schedule brokers onto nodes sized for them. A memory limit has no such caveat and is set above.

The client Service does not expose 6633. Keep it that way. There is no reason for a client to reach the mesh port, and every reason for it not to.

Verify the deployment

kubectl -n queen rollout status statefulset/queen

kubectl -n queen logs queen-0 | grep -E 'schema|config:|listening|mesh'

The mesh started when the log carries a mesh line reporting peers and auth. auth=hmac means QUEEN_SYNC_SECRET arrived; auth=off means it did not, and the mesh is in open mode. Then run the push-and-pop check from Deploy a broker through a port-forward:

kubectl -n queen port-forward svc/queen 6632:6632

What a chart would need to add

Stated so you can judge the gap if you write your own, and so nothing here reads as a claim that a chart exists:

• Rendering QUEEN_MESH_PEERS from the replica count instead of hard-coding it.
• A checksum annotation on the pod template so a Secret change triggers a roll.
• Optional Prometheus scrape configuration for /metrics/prometheus, which must never be exposed to a tenant on a multi-tenant deployment: its per-queue series sum across tenants.
• The JWT variables, and the PostgreSQL TLS variables, as values.
• The ingress or gateway resource that terminates TLS, since the broker has no HTTPS listener of its own.

Related