Laravel worker supervisors that follow the backlog

The supervisor is the part of Horizon that starts queue:work processes, keeps the right number on each queue and stops them cleanly at a deploy. Queen has two: a PHP engine that runs straight from the Composer package, and a Rust engine whose master stayed at 7.0 MiB of resident memory through a 45-minute soak while Horizon’s held 49.1 MiB (benchmark); with prefork, a fork server beside it holds the booted Laravel. Both read the same config/queen.php, publish the same status and obey the same commands, so you can start on one and move to the other without touching a pool.

php artisan queen:supervise                               # the PHP engine

php artisan queen:supervisor-install                      # the Rust engine, installed explicitly
vendor/bin/queen-supervisor --php php --artisan artisan

The workers are ordinary php artisan queue:work queen processes in both cases. The broker never starts PHP: the supervisor is a local process on each worker host, and the broker only answers it.

Requirements

• PHP 8.3 or newer, on Unix. Native Windows is not supported by either engine.
• The pcntl and posix extensions, for process control, the ownership checks of the state directory and the Composer launcher; symfony/process for the PHP engine.
• phar and a working proc_open, to install and smoke-test the native binary.
• An outer process monitor (systemd, Kubernetes, Supervisor) that restarts the master if it exits.

Thread-safe PHP, such as the build FrankenPHP embeds, works too. There chdir() does not move the process, so the launcher runs the verified binary by its absolute path: every check still runs, but the exec is not pinned to the verified inode as it is on ordinary PHP.

The queue connection on its own needs none of this. If another process manager runs a fixed number of workers, you can skip this page.

Prepare a private state directory

Status, the ownership lock, control requests, worker telemetry and the workers’ exit markers share one directory. Create it as the user that runs the supervisor:

install -d -m 0700 /srv/example/queen-supervisor-state

QUEEN_SUPERVISOR_STATE_DIRECTORY=/srv/example/queen-supervisor-state

The supervisor refuses to start on a directory that another user could write to, so do not loosen it to fit a Laravel storage/ directory with mode 0775. It checks ownership, modes, symlinks and replacement of the path, and fails closed. Keep it outside the web root. The native binary has an installation directory of its own, with a different policy, described below.

Configure a pool

Publish config/queen.php and edit supervisor.supervisors. Each entry is a pool, like a Horizon supervisor: a set of workers with one configuration (every key is in the configuration reference). This one gives a pool a budget of 20 workers across two queues:

'supervisor' => [
    'poll_interval' => 3,
    'shutdown_grace' => 75,
    'process_limit' => 20,
    'state_directory' => env('QUEEN_SUPERVISOR_STATE_DIRECTORY', storage_path('queen-supervisor')),
    'supervisors' => [
        'jobs' => [
            'connection' => 'queen',
            'consumer_group' => 'laravel',
            'queues' => ['high', 'default'],
            'balance' => 'auto',
            'strategy' => 'time',
            'min_processes' => 2,
            'max_processes' => 20,
            'min_processes_per_queue' => 1,   // every queue keeps a warm worker
            'fast_scale_up' => true,          // close half the gap per cycle
            'target_clear_seconds' => 60,
            'default_runtime_seconds' => 1,
            'balance_cooldown' => 3,
            'balance_max_shift' => 2,
            'scale_down_delay' => 10,
            'timeout' => 60,
            'retry_after' => 90,
            'tries' => 3,
            'memory' => 128,
        ],
    ],
],

Two fixed workers for slow jobs

For a queue that calls external services, two workers keep execution capacity available while one waits for a response. Put these settings in the published config/queen.php:

'prefetch' => 1,
'ack_batch' => 1,
'pop_ahead' => false,
'lease_renewal' => false,
'supervisor' => [
    'process_limit' => 2,
    'supervisors' => [
        'default' => [
            'connection' => 'queen',
            'consumer_group' => 'laravel',
            'queues' => ['default'],
            'balance' => 'simple',
            'processes' => 2,
            'min_processes' => 2,
            'max_processes' => 2,
            'timeout' => 60,
            'retry_after' => 90,
            'tries' => 3,
        ],
    ],
],

simple keeps this fixed target even with an empty queue. Keep processes, min_processes and max_processes aligned for a fixed pool: processes is clamped to the min/max bounds. The master starts ordinary PHP workers; it does not execute the jobs itself. Each worker claims one job at a time here, so a busy worker holds no prefetched tail. Spare workers can advance on other partitions; jobs in the same partition still wait in order.

The process budget must cover every pool’s maximum and its renewal reservation:

The four-slot budget remains required when the Rust master renews leases on Linux. The helper slot is reserved for fallback; it does not mean four PHP processes normally run. Without renewal, keep the worker timeout and every job’s timeout below retry_after. Turning on renewal keeps leases safe during a long wait; the prefetch tradeoff still applies.

Master settings and startup checks

The keys outside supervisors apply to the whole master. Two of them read an environment variable, named in parentheses; set the others in the file:

The supervisor checks the inequalities that keep a job safe before it starts a single worker, so a mistake shows up at the deploy instead of as a job that runs twice:

• retry_after is longer than the pool’s timeout, and shutdown_grace is longer than every pool’s timeout.
• In auto mode, max_processes covers every queue of the pool.
• The pools’ maxima fit process_limit. A worker with lease renewal reserves two child slots, one for Artisan and one for its renewal helper, even when the master renews the leases, because a worker the master refuses starts a helper.
• The control TTL and the heartbeat timeout exceed one bounded pass of the control loop. With remote status on, that pass includes one request timeout per broker endpoint for the publish, and the publish interval stays below the heartbeat timeout; with coordination on, one request timeout per broker endpoint for every four autoscaling pools.

To see exactly what either engine will run:

php artisan queen:supervisor-config --pretty

That form redacts tokens and header values. --for-engine is the form the Rust engine loads, with its credentials included (the write credential too, because that engine also publishes status and coordinates replicas), so never send it to a log or a build artifact.

Both engines print configuration warnings when they start, such as a pool with tries 1 on a connection with prefetch above 1 or pop_ahead. queen:supervise prints them on its output. queen:supervisor-config prints them on standard error, as Queen warning: … lines, and the Rust master passes that standard error to its own log while it reads the document from standard output.

How a pool scales

We measured that run on 2.6.0 with balance auto, target_jobs_per_process 10 and fast_scale_up: five workers three seconds after the dispatch, eight at nine seconds, the queue empty at fifteen, and the extra workers given back one at a time after scale_down_delay. The Rust engine gave the same curve within a second.

In auto mode the master reads the backlog of each queue every poll_interval and sizes the pool between min_processes and max_processes. The backlog is the consumer group’s effectivePending from the broker’s depth read: every job not yet acknowledged, running ones included, and no delayed job until its timer fires. The size strategy aims at one worker per target_jobs_per_process jobs of that backlog. The time strategy multiplies the backlog by the job runtime the workers report and aims to clear it within target_clear_seconds, using default_runtime_seconds until the first reports arrive. A pool changes by at most balance_max_shift workers per balance_cooldown, and a lower target has to hold for scale_down_delay before workers drain. A draining worker still counts against process_limit, helper included, so a reallocation never makes a burst of processes. Both engines send a pool’s depth reads in waves of at most 16 concurrent requests.

Every entry in supervisors is its own pool, so one application can mix workers dedicated to a queue with workers that move between queues, as Horizon supervisors do:

'supervisors' => [
    'payments' => ['queues' => ['payments'], 'balance' => 'simple', 'processes' => 2],
    'shared'   => ['queues' => ['emails', 'default', 'reports'], 'balance' => 'auto',
                   'min_processes' => 1, 'max_processes' => 20, 'min_processes_per_queue' => 1],
],

min_processes bounds a whole pool. min_processes_per_queue, Horizon’s per-queue minProcesses, keeps that many workers on every queue of an auto pool even with no backlog, so the first job on a quiet queue finds a worker already running. The floor for every queue has to fit max_processes.

By default each cycle adds at most balance_max_shift workers, one unless you raise it, as in Horizon: a burst that needs twenty workers takes nineteen cycles. With fast_scale_up, each cycle closes half of the remaining gap, and the same burst takes five. Scaling down keeps its step and its scale_down_delay.

Event-driven scaling

A pool that polls reacts at the pace of its cycle: three seconds to notice a burst, and three more for every step. With event_driven, the broker wakes the supervisor when jobs arrive:

// config/queen.php
'supervisor' => [
    'event_driven' => true,
    // ...
],

The supervisor holds a read-only long poll, POST /api/v1/fetch, over the stripes of every pool that follows the backlog, <partition_prefix>-0000 up to <partition_prefix>-<partitions - 1>. Every 2.0 broker serves it. A fetch reads a partition by offset: it takes no lease and moves no cursor, so it never takes a job from a worker, and the broker answers it as soon as one of the watched partitions grows. One fetch watches at most 1,024 stripes per connection, counting every queue of every pool on that connection: with more, the supervisor logs it at start, watches the first 1,024, and the regular poll finds jobs on the others.

When a watched partition grows, the pool reads its depth and adds workers at once, inside balance_cooldown too, and keeps growing every second while it is below its target. No queue loses a worker in such an early step, and scaling down keeps its cooldown and scale_down_delay.

A pool is woken at most once a second and not at all at max_processes, so a busy queue costs at most one fetch and one depth read per second. A job pushed to a partition of its own (QueenPartitionable) is found by the regular poll, as before, and a queue that does not exist yet is probed again every 30 seconds. The answer carries the first new job of each grown stripe, payload included, which the supervisor reads and throws away; an answer over 8 MiB is not read at all and wakes every watched queue. The Rust engine parks in a thread for up to 20 seconds at a time, the PHP engine for up to one second inside its loop.

On the Linux server, with event_driven and fast_scale_up, a pool of 1 to 64 workers reached 64 workers 8.2 s after a burst of 30,000 jobs began, and finished the jobs, dispatch included, after 9.5 s. Horizon took 37.0 s, and its dispatch was too slow for it to need more than about 34 workers (benchmark).

The long poll uses the token the supervisor reads depth with: read_bearer_token when you set one, the connection’s token otherwise. On the broker’s own JWT the fetch is a read, so a read-only token passes; through the embedded proxy it carries the authority of a pop, so there the token must be allowed to consume. When the broker refuses the long poll, the supervisor logs it once and keeps polling. Event-driven scaling needs PHP client 1.8.0 (supervisor 0.5.0).

Run the PHP engine

php artisan queen:supervise

It runs from the Composer package, with no native binary. A worker that exits non-zero shortly after it started is restarted after a capped exponential backoff; after five failures in a row the pool’s circuit opens, and after restart_backoff_max it lets one probe worker through until that worker has run for stable_after. The Rust engine does the same.

Since PHP client 1.9.0, three exits restart at once in both engines, without backoff: a worker that ran for stable_after, one that Laravel killed after a job timeout, and one that stopped at --memory after handling a job. The last two leave a marker in the state directory before they exit, which is how the master tells them from a crash. Every other non-zero exit counts as a failure, including a SIGKILL without that marker, such as the OOM killer’s. Before 1.9.0 a burst of job timeouts could open the circuit and hold a healthy pool at one worker.

Since PHP client 2.4.4 and supervisor 0.8.2, those three exits and a worker that stopped for queue:restart are also replaced outside balance_max_shift and whatever the circuit says, up to the size the pool had: a pool that loses most of its workers at once, as at a deploy, is refilled in one cycle, and a worker that stops at --max-time while a probe runs is replaced at once. Any other exit, even with status 0, waits for the budget and the circuit as before: Laravel’s worker exits 0 when it loses its database connection, every second through an outage.

Laravel’s worker catches what a pop throws, reports it, sleeps a second and pops again, so a worker whose every pop fails stays alive and looks like capacity. Since PHP client 2.4.2 a worker started by either engine tells the two cases apart. When its loop works another connection than its pool’s, or a pop throws a LogicException (an InvalidArgumentException included), no retry can help: the worker throws WorkerCannotConsume from its next loop, which reaches the application’s exception handler, and exits 1, a crash like any other. A job whose own $timeout its lease cannot cover no longer stops the worker since PHP client 2.4.4: that job fails alone, as a migration note explains. When the broker is down, slow or refusing, a restart cannot help either, and restarting every worker would turn a short outage into a restart storm: the worker stays, and leaves a small file in the state directory saying since when its pops fail. A pop that works, empty or not, clears the failure of its own queue, and the file goes once no queue fails, so a worker of --queue=high,low whose every pop of low fails is reported even while high answers (PHP client 2.4.4). A pop that failed because the broker reset the connection no longer makes Laravel stop the worker as if it had lost its database (2.4.4). queen:supervisor status reads the file; see below.

Run the Rust engine

The Composer package carries a launcher; the binary for your platform is installed in a separate, explicit step, so Composer never downloads and runs native code on its own:

install -d -m 0755 /srv/example/queen-supervisor-bin
export QUEEN_SUPERVISOR_INSTALL_PATH=/srv/example/queen-supervisor-bin
php artisan queen:supervisor-install
vendor/bin/queen-supervisor --php php --artisan artisan

The launcher reads QUEEN_SUPERVISOR_INSTALL_PATH from its own environment, not from .env, so the service that starts it needs the variable too.

The installer picks the version the package pins and the host’s target, verifies the release manifest, the archive’s SHA-256, the executable’s version and a local receipt, then publishes the binary by atomic rename. The launcher checks the receipt and the binary’s hash again before every start and then replaces itself with the binary, so the Rust master is the process your service manager signals. The installation leaf’s parent has to be a real directory (a symlink is refused), and an existing leaf must not be group- or world-writable, which is why it lives apart from Laravel’s storage/.

The user that runs the launcher must also own the installation, each of its directories and files, so install as that user. If root installs and www-data runs the supervisor, the launcher exits with code 70, and the message names the path, its owner, the launcher’s user and the ways out. Since PHP client 2.4.2, root can install for another user, by name or numeric uid:

php artisan queen:supervisor-install --owner=www-data

The installer installs and verifies as root, then gives each directory and file of the installation to that user, with the modes a normal install gives. It refuses --owner without root and on a thread-safe PHP, such as FrankenPHP’s. As root, it does not update an installation that already belongs to another user: run it as that user. chown -R www-data on the installation directory also works. For container images, see build the image.

PHP client 2.4.3 to 2.5.0 pin supervisor 0.8.1, and 2.3.0 to 2.4.2 pin 0.8.0. Its GitHub release, supervisor/v0.8.1, publishes static (musl) Linux builds and macOS builds for amd64 and arm64, with a manifest and its Sigstore bundle, so the online install works on those hosts. On any other host the installer fails closed; run the PHP engine there.

Without QUEEN_SUPERVISOR_INSTALL_PATH, the binary goes to storage/queen-supervisor-bin, which works only where storage/ is a real directory and not a symlink. For a host without network access, the --manifest option or supervisor_binary.manifest in config/queen.php names the manifest as an HTTPS URL or a local file, and the --archive option gives the local archive, which needs a local manifest:

php artisan queen:supervisor-install \
  --manifest=/srv/mirror/queen-supervisor-manifest.json \
  --archive=/srv/mirror/queen-supervisor-0.8.1-linux-amd64.tar.gz

A mirror can serve both over HTTPS from one base URL (QUEEN_SUPERVISOR_RELEASE_BASE_URL), and a deployment can pin the SHA-256 of a manifest it has already verified against its Sigstore bundle (supervisor_binary.manifest_sha256). The installer does not check the Sigstore bundle itself.

The full qualification of the Linux builds as release targets is still pending. The macOS builds are not yet signed or notarized. Windows has no build: the process and locking backends are not written.

Under systemd, run one foreground master and let systemd restart it:

[Service]
Type=simple
User=app
Group=app
WorkingDirectory=/srv/queen-app/current
UMask=0077
ExecStart=/srv/queen-app/current/vendor/bin/queen-supervisor --php /usr/bin/php --artisan artisan
Restart=always
RestartSec=2
KillMode=mixed
SendSIGKILL=yes
TimeoutStopSec=90

KillMode=mixed sends the first SIGTERM to the master alone, so the master drains each worker without killing its renewal helper early, and systemd still SIGKILLs the whole cgroup at the deadline. Keep TimeoutStopSec above shutdown_grace. For the PHP engine, the ExecStart is /usr/bin/php artisan queen:supervise. In a container whose entrypoint is the supervisor, the master is PID 1, and the processes that are orphaned come to it: a renewal helper after a forced kill, or a process a job started and left behind. Since PHP client 2.4.4 and supervisor 0.8.2 a master that is PID 1 reaps, on Linux, every zombie child it does not track, so a pod needs no init process; before, run one (docker run --init, Compose init: true, tini), or the zombies use up the container’s pids. A forked worker outlives a fork server the OOM killer took; a master that is PID 1 reaps that worker itself when it exits (PHP client 2.4.1).

Inspect and control either engine

The commands talk to the state directory, so they are the same for both engines:

php artisan queen:supervisor status
php artisan queen:supervisor status --json
php artisan queen:supervisor status --check
php artisan queen:supervisor status --check-capacity
php artisan queen:supervisor status --check-liveness
php artisan queen:supervisor pause
php artisan queen:supervisor continue
php artisan queen:supervisor terminate

Run them as the supervisor’s user: the state directory is private to it, and the commands refuse any other user, root included, without reading or writing anything. Since PHP client 2.4.2 the refusal names the state’s owner and the user asking and shows how to run as the owner, such as su -s /bin/sh www-data -c "php artisan queen:supervisor status --check" from root, or securityContext.runAsUser on a Kubernetes container (probes).

status --check exits non-zero unless the current generation is live and every pool is ready, which means its depth sample is current and it has workers whenever it wants some. A pool with surviving workers stays ready while a replacement waits in backoff or probe, and the degraded circuit shows in the pool’s health. --check-liveness checks only the master’s owner lock and heartbeat, which is what a liveness probe should ask. --check-capacity is the strict one: every pool runs at least as many workers as it wants, with a healthy restart circuit. It can be false for a moment during a normal scale-up, which is why it is separate from readiness. In status --json, each entry of pool_status carries capacity_satisfied, so a scale-up that lags shows there without making readiness flap.

A status the master cannot write, as on a full disk, no longer stops it at once (PHP client 2.4.4, supervisor 0.8.2): the workers keep running, the write is retried every pass, and the probes see the status grow stale. The master stops once the writes have failed for heartbeat_timeout, when every reader already calls it stale. Before, one failed write drained every worker.

A live worker whose pops have all failed for 60 seconds is not consuming (PHP client 2.4.2). A pool in which every running worker is in that state is not ready, and one with any such worker is not at full capacity; a shorter outage, such as a broker failing over, reports nothing, and the issues clear on their own at the first pop that works. Since PHP client 2.4.3 the 60 seconds count from the start of the first pop that failed. A broker that stops answering fails a pop only when its request times out, 35 seconds for each broker URL the client tries, so its workers show as not consuming as soon as that first pop gives up, about 105 seconds in with three URLs, not a minute later. status --json lists, on each entry of pool_status, every worker whose pops have failed since some moment, however short the time, in not_consuming (pid, seconds, failures, error), with the longest in not_consuming_seconds (null when there is none), so a monitor can apply a threshold of its own. These two fields come from the reader on the supervisor’s host, not from the status document the engines publish.

The issues status --json names are stable codes, and applications may filter on them:

--check fails on any readiness issue, and --check-capacity on any issue of either kind. Status also records the host name, which is how the dashboard tells published instances apart. Updated publishers also record the generation’s start, monotonic uptime at the heartbeat, engine version and installed client version when available. These fields are optional, so earlier publishers remain readable. The Rust engine requests client metadata during its internal configuration export; ordinary exports retain the older shape.

With remote status enabled, the broker’s Supervisors page lists each instance under its publication group and opens its queue/pool details in a drawer. The status contract defines the group and instance keys. Controls still run on the supervisor’s host.

Pause drains the current workers and starts none until continue; it never leaves a PHP process suspended while it holds prefetched jobs. Terminate sends SIGTERM to every worker and, after shutdown_grace, kills what is left of the process groups. Since PHP client 1.9.0 the workers get that SIGTERM before the master publishes its final status or leaves the coordination, so a slow broker cannot eat into the drain. Every control carries the exact instance_id of the master it targets, so a command left over from a previous generation never applies to its replacement.

Readiness is not a service level. Watch the age of the oldest job, completions, failures and the growth of the dead-letter queue separately (monitoring). And leave headroom in the PID limit of the unit or container: process_limit counts the supervised children, not the master, the init process or processes that job code starts.

Prefork workers

A worker started on its own boots Laravel: the autoloader, the configuration, every service provider, and its own compiled copy of the framework, in its heap or, with the command-line opcache on, in opcache memory of its own. With prefork, the master starts one fork server that boots Laravel once, opens no queue, database or cache connection, and forks every worker from itself. The workers share the opcache’s compiled code outright and the booted heap copy-on-write, and a new worker is ready in about half a millisecond: on the Linux server a fork took 0.48 ms, where a worker started on its own took 88 ms to boot the benchmark application, 174 ms with the command-line opcache on (dataset).

QUEEN_SUPERVISOR_PREFORK=true

On the Linux server, 64 forked workers with the opcache on held 1.7 MiB of private memory each after 60,000 jobs of 10 ms, against 28.9 MiB for a Horizon worker and 31.1 MiB for a Queen worker started on its own, and the container used 225 against 1,967 MiB (benchmark). The memory a job allocates stays private to its worker, so prefork saves the booted framework, about 27 MiB per worker there, not a ratio; how it works explains why.

Size a pod by the container’s memory, or by each worker’s private memory or proportional set size, not by the resident set that ps shows, which counts every shared page again in each worker. Measure after a day of real jobs before you lower the limits: a worker’s private memory grows with the pages it writes and with the largest job it has run.

A forked worker runs queue:work with exactly the arguments and environment a spawned one gets and leads its own session, so it is drained, restarted, fenced and counted like any other. The fork server is a fence too: when its master is gone it SIGKILLs every worker it forked before it exits. It learns that from its stdin closing, or with the Rust engine from SIGTERM, set as its parent-death signal. A SIGTERM or SIGINT while the master lives, such as systemd stopping the unit or Ctrl-C in a terminal, leaves the workers to the master, which drains them. If the fork server cannot start, or a fork fails or times out, the supervisor logs it and spawns workers from then on. The fork server stays open for the workers it already forked, and a worker forked after its request timed out is stopped, because no pool counts it. How a forked worker talks to its master lists every channel between them.

A service provider that opens a connection while Laravel boots would share it across every fork. Before the first fork, the fork server closes the database and Redis connections, log channels and mailers the boot opened, and each child closes them again. It warns about any other socket that stays open (queen:fork-server: 1 socket(s) the boot opened stay open (fd 7)): open that connection lazily, when a job first needs it. On Linux the fork server refuses to serve when the booted application runs another thread that does not end within 5 s, names the threads, and the master spawns its workers instead.

php artisan queue:restart stops every worker after its job. A forked worker says it stopped for the restart signal, and the master then starts a new fork server, so the workers forked from then on boot nothing and run the deployed code; the old server closes with the last worker it forked. A worker forked between the restart and that first exit leaves at once with the restart marker, and a master whose fork server had failed to boot, or to fork, starts one again at the next queue:restart (PHP client 2.4.1). A boot can block the master for up to a minute, so after a boot that failed the next one waits 60 seconds, doubling to 15 minutes, and workers are spawned meanwhile (PHP client 2.4.4, supervisor 0.8.2). A change to config/queen.php still needs the master restarted (php artisan queen:supervisor terminate, then your process manager starts it again), as it does without prefork. Prefork needs PHP client 1.7.0 (supervisor 0.4.0); the restart and the checks above need PHP client 2.1.0.

The prefork example prints the process tree and each worker’s private memory, forked and spawned.

Prefork for some pools only

A pool’s own prefork wins over the switch (PHP client 2.3.0, supervisor 0.8.0). Turn it off for the pool whose jobs must not run in a forked process, and keep the others forked:

// config/queen.php, under 'supervisor'
'prefork' => true,
'supervisors' => [
    'default' => [
        'queues' => ['default'],
    ],
    'kafka' => [
        'queues' => ['kafka'],
        'prefork' => false,
    ],
],

• false spawns the pool’s workers: each boots Laravel on its own, as without prefork.
• true forks them even when supervisor.prefork is off; the master starts the fork server for that pool alone.
• No key, or null, follows supervisor.prefork.
• The fork server still boots the whole application. On Linux, if the boot starts a thread, for example a Kafka client that a service provider creates, the fork server refuses to serve and every pool spawns. Create such a client when a job first needs it: the boot then stays fork-safe, and the pool that uses the client can spawn while the others fork.

The dashboard’s prefork setting names the pools that differ from the switch.

When not to use prefork

Every worker holds what the fork server’s boot holds. Leave prefork off when:

• The boot starts a thread or keeps a connection you cannot make lazy. gRPC and Kafka extensions (ext-grpc, ext-rdkafka) and some APM agents start threads while PHP or Laravel boots, and fork() copies only the thread that calls it, so each worker would lack those threads and any lock they held. The fork server refuses to fork then and the master spawns workers, which works but leaves the log warning on every start; turn prefork off instead. If only some jobs need such a client and the boot does not create it, turn prefork off for their pool alone (prefork for some pools only). A socket the boot opens is shared by every worker, and two workers that use it corrupt each other’s conversation.
• Your deploy cannot restart the master and runs Laravel before 12. Laravel 12 tells the worker why it stops. Before it, a forked worker’s exit at queue:restart looks like any other, so its replacement comes from the old fork server with the old code. Run php artisan queen:supervisor terminate at every deploy, or leave prefork off.
• The host has no pcntl or posix. The supervisors need both anyway. On macOS, which has no /proc, the thread check does not run.

Memory tools that sum each process’s resident set overstate forked workers, but that is a reason to read the container’s memory, not to leave prefork off.

Opcache and preload

Opcache is off on the command line by default (opcache.enable_cli=0), and for good reason without prefork: each command-line process creates opcache memory of its own, so every spawned worker holds more with it on. On the Linux server a Horizon worker held 35.9 MiB of private memory with it on and 28.9 MiB with it off. With prefork, the fork server fills the opcache once and every worker shares it, so turning it on saves memory instead: 1.7 MiB per forked worker with it on, 6.2 MiB with it off.

With prefork, enable it for the command-line PHP that runs the workers (php_binary); without prefork, leave it off. php -i | grep opcache.enable_cli shows what the workers get.

; conf.d/opcache-cli.ini, read by the command-line PHP
opcache.enable=1
opcache.enable_cli=1

opcache.preload compiles and loads a list of files when PHP starts, and on the command line PHP starts again in every process. With prefork the fork server preloads once and every worker inherits it; without it every spawned worker preloads on its own. Preload needs opcache.enable_cli=1 and, when PHP runs as root, opcache.preload_user. Preloaded code changes only when the process restarts, which a deploy already does through terminate. Preload was not part of the benchmark.

Lease renewal in the master

With lease renewal on, something has to renew each worker’s lease while its job runs. On Linux the Rust master does it itself, so no worker starts a PHP helper (PHP client 1.9.0, supervisor 0.6.0). The master listens on lease.sock in its state directory and hands the path to every worker of a pool with lease_renewal as QUEEN_SUPERVISOR_LEASE_SOCKET. Each worker connects once and sends its client settings and its renewal timing, and one master thread per worker runs the helper’s algorithm under the same rules: a renewal must be able to finish before the deadline less the safety margin; when it cannot, the master sends the worker SIGTERM and then SIGKILL after the kill grace; and when the connection breaks while a lease is held, it kills the worker at once.

The master accepts only processes of its own user, identifies each worker from the kernel’s socket credentials and signals it through a pidfd, so a recycled PID never receives a signal meant for someone else. On the Linux server, the application of 32 workers and their supervisor used 417 MiB with a renewal helper per worker and 116 MiB with renewal in the master (benchmark).

The master also covers a worker that crashes with prefetched jobs. A worker with prefetch above 1 journals, next to lease.sock, what its shutdown would hand back: one file with the transaction for its leased batch, written once per batch, and a small record of which jobs it still owes, rewritten before each ACK or release. When the worker dies holding its lease without a shutdown, the master sends that transaction while the lease is still the worker’s. The jobs that never started go back to their partitions without an extra attempt, the job that was running counts its run, and none of them waits for retry_after. Every ACK in the transaction names the lease, so the broker refuses the whole transaction once the lease is no longer the worker’s, and the master sends it only once. The master logs each hand-back, and the worker or the master removes the journal when the worker exits. The journal holds the batch’s payloads, in a directory only the supervisor’s user can read. Prefetch says which crashes are still charged an attempt.

A worker that cannot reach the master, or that the master refuses, logs why and starts its own helper, so renewal never stops. The PHP engine and macOS always use helpers, and a helper hands back a crashed worker’s journal the same way, from a private directory under the system’s temporary directory (queen-hand-back- and a random name). To use helpers under the Rust supervisor too, turn the master’s lease service off:

// config/queen.php
'supervisor' => [
    'lease_service' => false,
    // ...
],

On Linux, with a pool that renews leases, the master then logs:

lease renewal: off in the master, queen.supervisor.lease_service is false: one helper process per worker

Up to PHP client 2.0.0 the master read QUEEN_SUPERVISOR_LEASE_SERVICE from its environment instead; supervisor 0.7.0, pinned by PHP client 2.1.0, reads only the key.

Several replicas

A supervisor sizes its autoscaling pools from the whole backlog of their queues. Two masters that do this on their own, on two hosts or two pods, each reach for the full target, so together they run more workers than you planned. Turn coordination on in every replica and they share one target instead:

QUEEN_SUPERVISOR_COORDINATION=true

On every poll each replica renews its key, coordination/v1/<scope>/<instance_id>, in the broker’s key/value store, and lists the other replicas of the same scope in the same call. The key’s TTL is the control-loop bound, at most the heartbeat timeout. The scope is the broker endpoints, the consumer group and the set of queues: replicas coordinate when all three match, even while their names or limits differ during a rollout. The backlog then sizes one fleet target, each replica runs an even share of it, and the remainder goes to the replicas whose instance ids sort first. When the target is smaller than the number of queues with a backlog, replicas cover different queues. Both engines use the same keys and rule, so PHP and Rust replicas coordinate with each other. On the Linux server, two coordinated masters on one queue completed every job once, on both engines, when one of them was killed with SIGKILL during 40 jobs and when the two restarted one after the other during 60 jobs (benchmark).

• min_processes and max_processes apply to each replica: N replicas run between N times min_processes and N times max_processes, and you change N with the Deployment’s replica count or an autoscaler.
• Fixed pools (balance simple) are not split; every replica runs its processes.
• A replica that pauses or stops deletes its keys, and the others take its share at their next poll. A replica that crashes still counts until its key expires, so capacity can be short by its share for up to one control-loop bound.
• This is capacity sharing, not leader election. Replicas read the list at slightly different moments, so the shares match the target within a poll or two, and a replica that cannot reach the broker keeps its last list for one TTL and then sizes its pools alone: more workers than the target, never fewer.
• One call lists up to 100 replicas for each of four pools, so the broker’s QUEEN_KV_MAX_KEYS_PER_CALL must stay at 404 or more (1,024 by default).
• Registering is a key/value write, so it uses the connection’s write credential, never read_bearer_token.

Status reports replicas for each coordinated pool, and the dashboard shows each pool’s share. It warns only about masters that autoscale the same queue without coordinating. Coordination needs PHP client 1.7.0 (supervisor 0.4.0).

The rule counts masters: one master can run as many workers as process_limit allows. Several broker endpoints (QUEEN_URLS) give the supervisor failover for its depth reads; they do not make two masters safe.

Before production

1. Keep prefetch and ack_batch at 1 until a workload is short, idempotent and measured.
2. Turn on lease_renewal whenever prefetch is above 1, or when a job’s runtime cannot fit safely inside the lease.
3. Give the failed-jobs lock a cache store every worker host can see, when failed jobs can be retried or pruned from more than one process or host.
4. Monitor queen:supervisor status --check and the outer process monitor.
5. Send terminate at each deploy, so a fresh master loads the new code and configuration.
6. Keep jobs idempotent: delivery is at least once, and no supervisor makes an external side effect exactly once.
7. Rehearse a graceful and a forced shutdown with your longest real job before the rollout.

Next, enable the dashboard, or put these features together for Kubernetes.