Monitor Laravel jobs: metrics and alerts

The dashboard shows what the supervisors are doing. This page is about what the jobs are doing: how many of each class ran, how many failed and how long they took, which recent jobs carry a tag you care about, which queue has a job that has waited too long, and a Prometheus endpoint an autoscaler can read. All of it needs PHP client 1.7.0 or later; reading the Jobs page with a read-only token needs 2.0.0.

The workers record everything themselves, into the broker’s key/value store. We put it there on purpose: the numbers then cover every worker of every host or pod, whichever engine supervises it, a dashboard on any host reads the same numbers, and there is no Redis to keep and no snapshot command to schedule.

Jobs per class

The dashboard’s Jobs page lists every job class that ran in the window you pick: jobs processed, failed attempts, jobs per minute and the average runtime of an attempt. A supervisor starts its workers with --quiet (a pool’s quiet, on by default), so Laravel prints no line per job: these numbers are the runtimes such a worker leaves.

Each worker of a Queen connection counts these per class in memory and writes them at most once every ten seconds (also while it waits for its next job, and when it stops) to one key of its own, in a five-minute bucket:

jobs/v1/<bucket>/<worker>   {"classes": {"App\\Jobs\\SendInvoice": {"processed": 12, "failed": 1, "runtime_ms": 2600, "max_ms": 900}}}

That is one write per worker every ten seconds, never one per job, and no two workers ever write the same key. With no-op jobs and an in-process broker, recording cost a worker 12 µs of CPU per job, 19 µs with the listeners of monitored tags, beside 148 µs for the rest of Laravel’s work on such a job (Apple M5, PHP 8.4, clients/client-php/benchmarks/job-metrics.php). A write is about 220 bytes for one class and 80 more for each other class; it took 0.7 ms on a broker on the same machine. Keys expire after 25 hours, which bounds the longest window. max_ms, the longest single attempt (PHP client 1.9.0), is not on the Jobs page: the Configuration page’s tuning advice compares it with shutdown_grace. Records of earlier workers have no max_ms, and a class without one is not compared. Recording is best effort: since 1.9.0 it makes one attempt of at most two seconds, with no failover and no retry, and a failed write is dropped without ever reaching the job. A preforked worker starts its own counts after its fork.

An attempt’s runtime runs from Laravel’s JobProcessing event to the end of the attempt: the job’s middleware, its handle(), and the broker ACK when the worker sends it before the next job (ack_batch 1 and ack_async off, the default). Only the jobs of Queen connections are counted, not those of a Horizon worker of the same application.

The dashboard reads them in pages from the window’s first bucket, cached for fifteen seconds. One read takes at most 50,000 keys, a day of buckets for about 170 workers. Past that, the page marks the table as partial, covering the oldest part of the window, and a shorter window shows the rest.

That read uses the supervisor’s read_bearer_token when one is set, and a strictly read-only credential (a read-only JWT, or a proxy key with only the read scope) is enough. Since PHP client 2.0.0 the page lists the keys through POST /api/v1/resources/kv/list, which the broker and its embedded proxy both treat as a read. Only when that route answers 403 or 404 does it fall back to a getPrefix through POST /api/v1/kv, the key/value route that also writes and needs read-write access. Before 2.0.0 the page used getPrefix alone, so a read-only token showed the metrics as unavailable.

Read them from a script

The same route serves any HTTP client, with the same token. This sums the last hour per class, with max_ms, which the Jobs page does not show:

# Every key from the five-minute bucket of an hour ago on: keys sort by bucket.
after="jobs/v1/$(printf '%010d' $(( $(date +%s) / 300 * 300 - 3600 )))"
curl -s -X POST "$QUEEN_URL/api/v1/resources/kv/list" \
  -H "Authorization: Bearer $QUEEN_READ_TOKEN" -H 'Content-Type: application/json' \
  -d "{\"namespace\":\"queen-metrics\",\"prefix\":\"jobs/v1/\",\"after\":\"$after\",\"limit\":1000}" |
jq '[.rows[].value.classes | to_entries[]] | group_by(.key)
    | map((map(.value.processed + .value.failed) | add) as $runs | {
        class: .[0].key, runs: $runs, failed: (map(.value.failed) | add),
        average_ms: ((map(.value.runtime_ms) | add) / $runs | floor),
        max_ms: (map(.value.max_ms // 0) | max)})'

An answer with "truncated": true holds more rows: ask again with after set to its nextAfter. To end a window before now, drop the keys whose bucket is at or past its end. Keys expire after 25 hours, so save a window you want to compare on another day.

Compare with Horizon

Horizon’s Metrics page shows each class’s throughput and average runtime from the snapshots that horizon:snapshot takes, and keeps no history without that schedule. Its runtime runs from the job’s reservation to its deletion, and an attempt that ends released is left out. Compare the averages over the same window, and keep three differences in mind:

• Queen counts every attempt: a released one as processed, one that threw and is retried as failed, both with their runtime. A class that WithoutOverlapping or a rate limiter releases often shows more runs here, and shorter ones.
• Queen’s runtime includes the broker ACK, a few milliseconds, where Horizon’s includes a Redis delete. For short jobs, compare max_ms and the runs too.
• An attempt that outlives its timeout counts here, as failed, since PHP client 2.4.2.

Neither keeps percentiles. For a p50 or a p95, take per-job runtimes: Horizon’s recent jobs carry them, and on Queen a monitored tag that the class’s jobs carry from their tags() method records each job’s runtime for tags.retention_minutes, at one write per job.

Monitored tags

A job is tagged when it is pushed, as in Horizon: by its tags() method when it has one, otherwise with one Model:key tag per Eloquent model it carries. Queued listeners, mailables, notifications and broadcasts are unwrapped to the object that carries the models. The tags travel in the payload, so a worker reads them without unserializing the command. A model is always tagged by its key, even when it has a tags() relation, and a tags() that throws or returns no list is ignored: tagging never fails a dispatch.

class ChargeCustomer implements ShouldQueue
{
    public function __construct(public Customer $customer, public Invoice $invoice) {}

    // Optional: without it, the job is tagged App\Models\Customer:42 and App\Models\Invoice:7.
    public function tags(): array
    {
        return ['billing', 'customer:'.$this->customer->id];
    }
}

On the dashboard’s Tags page, add a tag to monitor. Within thirty seconds every worker records each completed or failed job that carries it, and the page lists the newest ones with their class, queue, attempts and runtime. The list of monitored tags is one key updated with compare-and-swap (the key/value store’s expect), so two operators adding tags at the same moment never overwrite each other. At most 50 tags can be monitored, and a job keeps at most 20 tags of up to 128 bytes. Horizon’s silenced jobs have no equivalent.

Long-wait alerts

queen:check-waits reports a queue whose oldest job has waited longer than its threshold. Schedule it every minute and give each connection:queue its threshold in seconds:

// routes/console.php
Schedule::command('queen:check-waits')->everyMinute();

// config/queen.php
'waits' => [
    'queen:default' => 60,
    'queen:emails' => 300,
],
'notifications' => [
    'mail' => env('QUEEN_NOTIFY_MAIL'),      // comma-separated addresses
    'throttle_minutes' => 5,
],

For each connection:queue it checks every consumer group a supervisor pool consumes that queue with, and the wait it compares is measured, never estimated. When the group has no pending job, nothing waits. Otherwise the broker’s lag for the group is the wait: how long the oldest message the group has not consumed has been in the queue, from GET /api/v1/consumer-groups/lagging. For a queue the group never consumed, it is the age of the oldest pending message, which also covers a queue no worker serves at all.

Above the threshold it dispatches Queen\Laravel\Events\LongWaitDetected (connection, queue, consumer group, seconds, threshold) and, when queen.notifications.mail is set, mails it. Each queue and group notifies at most once per throttle window, and an alert that fails to send is tried again at the next run. Listen to the event to send the alert anywhere else, such as Slack.

Queue sizes

On a Queen connection, Laravel’s size methods read the consumer group’s depth from the broker (GET /api/v1/resources/queues/:queue/depth) and the queue’s timers:

php artisan queue:monitor shows all four and compares size() with its --max, so running and delayed jobs count toward that threshold, as they do on Redis.

Prometheus and Kubernetes autoscaling

With a token set, GET /queen/metrics answers in the Prometheus text format. The scraper sends the token as a bearer token; the route has no session and no Gate, and answers 404 while it is off.

QUEEN_METRICS_ENABLED=true
QUEEN_METRICS_TOKEN=a-secret-of-at-least-32-characters

The endpoint reads what the supervisors publish, so on web pods it needs remote status. With coordinated replicas, scale the supervisor Deployment on the backlog. With KEDA’s Prometheus scaler, for pods of up to 10 workers, each sized for 10 jobs:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: queen-workers
spec:
  scaleTargetRef:
    name: queen-workers        # the supervisor Deployment
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: max(queen_queue_depth{queue="default",consumer_group="laravel"})
        threshold: "100"       # max_processes x target_jobs_per_process

Each new pod joins the coordination and takes its share at its next poll, and a pod removed by a scale-down leaves at once. Scale on Kubernetes has the whole deployment.

The broker dashboard can also inspect the same remote publications on its Supervisors page: one overview per instance, with worker capacity, process budget and pool details in a drawer. Publication groups follow the shared status contract; they do not replace consumer groups or change coordination membership.

The broker has Prometheus metrics of its own, about queues, partitions and the cluster (see monitoring the broker). This endpoint covers the Laravel side: workers, pools and the depth the supervisors see.