---
title: "Monitor Laravel jobs: metrics and alerts"
description: "Throughput, failures and runtime per job class from every worker of every host, monitored tags, long-wait alerts measured by the broker, and a Prometheus endpoint an autoscaler can read."
---

> Queen MQ documentation, for AI agents
> Complete self-contained summary of Queen MQ: https://queenmq.com/llms-brief.txt
> Fetch that first when the question is about the product rather than about this page.
> Index of all pages: https://queenmq.com/llms.txt

# Monitor Laravel jobs: metrics and alerts

The [dashboard](/integrations/laravel/dashboard/) shows what the supervisors are doing. This page is about
what the jobs are doing: how many of each class ran, how many failed and how long they took, which
recent jobs carry a tag you care about, which queue has a job that has waited too long, and a
Prometheus endpoint an autoscaler can read. All of it needs PHP client 1.7.0 or later; reading the
Jobs page with a read-only token needs 2.0.0.

**Why:** a queue that falls behind is quiet until someone complains. Metrics per job class, monitored tags and an alert on how long the oldest job has waited turn that into a page before the complaint.

The workers record everything themselves, into the broker's [key/value store](/writing-applications/kv-locks-timers/). We
put it there on purpose: the numbers then cover every worker of every host or pod, whichever engine
supervises it, a dashboard on any host reads the same numbers, and there is no Redis to keep and no
snapshot command to schedule.

| | Horizon | Queen |
| --- | --- | --- |
| Per-job metrics | snapshots taken by the scheduled `horizon:snapshot` | live, from every worker, over the last hour, six hours or day |
| Tags | automatic and `tags()`, monitored from the dashboard | automatic and `tags()`, monitored from the dashboard |
| Long-wait alerts | estimated from the backlog and recent runtimes | the broker's age of the oldest waiting job, per consumer group |
| Alert routes | mail, Slack, SMS | the `LongWaitDetected` event, and mail |
| Autoscaler metrics | not built in | a Prometheus endpoint for KEDA or an HPA |

## Jobs per class

The dashboard's **Jobs** page lists every job class that ran in the window you pick: jobs
processed, failed attempts, jobs per minute and the average runtime of an attempt. A supervisor
starts its workers with `--quiet` (a pool's `quiet`, on by default), so Laravel prints no line per
job: these numbers are the runtimes such a worker leaves.

Each worker of a Queen connection counts these per class in memory and writes them at most once
every ten seconds (also while it waits for its next job, and when it stops) to one key of its own,
in a five-minute bucket:

```text
jobs/v1/<bucket>/<worker>   {"classes": {"App\\Jobs\\SendInvoice": {"processed": 12, "failed": 1, "runtime_ms": 2600, "max_ms": 900}}}
```

That is one write per worker every ten seconds, never one per job, and no two workers ever write
the same key. With no-op jobs and an in-process broker, recording cost a worker 12 µs of CPU per
job, 19 µs with the listeners of monitored tags, beside 148 µs for the rest of Laravel's work on
such a job (Apple M5, PHP 8.4, `clients/client-php/benchmarks/job-metrics.php`). A write is about
220 bytes for one class and 80 more for each other class; it took 0.7 ms on a broker on the same
machine. Keys expire after 25 hours, which bounds the longest window. `max_ms`, the longest
single attempt (PHP client 1.9.0), is not on the Jobs page: the Configuration page's
[tuning advice](/integrations/laravel/dashboard/#tuning-advice-and-settings) compares it with
`shutdown_grace`. Records of earlier workers have no `max_ms`, and a class without one is not
compared. Recording is best effort: since 1.9.0 it makes one attempt of at most two seconds, with
no failover and no retry, and a failed write is dropped without ever reaching the job. A preforked worker starts its
own counts after its fork.

An attempt's runtime runs from Laravel's `JobProcessing` event to the end of the attempt: the job's
middleware, its `handle()`, and the broker ACK when the worker sends it before the next job
(`ack_batch` 1 and `ack_async` off, the default). Only the jobs of Queen connections are counted,
not those of a Horizon worker of the same application.

| The attempt ends when | It counts as |
| --- | --- |
| `handle()` returns | processed |
| the job released itself, or middleware such as `WithoutOverlapping` released it | processed |
| it throws, whether Laravel retries it or not | a failed attempt |
| the job calls `$this->fail()`, or a delivery is past its tries | a failed attempt (`$this->fail()` counted as processed before PHP client 2.4.2) |
| it outlives its timeout | a failed attempt, of about the timeout, written before Laravel kills the worker (not recorded before PHP client 2.4.2) |

| Key of `config/queen.php` | Default | Meaning |
| --- | --- | --- |
| `job_metrics.enabled` | `true` | record per-class metrics |
| `job_metrics.connection` | `queen` | the Queen connection whose broker stores them |
| `job_metrics.namespace` | `queen-metrics` | the key/value namespace |

The dashboard reads them in pages from the window's first bucket, cached for fifteen seconds. One
read takes at most 50,000 keys, a day of buckets for about 170 workers. Past that, the page marks
the table as partial, covering the oldest part of the window, and a shorter window shows the rest.

That read uses the supervisor's `read_bearer_token` when one is set, and a strictly read-only
credential (a `read-only` JWT, or a proxy key with only the `read` scope) is enough. Since PHP
client 2.0.0 the page lists the keys through `POST /api/v1/resources/kv/list`, which the broker and
its [embedded proxy](/manage/tenants/) both treat as a read. Only when that
route answers `403` or `404` does it fall back to a `getPrefix` through `POST /api/v1/kv`, the
key/value route that also writes and needs read-write access. Before 2.0.0 the page used
`getPrefix` alone, so a read-only token showed the metrics as unavailable.

### Read them from a script

The same route serves any HTTP client, with the same token. This sums the last hour per class,
with `max_ms`, which the Jobs page does not show:

```bash
# Every key from the five-minute bucket of an hour ago on: keys sort by bucket.
after="jobs/v1/$(printf '%010d' $(( $(date +%s) / 300 * 300 - 3600 )))"
curl -s -X POST "$QUEEN_URL/api/v1/resources/kv/list" \
  -H "Authorization: Bearer $QUEEN_READ_TOKEN" -H 'Content-Type: application/json' \
  -d "{\"namespace\":\"queen-metrics\",\"prefix\":\"jobs/v1/\",\"after\":\"$after\",\"limit\":1000}" |
jq '[.rows[].value.classes | to_entries[]] | group_by(.key)
    | map((map(.value.processed + .value.failed) | add) as $runs | {
        class: .[0].key, runs: $runs, failed: (map(.value.failed) | add),
        average_ms: ((map(.value.runtime_ms) | add) / $runs | floor),
        max_ms: (map(.value.max_ms // 0) | max)})'
```

An answer with `"truncated": true` holds more rows: ask again with `after` set to its `nextAfter`.
To end a window before now, drop the keys whose bucket is at or past its end. Keys expire after 25
hours, so save a window you want to compare on another day.

### Compare with Horizon

Horizon's Metrics page shows each class's throughput and average runtime from the snapshots that
`horizon:snapshot` takes, and keeps no history without that schedule. Its runtime runs from the
job's reservation to its deletion, and an attempt that ends released is left out. Compare the
averages over the same window, and keep three differences in mind:

- Queen counts every attempt: a released one as processed, one that threw and is retried as
  failed, both with their runtime. A class that `WithoutOverlapping` or a rate limiter releases
  often shows more runs here, and shorter ones.
- Queen's runtime includes the broker ACK, a few milliseconds, where Horizon's includes a Redis
  delete. For short jobs, compare `max_ms` and the runs too.
- An attempt that outlives its timeout counts here, as failed, since PHP client 2.4.2.

Neither keeps percentiles. For a p50 or a p95, take per-job runtimes: Horizon's recent jobs carry
them, and on Queen a [monitored tag](#monitored-tags) that the class's jobs carry from their
`tags()` method records each job's runtime for `tags.retention_minutes`, at one write per job.

## Monitored tags

A job is tagged when it is pushed, as in Horizon: by its `tags()` method when it has one, otherwise
with one `Model:key` tag per Eloquent model it carries. Queued listeners, mailables, notifications
and broadcasts are unwrapped to the object that carries the models. The tags travel in the payload,
so a worker reads them without unserializing the command. A model is always tagged by its key, even
when it has a `tags()` relation, and a `tags()` that throws or returns no list is ignored: tagging
never fails a dispatch.

```php
class ChargeCustomer implements ShouldQueue
{
    public function __construct(public Customer $customer, public Invoice $invoice) {}

    // Optional: without it, the job is tagged App\Models\Customer:42 and App\Models\Invoice:7.
    public function tags(): array
    {
        return ['billing', 'customer:'.$this->customer->id];
    }
}
```

On the dashboard's **Tags** page, add a tag to monitor. Within thirty seconds every worker records
each completed or failed job that carries it, and the page lists the newest ones with their class,
queue, attempts and runtime. The list of monitored tags is one key updated with compare-and-swap
(the key/value store's `expect`), so two operators adding tags at the same moment never overwrite
each other. At most 50 tags can be monitored, and a job keeps at most 20 tags of up to 128 bytes.
Horizon's silenced jobs have no equivalent.

| Key of `config/queen.php` | Default | Meaning |
| --- | --- | --- |
| `tags.enabled` | `true` | tag jobs and record the monitored ones |
| `tags.connection` | `queen` | the Queen connection whose broker stores them |
| `tags.namespace` | `queen-metrics` | the key/value namespace |
| `tags.retention_minutes` | `1440` | how long a recorded job stays listed, in minutes |

## Long-wait alerts

`queen:check-waits` reports a queue whose oldest job has waited longer than its threshold. Schedule
it every minute and give each `connection:queue` its threshold in seconds:

```php
// routes/console.php
Schedule::command('queen:check-waits')->everyMinute();
```

```php
// config/queen.php
'waits' => [
    'queen:default' => 60,
    'queen:emails' => 300,
],
'notifications' => [
    'mail' => env('QUEEN_NOTIFY_MAIL'),      // comma-separated addresses
    'throttle_minutes' => 5,
],
```

For each `connection:queue` it checks every consumer group a supervisor pool consumes that queue
with, and the wait it compares is measured, never estimated. When the group has no pending job,
nothing waits. Otherwise the broker's lag for the group is the wait: how long the oldest message the
group has not consumed has been in the queue, from `GET /api/v1/consumer-groups/lagging`. For a
queue the group never consumed, it is the age of the oldest pending message, which also covers a
queue no worker serves at all.

Above the threshold it dispatches `Queen\Laravel\Events\LongWaitDetected` (connection, queue,
consumer group, seconds, threshold) and, when `queen.notifications.mail` is set, mails it. Each
queue and group notifies at most once per throttle window, and an alert that fails to send is tried
again at the next run. Listen to the event to send the alert anywhere else, such as Slack.

| Key of `config/queen.php` | Default | Meaning |
| --- | --- | --- |
| `waits` | one entry, `queen:` followed by `QUEEN_QUEUE`, at `60` | the threshold, in seconds, of each `connection:queue` |
| `notifications.mail` (`QUEEN_NOTIFY_MAIL`) | none | the addresses to mail an alert to |
| `notifications.throttle_minutes` | `5` | the throttle window of one queue and group, in minutes |

## Queue sizes

On a Queen connection, Laravel's size methods read the consumer group's depth from the broker
(`GET /api/v1/resources/queues/:queue/depth`) and the queue's [timers](/writing-applications/kv-locks-timers/):

| Method | Counts |
| --- | --- |
| `Queue::size()` | every job not yet acknowledged, waiting or under a live lease, plus the delayed jobs Laravel holds in timers |
| `pendingSize()` | the jobs a worker can claim now, the depth's `ready` |
| `reservedSize()` | the jobs under a live lease, the depth's `processing` |
| `delayedSize()` | Laravel's pending timers for the queue |

`php artisan queue:monitor` shows all four and compares `size()` with its `--max`, so running and
delayed jobs count toward that threshold, as they do on Redis.

## Prometheus and Kubernetes autoscaling

With a token set, `GET /queen/metrics` answers in the Prometheus text format. The scraper sends the
token as a bearer token; the route has no session and no Gate, and answers `404` while it is off.

```dotenv
QUEEN_METRICS_ENABLED=true
QUEEN_METRICS_TOKEN=a-secret-of-at-least-32-characters
```

| Key of `config/queen.php` | Default | Meaning |
| --- | --- | --- |
| `metrics.enabled` (`QUEEN_METRICS_ENABLED`) | `false` | serve the endpoint |
| `metrics.path` | `queen/metrics` | its route path |
| `metrics.token` (`QUEEN_METRICS_TOKEN`) | none | the bearer token; with the endpoint on, a missing token or one shorter than 32 characters stops the application at boot |

| Metric | Labels | Meaning |
| --- | --- | --- |
| `queen_queue_depth` | connection, consumer_group, queue | jobs not yet acknowledged, waiting or running, as a supervisor last sampled them |
| `queen_supervisor_instances` | availability | supervisor instances, live and stale |
| `queen_supervisor_up` | instance_id, hostname | 1 while the instance's heartbeat is current |
| `queen_supervisor_heartbeat_age_seconds` | instance_id, hostname | age of the instance's last status |
| `queen_workers`, `queen_workers_desired`, `queen_workers_draining` | instance_id, hostname, supervisor, queue | worker processes per pool |
| `queen_pool_replicas` | instance_id, hostname, supervisor, queue | coordinated replicas sharing the pool's target |
| `queen_shared_queue_supervisors` | connection, consumer_group, queue | masters that autoscale one queue without coordinating |

The endpoint reads what the supervisors publish, so on web pods it needs
[remote status](/integrations/laravel/dashboard/#a-supervisor-on-another-host). With
[coordinated replicas](/integrations/laravel/supervisors/#several-replicas), scale the supervisor
Deployment on the backlog. With KEDA's Prometheus scaler, for pods of up to 10 workers, each sized
for 10 jobs:

```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: queen-workers
spec:
  scaleTargetRef:
    name: queen-workers        # the supervisor Deployment
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: max(queen_queue_depth{queue="default",consumer_group="laravel"})
        threshold: "100"       # max_processes x target_jobs_per_process
```

Each new pod joins the coordination and takes its share at its next poll, and a pod removed by a
scale-down leaves at once. [Scale on Kubernetes](/integrations/laravel/kubernetes/) has the whole
deployment.

The broker dashboard can also inspect the same remote publications on its
[Supervisors page](/manage/upgrades-monitoring/): one overview per instance, with
worker capacity, process budget and pool details in a drawer. Publication groups follow the
[shared status contract](/reference/supervisor-status/); they do not replace consumer groups or
change coordination membership.

The broker has Prometheus metrics of its own, about queues, partitions and the cluster (see
[monitoring the broker](/manage/upgrades-monitoring/)). This endpoint covers the Laravel side: workers,
pools and the depth the supervisors see.

Source: https://queenmq.com/integrations/laravel/monitoring/index.mdx