---
title: "A million ordered partitions"
description: "1,000,000 FIFO lanes in one PostgreSQL, created during the run under load."
---

> Queen MQ documentation, for AI agents
> Complete self-contained summary of Queen MQ: https://queenmq.com/llms-brief.txt
> Fetch that first when the question is about the product rather than about this page.
> Index of all pages: https://queenmq.com/llms.txt

# A million ordered partitions

**One broker against one PostgreSQL held 1,000,000 ordered partitions in a single queue, none
preallocated, created during the run at a thousand a second while it served 200,000 messages a
second with leased pops and explicit acknowledgement. 722,265,600 messages, zero push, pop or ack
errors, and a database that stopped growing.** Queen 1.0.0, 32 vCPU / 62 GiB, artifacts in
`benchmark-queen/2026-08-11-1m-partitions/`.

A partition costs a row and an index range, not a file or a process. The serve path does not care
how many there are.

## Configuration

The space is a million, the working set a thousand: the loader rotates which 1,000 partitions
take pushes, so each sees 0.2 messages a second and each active one 200.

```text
goload -mode openloop -> queue=partbench partitions=1000000 consumers=300 manualAck=true ackAsync=true
  offered: rate=200000 msg/s | push-batch=100 -> 2000.0 req/s across 41 pacer workers | payload=256B
  partitions: space=1000000 | active=1000/s (rotate) | 200 msg/s per active partition | space covered in 1000 s
```

Pop batch 1,000, wildcard pops, retention on with `RETENTION_PARALLELISM=16`, **deduplication
off**, one hour of load of which 43 minutes on the completed space.

## Result

From the final line of `raw-final-dedupoff/g.out`:

| Metric | Value |
| --- | --- |
| Messages offered | 722,270,400 |
| Messages accepted (pushed) | **722,265,600** |
| Shed by the loader's in-flight cap | **0** |
| Messages popped | 722,263,300 |
| Messages acked | 722,261,300 |
| **Push / pop / ack errors** | **0 / 0 / 0** |
| Partitions created | **1,000,000**, at 1,000 per second |
| Average ack round trip | 25.54 ms |
| Restarts, incidents | **0, 0** |

### Latency in two phases

Creating a lane costs a row and a lock, so a thousand a second is expensive.

| Phase | 30 s reports | p50 | p99 | Worst p99 | p99.9 |
| --- | --: | --: | --: | --: | --: |
| While creating the million | 34 | 40.5 ms | 214.1 ms | 346.1 ms | 339.8 ms |
| **On the completed space** | **86** | **27.5 ms** | **115.3 ms** | **209.9 ms** | **189.9 ms** |

### The database stopped growing

Over the last nine minutes:

| | 11:56 | 11:59 | 12:01 | 12:04 |
| --- | --: | --: | --: | --: |
| `log_segments` | 3,085 MB | 3,085 MB | 3,085 MB | 3,085 MB |
| `log_txns` | 5,301 MB | 5,302 MB | 5,302 MB | 5,302 MB |
| Database | 9.37 GB | 9.40 GB | 9.43 GB | 9.47 GB |

The cause is in the retention log: **4,002 rows deleted per second over the last eight sweeps,
against the 4,000 the offered rate creates.**

### What it weighs

| Object | Size |
| --- | --: |
| `log_partitions`, 1,000,000 rows | **315 MB** |
| `log_consumers`, 1,000,000 cursors | **326 MB** |
| **The lanes and their cursors** | **641 MB** |
| `log_segments`, messages inside the retention window | 3.01 GB |
| `log_txns`, the transaction sidecar | 5.18 GB |
| Database total | 9.47 GB |
| **Broker resident memory** | **5.03 GB** |

Only the first two rows scale with cardinality. The sizing rule is the last: **about 5 KB of
resident memory per partition**.

### Resources, 1 Hz host sampler

| Resource | Value |
| --- | --- |
| PostgreSQL CPU | 10.15 cores of 32 |
| Broker CPU | 3.84 cores |
| Broker resident memory | 5.03 GB, no drift |
| Active backends | 28 |
| PostgreSQL commits | about 16,500 per second |
| WAL fsync | 247 µs |
| Connection pool | never above 208 of 300 |

The [hot list](/internals/hotlist) holds only partitions with data, so its cost follows the work:
a few thousand entries all run.

## Three costs that follow the space

Invisible at two hundred partitions, decisive at a million.

**[Deduplication](/internals/dedup) holds every hash, not the ones in the window.** The cache
grew 235 MB a minute, 19.6 bytes per message against a model asking 192 MB, and filled 6 GB then
16 GB: a block seals only when it fills, nearly six hours at 0.2 messages a second per partition.
Saturated, pushes fall back to a full-window SQL probe, **still authoritative, no duplicate
admitted**, but the cost lands on PostgreSQL.

**[Retention](/internals/retention) walks one row per partition per cycle.** At
`RETENTION_PARALLELISM=4` the sweep grew from 108 s at 400,000 partitions to 402 s at a million;
at 16 it settles near 195 s, which is what this run used.

**The [statistics refresh](/internals/stats) took 6.8 seconds** at 825,706 partitions, harmlessly.

## What it does not establish

- **Not a 24-hour run.** One hour, 43 minutes of it on the completed space.
- **Deduplication was off**, which is what made the run possible: this run and
  [Throughput 24h 1M](/benchmarks/soak-24h) do not add up.
- **One shape.** A contiguous rotating window is the friendly case for index locality.
- **A compressible payload.**
- **Nothing about high availability, failure or the proxy.**

Source: https://queenmq.com/benchmarks/cardinality-1m/index.mdx