---
title: "Kafka protocol support"
description: "Which Kafka APIs the queen-kafka facade speaks and at which versions, what it deliberately does not implement, and every place its behaviour differs from a real broker."
---

> Queen MQ documentation, for AI agents
> Complete self-contained summary of Queen MQ: https://queenmq.com/llms-brief.txt
> Fetch that first when the question is about the product rather than about this page.
> Index of all pages: https://queenmq.com/llms.txt

# Kafka protocol support

`queen-kafka` is a separate binary that speaks the Kafka wire protocol to clients and plain
HTTP to a Queen broker or proxy as an ordinary client. An unmodified Kafka producer or
consumer reaches Queen by changing `bootstrap.servers`, and nothing else. Running it is
[the operator page](/deploy/kafka); this page is the contract a client gets.

> **Caution**
>
> The facade ships in the broker image and is started by `QUEEN_KAFKA_EMBEDDED=true`. What is
> narrower than the broker's own guarantees is the verification behind this page: the behaviour
> below is checked by the facade's own compatibility suite rather than by a released client matrix,
> and the deviations at the bottom are the ones known today rather than a closed list.

The mapping is small enough to state in four lines. A Kafka topic is a Queen queue. Kafka
partition `n` is Queen partition `n`. One record is one Queen message, wrapped in the payload
envelope below. In the default single-node mode there is exactly one broker, node 0, and it is the
facade itself: it advertises itself as leader of every partition and answers `FindCoordinator` with
its own address, whatever the Queen cluster behind it looks like. Two or three facades can instead
present themselves to clients as one Kafka cluster, which is an operator decision described on
[the operator page](/deploy/kafka#cluster-mode); the differences a client can read are named where
they arise below.

## The support matrix

Every Kafka client opens a connection with `ApiVersions` and then speaks only what the answer
allowed. That answer is built from one table in `protocols/queen-kafka/src/versions.rs`, which is also
the gate every incoming request is checked against, so a version that is advertised is a
version that is handled.

The facade advertises **32 Kafka APIs**. Every row is read out of `protocols/queen-kafka/src/versions.rs` at build time, which is the same table the ApiVersions response is built from and the same table every incoming request is checked against.

| API | Versions | Where the window ends, and why |
| --- | --- | --- |
| `AddOffsetsToTxn` | v0–v3 | The whole window a client uses. Every field of the schema is marked v0-v4 and v3 is already the flexible encoding, so v3 answers everything v4 does; v4 exists for KIP-890's transaction protocol 2, in which the client stops sending this API at all and the coordinator infers the offsets partition. Advertising it would advertise a protocol this facade does not run. |
| `AddPartitionsToTxn` | v0–v3 | Ceiling: v4 is a DIFFERENT REQUEST, not a wider one. The flat (transactional_id, producer_id, producer_epoch, topics) of v0-v3 becomes a `transactions[]` array with a `verify_only` flag — KIP-890's coordinator-to-partition-leader verification, which a client never sends and only another broker does. Floor: the schema's own, since KIP-896 dropped nothing here. |
| `AlterConfigs` | v0–v2 | The whole schema window: every field of both schemas is marked v0-v2, so nothing varies inside it but the flexible encoding (v2). Floor: the schema's own, and it is 0, because KIP-896 dropped nothing from this key and inventing a floor above 0 would refuse a version a real broker serves. This is the deprecated FULL-REPLACEMENT form, honoured literally: a key the request does not name is reset to its default. Prefer IncrementalAlterConfigs. |
| `ApiVersions` | v0–v3 | Ceiling: one below the schema, on purpose. v3 is what every client in the compatibility matrix negotiates against a 3.x broker, and it keeps the v0 fallback on a path a real client reaches. |
| `CreateAcls` | v1–v3 | The same window and the same refusal as DescribeAcls, with one difference on the wire that a client can read: the error is carried PER CREATION rather than at the top level, because Kafka's own error response maps over the request. An empty creations list therefore answers an empty result list and no error at all. |
| `CreatePartitions` | v0–v3 | The whole schema window; nothing varies inside it but the flexible encoding (v2). What is advertised is a REFUSAL. Queen declares no width per queue — a partition exists once something has been written to it — and while a topic may carry its own width floor, that floor is declared ONCE, at CreateTopics, and this API is not that writer. Two of the three answers are Apache Kafka's own sentences byte for byte, since a DECREASE and an EQUAL count are refused by a real broker too, and only an increase is a capability gap. The alternative, no row at all, would tell an operator to upgrade their broker, which is the wrong diagnosis in all three cases. |
| `CreateTopics` | v2–v6 | Ceiling: v7 answers a `topic_id` UUID this facade has no registry to mint, the same boundary Metadata stops at. v4 (KIP-464) is where -1 means 'you choose' for the partition count and replication factor, v5 carries the created topic's real configs back, and v6 is where a client understands THROTTLING_QUOTA_EXCEEDED. Floor: the schema's own. |
| `DeleteAcls` | v1–v3 | The same window and the same refusal, per FILTER, each with an empty matching_acls. Advertising these three rather than leaving them out is what turns "this broker is too old" into the sentence a real Kafka with security off prints. |
| `DeleteGroups` | v0–v2 | The whole schema. v2 is the flexible encoding and adds no field, and there is no version of this API that asks for something the facade cannot answer. |
| `DeleteTopics` | v1–v5 | Ceiling: v6 replaces the name list with entries carrying a name or a topic id, and an id is a name this facade cannot resolve. v5 adds `error_message`, which is where 'there is no such queue' gets to say so. Floor: the schema's own. |
| `DescribeAcls` | v1–v3 | The whole schema window, and no ceiling to argue: every field of the request and the response is marked v1-v3, so nothing varies inside it but the flexible encoding (v2). Floor: the schema's own, which is KIP-896's, since v0 was dropped. What is advertised is a REFUSAL. Every call answers SECURITY_DISABLED, which is what an Apache Kafka broker with no authorizer answers, because Queen has no ACL model to answer anything else from. |
| `DescribeConfigs` | v1–v4 | The whole schema, and the only row here with no ceiling to argue: nothing in the window asks for something the facade cannot answer. v3's `config_type` and `documentation` are answered truthfully for the keys reported, and the answer is short because a key is reported only where the facade can name what enforces it. |
| `DescribeGroups` | v0–v3 | Ceiling: v4 carries `group_instance_id` (static membership, out of scope), the same rule that caps JoinGroup at 4. v3's `include_authorized_operations` is answered with Kafka's own omitted sentinel, because the facade has no ACL model and a computed bitfield would be an invented permission set. |
| `EndTxn` | v0–v3 | Ceiling: v5's RESPONSE carries `producer_id` and `producer_epoch` — the transaction-protocol-2 epoch bump performed inside EndTxn, which this facade does not perform — and v4 is the version pair that exists on the way to it. v3 is the flexible encoding and asks for nothing that cannot be answered. |
| `Fetch` | v4–v6 | Ceiling: v7 introduces fetch sessions (KIP-227), which are per-connection broker state. The facade keeps none by design, so the cap deletes `session_id`, `session_epoch` and `forgotten_topics_data` rather than half-answering them. v4 is the schema's own floor. |
| `FindCoordinator` | v0–v3 | Ceiling: v4 is the batched form, which exists for clusters where groups live on different brokers. In single-node mode the answer is this process for every key, group or transaction. In cluster mode a GROUP key resolves to the rendezvous owner over the live node set, and a TRANSACTION key is refused TRANSACTIONAL_ID_AUTHORIZATION_FAILED — fatal on purpose, so `initTransactions()` stops instead of looping on a retriable code for the whole of `max.block.ms`. |
| `Heartbeat` | v0–v2 | Ceiling: v3 carries `group_instance_id` (static membership, out of scope). |
| `IncrementalAlterConfigs` | v0–v1 | The whole schema window (v1 is the flexible encoding and adds no field), and the key that matters: `kafka-configs.sh --alter` has sent this since Kafka 2.3 and has no fallback to the deprecated key 33, so this is what an operator's command actually lands on. What it can write is bounded by what the facade can write LOSSLESSLY. Queen's configure route is a whole-row upsert whose columns mostly cannot be read back, so an alter lands only on a topic this facade created and every other topic is refused with the reason. |
| `InitProducerId` | v0–v4 | Ceiling: v5 exists for KIP-890's transaction protocol 2, which this facade does not perform — the same ceiling argument the four transaction rows make. Since M9 this key grants a `transactional.id` as well as an idempotent one, so what stops at v5 is a PROTOCOL the facade does not run rather than a feature it refuses; in cluster mode a transactional id is still answered TRANSACTIONAL_ID_AUTHORIZATION_FAILED. v3 is load-bearing rather than a nicety: it is KIP-360's epoch bump, and it is what turns a sequence window this facade has lost (a restart, an evicted entry) into a reset the producer recovers from instead of a fatal error. Read the REQUEST schema for the cap, not the key's `valid_versions()`, which answers wider because it takes the maximum of request and response. |
| `JoinGroup` | v0–v4 | Ceiling: v5 carries `group_instance_id` (static membership, out of scope). v4 is also where MEMBER_ID_REQUIRED lands, and that is implemented. |
| `LeaveGroup` | v0–v2 | Ceiling: v3 carries `group_instance_id`, and is also where one request may remove several members at once. Below it a request is exactly one member, which is the shape the coordinator has. |
| `ListGroups` | v0–v4 | Ceiling: v5 adds `group_type`, the KIP-848 discriminator between a classic group and a consumer-protocol one, and KIP-848 is excluded by plan — answering a group TYPE question would claim a taxonomy the facade does not implement. v4 is KIP-518's `states_filter` and `group_state`, and both are honoured rather than ignored. |
| `ListOffsets` | v1–v5 | Ceiling: v7 adds the MAX_TIMESTAMP sentinel, a time-index question Queen cannot answer. v5 is the last version whose whole surface is the two watermark sentinels. v1 is the schema's own floor. |
| `Metadata` | v0–v9 | Ceiling: v10 adds topic ids, and a client may then address a topic by a UUID this facade has no registry to resolve. v9 is already the flexible encoding and carries every field a client needs. |
| `OffsetCommit` | v2–v6 | Ceiling: v7 carries `group_instance_id` (static membership, out of scope). Floor: v0 and v1 are the ZooKeeper-era offset store. |
| `OffsetDelete` | v0 | One version, so there is no window to argue. It is the last thing `kafka-consumer-groups.sh` could not do here, and Kafka's guard for it is not membership but SUBSCRIPTION: a live consumer group's subscribed topics are refused and everything else is deletable. The facade keeps that rule exactly rather than approximating it, because the coordinator holds each member's JoinGroup metadata verbatim and those bytes are a ConsumerProtocolSubscription. |
| `OffsetFetch` | v1–v7 | Ceiling: v8 fetches offsets for several groups in one request and changes the response shape. v7's `require_stable` is answered honestly rather than ignored: it asks the broker to withhold offsets belonging to an open transaction, and since M9 an offset belonging to an open transaction is not in the store at all — the store write happens at COMMIT, in the same Postgres transaction as the records. Every offset returned is stable by construction, so UNSTABLE_OFFSET_COMMIT (88) is a code this facade never needs. |
| `Produce` | v3–v9 | Floor: v3 is the first version whose records are RecordBatch v2. Ceiling: v10 adds the leader-change hint `current_leader`, which has no meaning against one broker with no elections, and v13 addresses topics by UUID. |
| `SaslAuthenticate` | v0–v1 | Ceiling: v2 is the flexible encoding and adds no field. v1's `session_lifetime_ms` is answered 0, which is what stops every client re-authenticating on a timer this facade does not run. |
| `SaslHandshake` | v0–v1 | Both versions, because they are the two SASL flows: after v0 the tokens travel as raw bytes in ordinary frames, after v1 inside SaslAuthenticate requests. Both are implemented. |
| `SyncGroup` | v0–v2 | Ceiling: v3 carries `group_instance_id` (static membership, out of scope). |
| `TxnOffsetCommit` | v0–v3 | The FLOOR is the load-bearing number here, and it is measured rather than preferred: TxnOffsetCommitRequest$Builder.build(short) in kafka-clients 3.9.2 throws UnsupportedVersionException below v3 whenever group metadata is set, and every KIP-447 consume-transform-produce loop sets it — so advertising this API below 3 would make the flagship use case throw before any wire traffic. Ceiling: the same transaction-protocol-2 bump as the other three, which adds no field a client fills in. |

### Not offered

A client that sends one of these gets no response frame at all: the connection closes, with the reason in the facade's log. That is Apache Kafka's own behaviour for an unparseable request, and it is unreachable for a client that read the ApiVersions answer, which is every client. Each row is a decision with a test behind it: the facade's own suite asserts that none of these keys is advertised, so offering one by accident fails a test rather than shipping.

| API | What a client wants it for | Why it is not here |
| --- | --- | --- |
| `ConsumerGroupHeartbeat` | The KIP-848 broker-side rebalance protocol. | Excluded by plan; groups use the classic Join/Sync protocol. |
| `DeleteRecords` | Truncates a partition below an offset. `kafka-delete-records.sh`, and the "Clear messages" button in kafka-ui and AKHQ. | Queen has no truncate-to-offset primitive: a queue's log start moves by retention and by dropping the queue, both time-driven. Implementing it would mean reporting a low watermark that did not move, which is a fabricated value a tool would act on. DeleteTopics then CreateTopics is the workaround, and both work. |
| `OffsetForLeaderEpoch` | Detects log truncation after a leader change. | Every leader epoch this facade reports is -1, in Metadata, in ListOffsets, in OffsetFetch and in every record batch, so a consumer's subscription state never holds one and the request is never built. No tool sends it directly, and the cost of the absence is nothing measurable. |
| `DescribeLogDirs` | Per-partition storage sizes. `kafka-log-dirs.sh`, and the Size column in kafka-ui. | Queen's storage is Postgres segments; there are no log directories, and answering would mean inventing a path and per-partition byte sizes. The best future candidate of the absences: retained bytes are real and already on the queue listing, so honest sizes under one synthetic log dir are possible once Queen reports them per partition rather than per queue. Until then kafka-ui renders the page with a blank Size column. |
| `CreateDelegationToken` | Mints a broker-signed token from an authenticated principal. | A delegation token is derived from a SCRAM principal and signed with a cluster secret. This facade mints no credentials; Queen does. `kafka-delegation-tokens.sh` fails, and nothing in the client matrix uses it. |
| `RenewDelegationToken` | Extends a delegation token's life. | Same reason. |
| `ExpireDelegationToken` | Revokes a delegation token early. | Same reason. |
| `DescribeDelegationToken` | Lists the tokens a principal holds. | Same reason. |
| `ElectLeaders` | Moves a partition's leadership to its preferred replica. `kafka-leader-election.sh`. | One logical broker and no replicas: every Metadata answer is replicas=[0], isr=[0]. In cluster mode a partition's leader is a rendezvous hash over the live set, which is deterministic and not movable. There is no preferred replica to elect and no unclean election to permit. |
| `AlterPartitionReassignments` | Moves replicas between brokers. `kafka-reassign-partitions.sh`, Cruise Control. | There are no replicas to move; durability is Postgres's. A reassignment API over one logical broker would accept a plan and then have nothing to do with it. |
| `ListPartitionReassignments` | Reports reassignments in flight. | Same reason. No UI in the client matrix calls it on a render path. |
| `DescribeClientQuotas` | Reads the produce/fetch quotas for a (user, client-id). | The facade DOES have quotas, as Queen's 429 with Retry-After surfaced as throttle_time_ms, but they are the Cloud proxy's and they are per TENANT, which is not expressible in Kafka's entity model. Describing them would mean inventing an entity mapping. The one absence with a real future story: a read-only version of this key mapping the tenant onto a user entity, once the proxy exposes the cap. |
| `AlterClientQuotas` | Writes those quotas. | Same mapping problem, and a second and independent reason it must never work: it would let a tenant raise its own rate cap from a Kafka client, which is a privilege escalation. This key stays absent even if the read half ever lands. |
| `DescribeUserScramCredentials` | Lists SCRAM users. | SASL here is PLAIN only and the credential is a Queen bearer token verified by Queen. There is no local user store to describe. |
| `AlterUserScramCredentials` | Creates or rotates a SCRAM credential. | Supporting SCRAM would require the facade to hold salted password verifiers: to become a credential store, with its own rotation, its own secrets at rest and its own blast radius. That is a security posture change rather than a protocol gap, and it is not a decision this milestone could take. |
| `DescribeQuorum` | Describes the KRaft metadata quorum. `kafka-metadata-quorum.sh`. | No Raft log and no voters, so every field would be invented. The one thing UIs actually render from it, a controller id, is already in every Metadata answer. kafka-ui feature-detects and hides its KRaft panel. |
| `DescribeCluster` | Cluster id, controller and broker list in one call. | Answerable truthfully, and deliberately not answered: every client in the compatibility matrix already resolves describeCluster() from a plain Metadata request, so advertising it would move five live suites onto a code path none of them exercises today for a measured gain of zero. The trigger that flips this is the first client whose describeCluster() stops falling back. |
| `DescribeProducers` | Per-partition producer state: id, epoch, last sequence. `kafka-transactions.sh find-hanging`. | The facade's idempotence window is PROCESS state and is deliberately lost on restart. Answering from it would advertise durable producer state the facade does not have; answering an empty list would say nothing is producing while producers produce. Both are lies. |
| `DescribeTransactions` | Describes one transaction's state. | Transactions landed in M9 and this key still does not, for the reason that survived them: an open transaction is a stage held by ONE facade process, on the connection that opened it. A node can describe only its own, so an operator asking a load-balanced address would get a different answer per connection, and `kafka-transactions.sh` would report a transaction as absent because it asked the wrong node. |
| `ListTransactions` | Lists transactions in flight. | Same reason, and it is the one that bites harder: a LIST that is per node reads as the whole cluster's and is not. |

## Transactions and the idempotent producer

Both work, and neither asks a client for configuration. A producer at stock defaults negotiates
`InitProducerId` and gets the idempotent path; a producer carrying a `transactional.id` gets a real
transaction. `enable.idempotence=false` was mandatory against this facade until 2026-08-29 and is
now only a way of asking for less than it offers.

**Idempotent produce.** `InitProducerId` allocates a producer id, and the facade keeps the
per-(producer, topic-partition) sequence window that turns a retried batch into a no-op. That window
is in memory and per facade process: a restart, an eviction past the 65,536 tracked
producer-partitions, or a connection that lands on another facade all leave no entry, and the next
batch is answered `OUT_OF_ORDER_SEQUENCE_NUMBER`. The client recovers through KIP-360's epoch bump,
and the cost is at-least-once for at most the five batches that were in flight. A real Kafka broker
persists producer state in its log; this one holds nothing durable of its own, by design.

**A transaction is a stage, held by the process the client opened it on.** The `transactional.id` is
claimed in Queen's key/value store by a compare-and-set, and that claim is the fencing: a second
producer taking the same id bumps the epoch, and the first is answered `PRODUCER_FENCED` from then
on. A transactional `Produce` writes nothing, it stages the records in that process.
`AddOffsetsToTxn` and `TxnOffsetCommit` stage the consumer offsets beside them. `EndTxn(commit)`
sends the whole stage as one `POST /api/v1/transaction`, with the fence check as the first operation
of the bundle, so the records and the offsets commit in one PostgreSQL transaction or neither of
them does. `EndTxn(abort)` drops the stage and writes nothing at all.

That covers the stock Java transactional producer, Spring's `KafkaTransactionManager`, franz-go's
`GroupTransactSession` and librdkafka's transactional API: consume, transform, produce and commit,
all in one process. It does not cover a two-phase commit finished somewhere else. Flink's
`KafkaSink EXACTLY_ONCE` and Spark's structured-streaming writer commit after a failover from a
different process, that `EndTxn` reaches a facade holding no stage, and the answer is
`INVALID_TXN_STATE`, which is fatal.

Because the stage is memory, it is capped in five places where Kafka has nothing to cap:
`QUEEN_KAFKA_TXN_MAX_BYTES` (8 MiB for one transaction), `QUEEN_KAFKA_TXN_MAX_RECORDS` (50,000),
`QUEEN_KAFKA_TXN_MAX_STAGED_BYTES` (128 MiB for the whole process), `QUEEN_KAFKA_TXN_MAX_OPEN`
(1,024) and `QUEEN_KAFKA_TXN_MAX_TIMEOUT_MS` (900,000, which is Kafka's own default for
`transaction.max.timeout.ms`). Two more are derived rather than chosen and cannot be set: at most
200 partitions and at most 62 offsets in one transaction. Past a cap the transaction is answered
`MESSAGE_TOO_LARGE` or `INVALID_COMMIT_OFFSET_SIZE` and has to be aborted rather than retried,
because waiting does not make a 12 MiB transaction fit an 8 MiB stage.

**Cluster mode refuses transactions outright.** With `QUEEN_KAFKA_NODE_ID` set, `FindCoordinator`
for a transaction key and `InitProducerId`'s transactional branch both answer
`TRANSACTIONAL_ID_AUTHORIZATION_FAILED`, which every client treats as fatal, so
`initTransactions()` returns in milliseconds instead of looping for the whole of `max.block.ms`.
It is a refusal by configuration and not a capability gap: a stage held in one process cannot be
honoured by whichever node the client's next request is routed to.

## Non-goals

These are not gaps waiting to be filled in a patch release. Each is excluded by design, and the
exclusion is enforced rather than announced.

**Log compaction.** There is no compaction and no tombstone semantics. A null value is stored
and delivered as a null value, not as a delete. Queue [retention](/reference/queue-options) is
Queen's, is time-based, and is off by default on a topic the facade auto-creates. This is what keeps
Kafka Streams and Connect's exactly-once source out, and transactions did not change it: what those
two depend on is the compacted changelog and offset topics, not the transaction protocol.

**KIP-848.** `ConsumerGroupHeartbeat` is not advertised, so a client configured for the new
broker-side rebalance protocol falls back to the classic `JoinGroup`/`SyncGroup` one, which is
what the coordinator implements.

**Static membership.** `group.instance.id` (KIP-345) is out of scope, and it is kept out by the
version caps rather than by ignoring the field: `JoinGroup` stops at v4, `SyncGroup`,
`Heartbeat` and `LeaveGroup` at v2, `OffsetCommit` at v6, each one version below where the
field appears. A client cannot send what it cannot negotiate, so a fleet configured for static
membership is refused the version rather than silently given dynamic behaviour, which would
show up as unexplained rebalances on exactly the deployments that configured static membership
to avoid them.

**Offset expiry.** Kafka drops the committed offsets of a group that has been empty for
`offsets.retention.minutes`, seven days by default. The facade never does. A committed offset
that expired would be a consumer group silently resuming from `auto.offset.reset`, which is
either a replay of the whole topic or a jump past everything it had not read. The cost is
stated rather than hidden: a facade that has served a million throwaway group ids holds a
million keys. `DeleteGroups` and `OffsetDelete` are both advertised and both remove them
irreversibly, so removal here is an administrator's action rather than a timer nobody set.

## Deviations from a real broker

Everything below is deliberate and is documented in the code that does it. None of it is a
bug report. Three are marked *awaiting ratification*: they are shipped choices that have been
measured against `apache/kafka:3.9.1` and not yet signed off, so they are the entries most likely
to move.

**A transactional produce answers `base_offset = -1`.** *Awaiting ratification.* Kafka appends the
batch as it arrives and answers the offset it landed at. Here no offset exists until
`EndTxn(commit)` allocates them all, and the client has to be answered before that. The Java
client keeps the `-1` in `RecordMetadata` rather than adding a batch index to it, so no fabricated
offset ever reaches an application: what an application loses is `RecordMetadata.offset()` inside a
transaction, and what it gains is that nothing partial is ever in the log. Every non-transactional
produce still answers a real offset.

**`read_uncommitted` sees less here than it does on Kafka.** *Awaiting ratification.* An open
transaction's records are invisible until the commit, and an aborted transaction's are never
visible at all, so the two isolation levels return the same records and setting one costs nothing.
`read_uncommitted` is the default, and an ordinary consumer sees the same records in the same
order, later by the producer's own commit cadence; no client library exposes "records that may yet
be rolled back" as a state an application can act on. The upside is measurable in lag: a
`read_committed` consumer reaches 0 here, where against Kafka it stops at one per partition because
of the commit marker.

**A batch from a producer this facade has never seen is refused.** *Awaiting ratification.*
Sequence 42 from an unknown producer id is answered `OUT_OF_ORDER_SEQUENCE_NUMBER`; Apache Kafka
3.9.1 accepts it. In Kafka an absent producer entry means the state aged out, which is rare; here it
means the facade restarted, which is not, so accepting would leave the sequence window silently
unenforced after every restart. The cost of refusing was measured rather than assumed: the producer
recovers through KIP-360's epoch bump and nothing is lost.

**A committed transaction advances the log end offset by N and not by N+1.** Kafka writes a commit
or abort marker into the data partition; this facade writes the records and nothing else. Nothing a
client reads differs, which the differential runner measures on both brokers for the same
transaction. An aborted transaction advances the offset by 0 here, where Kafka advances it by N+1
and leaves the aborted records in the log for the client to filter out. There is nothing to filter
here, because there is nothing there.

**`acks=0` gets no response frame.** That is Kafka's own contract, and the consequence is worth
saying out loud: the offsets such a producer reports are ones it invented by counting, because
nothing was sent back for it to read. The request is still processed exactly as any other, and
anything that went wrong is written to the facade's log, which is the only place a
fire-and-forget producer's failures exist at all.

**`acks=1` and `acks=-1` are the same path.** Every push is a durable, synchronously committed
write. Queen has no in-memory tier to make `acks=1` cheaper with today, so both are answered
from the durable path, which is the safe direction to be wrong in.

**`timeout_ms` on a produce is not acted on.** It is how a Kafka leader bounds waiting for its
followers, and there are no followers here. What bounds the wait is the facade's own ten-second
budget for a call to Queen, well below every client's 30 second `request.timeout.ms`.

**Topics beginning with `__` do not exist.** Kafka's internal topics are answered
`UNKNOWN_TOPIC_OR_PARTITION` on Metadata, Fetch, ListOffsets and OffsetCommit alike, so a name
the listing hides cannot be made visible by addressing it directly, and
`allow.auto.create.topics` can never conjure a queue called `__consumer_offsets`. The rule is by
prefix, not by a list of known names.

**Auto-create cannot be turned off for Metadata v0 to v3.** `allow_auto_topic_creation` is a v4
field. Below it the request carries no such flag and the facade reads the schema default, which
is "allowed". There is no broker-side knob to override that, so on those versions a Metadata
request that names an absent topic creates it.

**A concrete timestamp answers "no match".** `ListOffsets` resolves the two sentinels exactly:
`-1` is the high watermark and `-2` is the log start. Any other timestamp is answered offset
`-1` with error code 0, which is the protocol's own "this broker found no match" and is what
`offsetsForTimes` returns null for. Queen keys segments by offset and has no per-segment time
index, so the alternative would be a scan or an approximate answer, and an approximate answer is
a consumer silently starting somewhere other than where it asked. The partition is still probed
for existence, so a timestamp query against a topic that is not there is still
`UNKNOWN_TOPIC_OR_PARTITION`.

**A heartbeat during the sync phase answers no error.** A member that has already had its
`JoinGroup` response and owes a `SyncGroup` is told to carry on rather than
`REBALANCE_IN_PROGRESS`. Telling it to rebalance would invite a rejoin, and a rejoin reopens the
join window it is on the far side of, so the group would chase its own tail for as long as the
client kept heartbeating. A member in the join phase does get `REBALANCE_IN_PROGRESS`, which is
how every rebalance reaches the members that did not cause it.

**Fetch fills partitions in request order.** The answer is one frame with a byte budget, and
partitions are filled in the order the request named them until the budget is gone. The first
batch is always whole enough to carry one record however large it is, so a consumer can never
stall on an oversized record. Apache Kafka rotates the partition order between fetches to keep
trailing partitions from starving behind saturated leaders, and the facade does not yet.

**A topic's partition count is `max(live, floor)`,** where `floor` is the width the topic
declared at CreateTopics if it declared one, and `QUEEN_KAFKA_DEFAULT_PARTITIONS` otherwise. A
Queen queue declares no width: a lane materialises the first time something is pushed to it, so
a queue created a second ago has zero partitions and a topic with zero partitions is one a
producer cannot send to. Taking the maximum makes a fresh topic usable immediately, never
shrinks as lanes materialise (a Kafka partition count that shrinks re-hashes existing keys onto different
partitions and loses ordering), and still covers a native Queen queue wider than the configured
default. The advertised width is clamped at 100,000, because every partition is written out in
full in every Metadata response.

**Queen partitions whose names are not decimal indexes are invisible.** A Kafka client can only
name a partition by number, so lanes called `Default`, `eu-west` or a tenant id are counted but
not addressable. Producing through the facade always creates numeric lanes, so this affects only
queues that Kafka clients share with native producers.

**Committed offsets live in Queen's key/value store, not in a consumer group.** One key per
(group, topic, partition) under the `queen-kafka` namespace, spelled
`qk:group:<group>:<topic>:<partition>` with everything outside `[A-Za-z0-9._-]` percent-encoded.
They survive a facade restart, and they are still not the broker's native consumer-group cursor:
KV is the single source of truth for a Kafka group's position and nothing outside the facade
writes one.

They are no longer invisible, though. The three consumer-group views read both stores and mark
every row with an additive `kind` of `queen` or `kafka`, applying the same lag arithmetic to
each, so a Kafka group's lag stands beside a native one and matches what Kafka's own admin API
reports for it. That mirror is read only. One place it cannot show both is the single-group
detail view, which is keyed by queue name: where a Queen group and a Kafka group share a name on
one queue the native entry wins, and the `kind` field says which store the answer came from. The
list view renders both.

**A facade restart is a broker restart.** Group membership is in memory and nothing else in the
facade is durable. A restarted process knows no members, answers `UNKNOWN_MEMBER_ID` to the
first heartbeat of every survivor, and they rejoin and resume from their committed offsets.
That is the sequence a real broker failover produces and every client already implements it.

**Two credentials of one tenant are one group where the deployment can say so, and two where it
cannot.** The group registry is keyed by (identity, group id), because `orders-consumer` is a name
two strangers pick and a registry keyed by the group id alone would put two tenants in one group.
The identity comes from asking Queen `GET /auth/me` once per credential, and everything turns on
whether that answer names the tenant.

In **Queen Cloud** it does. The proxy resolves an api-key bearer through its normal credential
path, so a key rotation or a per-service key resolves to the same tenant and the two credentials
are ONE group over one set of committed offsets, which is what a group of that name should be.

Against a **broker with `JWT_ENABLED` set** it does not: the endpoint does not identify a bearer,
so the identity falls back to the hashed credential and the old behaviour stands. Two consumers of
one group id become the sole member of two groups, each assigned everything by its own leader, both
writing the same offset keys. The facade cannot resolve that from the inside, so it logs when one
group id goes live under a second credential. In that deployment, give one consumer group one
credential.

## The payload envelope

A Kafka record is arbitrary bytes and a Queen payload is JSON, so the facade defines exactly one
shape and both directions read it from one module.

```json
{
  "k": "b3JkZXItNDI=",
  "v": "eyJ0b3RhbCI6IDEyfQ==",
  "h": [{ "k": "content-type", "v": "YXBwbGljYXRpb24vanNvbg==" }],
  "t": 1787824800123
}
```

`k` is the record key and `v` the value, both base64. `h` is the headers, each with a plain-text
name and a base64 value, and it is omitted when there are none. `t` is the producer's CreateTime
in milliseconds, omitted when the record carries none. Base64 rather than a JSON string because
a Kafka value is not text: a protobuf or Avro payload is not valid UTF-8. `k` and `v` are always
written, including as JSON `null`, because a null key and a null value both mean something in
Kafka and neither is the same as an empty one.

The record's offset and partition are not in the envelope. They are Queen's, and writing them
into the payload would mean storing a number the broker assigns after the payload is built.

A payload that is not an envelope is served as itself: a Kafka consumer reading a queue that
native Queen producers write gets no key, the stored JSON document as the value, and the stored
timestamp. Recognition is strict, an object whose keys are a subset of `k`/`v`/`h`/`t` carrying
both `k` and `v`, with every field of the right type and every base64 field decodable, so
anything the facade did not write is passed through rather than misread.

## Throttling

A Cloud tenant that is frozen or over its rate cap gets a `429` from the proxy with a
`Retry-After`. Kafka has a field for exactly that, and the facade puts the same number in it on
the three responses in the hot path.

| Response | `throttle_time_ms` | What the partitions say |
| --- | --- | --- |
| Produce | `Retry-After` | `REQUEST_TIMED_OUT`, which is retriable in every client's produce path |
| Fetch | `Retry-After` | no records, error code 0, watermarks at `-1` |
| Metadata | `Retry-After` | `LEADER_NOT_AVAILABLE`, retriable |

Every Kafka client backs off on `throttle_time_ms` natively, without surfacing anything to the
application. The offset APIs are deliberately not throttled: a `429` there is answered
`COORDINATOR_NOT_AVAILABLE`, which the client retries on its own backoff, and a consumer that
sleeps its commit is a consumer that sleeps its whole poll loop.

## What an unsupported request gets

A version outside the advertised window, or an API key that is not in the table at all, closes
the connection with the reason in the facade's log and no response frame. That is what Apache
Kafka does with a request it cannot parse, and it is unreachable for any client that read the
`ApiVersions` answer, which is every client. The negotiation is the refusal.

Source: https://queenmq.com/reference/kafka/index.mdx