[KAFKA-15063] Throttle number of active PIDs

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.0, 3.1.0, 3.0.0, 3.2.0, 3.3, 3.4.0
• Component/s: core, producer
• Assignee: Unassigned
• Reporter: Omnia Ibrahim
• Created: 06/Jun/23 14:49
• Updated: 06/Jun/23 15:08

Description

Ticket to track KIP-936. Since KIP-679 idempotent producers became the default in Kafka as a result of this all producer instances will be assigned PID. The increase of the number of PIDs stored in Kafka brokers by ProducerStateManager exposes the broker to OOM errors if it has a high number of producers, a rogue or misconfigured client(s).

The broker is still exposed to OOM even after KIP-854 introduced a separate config to expire PID from transaction IDs if there is a high number of PID before producer.id.expiration.ms is exceeded.

As a result of this, the broker will keep experiencing OOM and become offline. The only way to recover from this is to increase the heap.  

 

KIP-936 is proposing throttling the number of PIDs per KafkaPrincipal

See the KIP-936 details here  https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs