[KAFKA-4725] Kafka broker fails due to OOM when producer exceeds throttling quota for extended periods of time

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.10.1.1
• Fix Version/s: 0.10.2.0
• Component/s: core, producer
• Labels: reliability
• Environment: Ubuntu Trusty (14.04.5), Oracle JDK 8
• Assignee: Unassigned
• Reporter: Jeff Chao
• Reviewer: Jun Rao
• Created: 02/Feb/17 18:57
• Updated: 07/Feb/17 23:13
• Resolved: 07/Feb/17 23:13

Description

Steps to Reproduce:

1. Create a non-compacted topic with 1 partition
2. Set a produce quota of 512 KB/s
3. Send messages at 20 MB/s
4. Observe heap memory growth as time progresses

Investigation:

While running performance tests with a user configured with a produce quota, we found that the lead broker serving the requests would exhaust heap memory if the producer sustained a inbound request throughput greater than the produce quota.

Upon further investigation, we took a heap dump from that broker process and discovered the ThrottledResponse object has a indirect reference to the byte[] holding the messages associated with the ProduceRequest.

We're happy contributing a patch but in the meantime wanted to first raise the issue and get feedback from the community.

Issue Links

• links to: GitHub Pull Request #2496