[KAFKA-3937] Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.8.2.2, 0.9.0.1, 0.10.0.0
• Fix Version/s: 0.10.1.0
• Component/s: clients
• Environment: Linux, latest oracle java-8
• Assignee: William Yu
• Reporter: Tom Crayford
• Reviewer: Ismael Juma
• Created: 08/Jul/16 14:18
• Updated: 12/Apr/17 23:21
• Resolved: 23/Aug/16 22:50

Description

In https://issues.apache.org/jira/browse/KAFKA-3933, we discovered that brokers can crash when performing log recovery, as they leak native memory whilst decompressing compressed segments, and that native memory isn't cleaned up rapidly enough by garbage collection and finalizers. The work to fix that in the brokers is taking part in https://github.com/apache/kafka/pull/1598. As part of that PR, Ismael Juma asked me to fix similar issues in the client. Rather than have one large PR that fixes everything, I'd rather break this work up into seperate things, so I'm filing this JIRA to track the followup work. I should get to a PR on this at some point relatively soon, once the other PR has landed.

Issue Links

• is related to: FLINK-6301 Flink KafkaConnector09 leaks memory on reading compressed messages due to a Kafka consumer bug
• links to: GitHub Pull Request #1743
• links to: GitHub Pull Request #1760