Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3937

Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 0.8.2.2, 0.9.0.1, 0.10.0.0
    • 0.10.1.0
    • clients
    • None
    • Linux, latest oracle java-8

    Description

      In https://issues.apache.org/jira/browse/KAFKA-3933, we discovered that brokers can crash when performing log recovery, as they leak native memory whilst decompressing compressed segments, and that native memory isn't cleaned up rapidly enough by garbage collection and finalizers. The work to fix that in the brokers is taking part in https://github.com/apache/kafka/pull/1598. As part of that PR, Ismael Juma asked me to fix similar issues in the client. Rather than have one large PR that fixes everything, I'd rather break this work up into seperate things, so I'm filing this JIRA to track the followup work. I should get to a PR on this at some point relatively soon, once the other PR has landed.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            williamyu William Yu
            tcrayford-heroku Tom Crayford
            Ismael Juma Ismael Juma
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment