Kafka
  1. Kafka
  2. KAFKA-3937

Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.8.2.2, 0.9.0.1, 0.10.0.0
    • Fix Version/s: 0.10.1.0
    • Component/s: clients
    • Labels:
      None
    • Environment:
      Linux, latest oracle java-8

      Description

      In https://issues.apache.org/jira/browse/KAFKA-3933, we discovered that brokers can crash when performing log recovery, as they leak native memory whilst decompressing compressed segments, and that native memory isn't cleaned up rapidly enough by garbage collection and finalizers. The work to fix that in the brokers is taking part in https://github.com/apache/kafka/pull/1598. As part of that PR, Ismael Juma asked me to fix similar issues in the client. Rather than have one large PR that fixes everything, I'd rather break this work up into seperate things, so I'm filing this JIRA to track the followup work. I should get to a PR on this at some point relatively soon, once the other PR has landed.

        Issue Links

          Activity

          Hide
          William Yu added a comment -

          You're referring to the Kafka consumer when you say 'kafka client' correct? I believe this is the bug ticket I was looking for to confirm the behavior that I'm seeing from my consumer application. It looks to be leaking native memory while consuming from a topic as my java -Xmx setting is set for 10gb, but the resident memory is showing 23gb usage.

          Show
          William Yu added a comment - You're referring to the Kafka consumer when you say 'kafka client' correct? I believe this is the bug ticket I was looking for to confirm the behavior that I'm seeing from my consumer application. It looks to be leaking native memory while consuming from a topic as my java -Xmx setting is set for 10gb, but the resident memory is showing 23gb usage.
          Hide
          Ismael Juma added a comment -

          William Yu, is your topic compressed?

          Show
          Ismael Juma added a comment - William Yu , is your topic compressed?
          Hide
          William Yu added a comment - - edited

          Ismael Juma, yes the topic is compressed, our producer is setup with the following property: "compression.type=gzip"

          Show
          William Yu added a comment - - edited Ismael Juma , yes the topic is compressed, our producer is setup with the following property: "compression.type=gzip"
          Hide
          Ismael Juma added a comment -

          Seems like this one then. It's the first time we're hearing about someone experiencing it, so it would be good to fix the issue and then have you test it. Tom Crayford, do you think you'll have time to look into this soon?

          Show
          Ismael Juma added a comment - Seems like this one then. It's the first time we're hearing about someone experiencing it, so it would be good to fix the issue and then have you test it. Tom Crayford , do you think you'll have time to look into this soon?
          Hide
          William Yu added a comment -

          FYI: My brokers and client are both on kafka: 0.9.0.1.

          Show
          William Yu added a comment - FYI: My brokers and client are both on kafka: 0.9.0.1.
          Hide
          ASF GitHub Bot added a comment -

          GitHub user wiyu opened a pull request:

          https://github.com/apache/kafka/pull/1743

          KAFKA-3937: Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages

          @ijuma - Creating this PR against 0.9.0 as this is what we're using in prod. I can modify this for trunk if the logic solution looks good.

          I modified the record iterator to inherit from closable and implemented the method to call streams.close(). The stream will now be closed when we are done traversing the inner iterator. I had to move the `try {} catch{}` out of the `if` portion to deal with IOException coming from `innerDone()`.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/wiyu/kafka compressor_memory_leak_in_fetcher

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/kafka/pull/1743.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1743


          commit b56ec1563e34d40c338224970c6b8cd3802e4dcd
          Author: William Yu <wyu@unified.com>
          Date: 2016-08-16T14:09:28Z

          KAFKA-3937: Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages


          Show
          ASF GitHub Bot added a comment - GitHub user wiyu opened a pull request: https://github.com/apache/kafka/pull/1743 KAFKA-3937 : Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages @ijuma - Creating this PR against 0.9.0 as this is what we're using in prod. I can modify this for trunk if the logic solution looks good. I modified the record iterator to inherit from closable and implemented the method to call streams.close(). The stream will now be closed when we are done traversing the inner iterator. I had to move the `try {} catch{}` out of the `if` portion to deal with IOException coming from `innerDone()`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wiyu/kafka compressor_memory_leak_in_fetcher Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1743.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1743 commit b56ec1563e34d40c338224970c6b8cd3802e4dcd Author: William Yu <wyu@unified.com> Date: 2016-08-16T14:09:28Z KAFKA-3937 : Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages
          Hide
          William Yu added a comment -

          Ismael Juma Took a stab at making the fix. I made it against the 0.9.0 branch as that is what my clients and brokers are running as. I tested it in my environment and i saw native memory was not going above the java -Xmx setting as the streams were being closed when the RecordIterator completed.

          Show
          William Yu added a comment - Ismael Juma Took a stab at making the fix. I made it against the 0.9.0 branch as that is what my clients and brokers are running as. I tested it in my environment and i saw native memory was not going above the java -Xmx setting as the streams were being closed when the RecordIterator completed.
          Hide
          ASF GitHub Bot added a comment -

          Github user wiyu closed the pull request at:

          https://github.com/apache/kafka/pull/1743

          Show
          ASF GitHub Bot added a comment - Github user wiyu closed the pull request at: https://github.com/apache/kafka/pull/1743
          Hide
          ASF GitHub Bot added a comment -

          GitHub user wiyu opened a pull request:

          https://github.com/apache/kafka/pull/1760

          KAFKA-3937: Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages

          @ijuma - Making the change against trunk based on your suggestions to have the stream closing handled in the private RecordIterator constructor which I understand is only to be used only if the block of message(s) are compressed.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/wiyu/kafka compressor_memory_leak_in_fetcher

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/kafka/pull/1760.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1760


          commit 56451a38399875ae9d1ccdb2f873de88eae6f940
          Author: William Yu <wyu@unified.com>
          Date: 2016-08-18T17:25:10Z

          KAFKA-3937: Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages


          Show
          ASF GitHub Bot added a comment - GitHub user wiyu opened a pull request: https://github.com/apache/kafka/pull/1760 KAFKA-3937 : Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages @ijuma - Making the change against trunk based on your suggestions to have the stream closing handled in the private RecordIterator constructor which I understand is only to be used only if the block of message(s) are compressed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wiyu/kafka compressor_memory_leak_in_fetcher Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1760.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1760 commit 56451a38399875ae9d1ccdb2f873de88eae6f940 Author: William Yu <wyu@unified.com> Date: 2016-08-18T17:25:10Z KAFKA-3937 : Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages
          Hide
          William Yu added a comment -

          Ismael Juma I've created a new PR based on your comments. Let me know what you think. thanks

          Show
          William Yu added a comment - Ismael Juma I've created a new PR based on your comments. Let me know what you think. thanks
          Hide
          ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/kafka/pull/1760

          Show
          ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1760
          Hide
          William Yu added a comment -

          Ismael Juma So, we don't plan on upgrading to kafka 0.10.x anytime in the near future as we still have to get all our services onto the kafka 0.9.x client from 0.8.2. Is Kafka 0.9.x still being patched and released? If it is can, then I can submit a patch for 9. Thanks.

          Show
          William Yu added a comment - Ismael Juma So, we don't plan on upgrading to kafka 0.10.x anytime in the near future as we still have to get all our services onto the kafka 0.9.x client from 0.8.2. Is Kafka 0.9.x still being patched and released? If it is can, then I can submit a patch for 9. Thanks.
          Hide
          Ismael Juma added a comment -

          William Yu, we occasionally commit to the `0.9.0` branch, but there is no set plan for a 0.9.0.2 release. It doesn't mean it won't happen, but resources at the moment are mostly focused on 0.10.1.0.

          Show
          Ismael Juma added a comment - William Yu , we occasionally commit to the `0.9.0` branch, but there is no set plan for a 0.9.0.2 release. It doesn't mean it won't happen, but resources at the moment are mostly focused on 0.10.1.0.

            People

            • Assignee:
              William Yu
              Reporter:
              Tom Crayford
              Reviewer:
              Ismael Juma
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development