Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.1
    • Fix Version/s: 1.7.0
    • Component/s: Sinks+Sources
    • Labels:

      Description

      Reported here: https://github.com/harishreedharan/flume/issues/1

      "I tried flume-ng-kafka-sink and it worked fine. But I noticed that the cpu utilization stay at 100% and never dropped down all the time even at the time the channel is empty.

      I looked into the source code and found that "process" function in KafkaSink always return Status.READY even if no events available in channel. That causes the SinkRunner keep running achieving event from channel and get nothing.

      Do we need to change to return Status.BACKOFF in "process" function in KafkaSink when it notices that there is no events processed in current round? So that the SinkRunner has a chance to take a rest when there is no event in channel. If this proposal feasible, function "testEmptyChannel" in TestKafkaSink also need to be changed. "

      1. FLUME-2632-0.patch
        1 kB
        Ashish Paliwal

        Issue Links

          Activity

          Hide
          paliwalashish Ashish Paliwal added a comment - - edited

          Gwen Shapira Can you please have a look at it, minor fix.

          Show
          paliwalashish Ashish Paliwal added a comment - - edited Gwen Shapira Can you please have a look at it, minor fix.
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          This was reviewed quite a while ago. Any reason it hasn't been applied? I am seeing the same issue.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - This was reviewed quite a while ago. Any reason it hasn't been applied? I am seeing the same issue.
          Hide
          paliwalashish Ashish Paliwal added a comment -

          Johny Rufus/Hari Shreedharan/Gwen Shapira Can anyone of you please have a look and do the needful

          Show
          paliwalashish Ashish Paliwal added a comment - Johny Rufus / Hari Shreedharan / Gwen Shapira Can anyone of you please have a look and do the needful
          Hide
          mlerch223 Mike Lerch added a comment -

          We're using the KafkaSink in a larger production environment, and this bug eats up 6 cores at 100% on each instance until the first messages arrive. Would be nice to see this fixed.

          Show
          mlerch223 Mike Lerch added a comment - We're using the KafkaSink in a larger production environment, and this bug eats up 6 cores at 100% on each instance until the first messages arrive. Would be nice to see this fixed.
          Hide
          jrufus Johny Rufus added a comment -

          +1,
          Looks like I +1ed this on the review request long time back, will run tests and commit

          Show
          jrufus Johny Rufus added a comment - +1, Looks like I +1ed this on the review request long time back, will run tests and commit
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          I am now running this in production in my environment and it works just fine. I'd be happy to commit it but that seems a bit strange since I didn't author it and wasn't part of the review.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - I am now running this in production in my environment and it works just fine. I'd be happy to commit it but that seems a bit strange since I didn't author it and wasn't part of the review.
          Hide
          paliwalashish Ashish Paliwal added a comment -

          It's a minor fix. Since I authored it, I cannot commit the changes. If its working fine, lets get it out of way

          Show
          paliwalashish Ashish Paliwal added a comment - It's a minor fix. Since I authored it, I cannot commit the changes. If its working fine, lets get it out of way
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d6bf08b54e467a6bdc6a5fc0edd41c51200e9da1 in flume's branch refs/heads/trunk from Johny Rufus
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d6bf08b ]

          FLUME-2632: High CPU on KafkaSink

          (Ashish Paliwal via Johny Rufus)

          Show
          jira-bot ASF subversion and git services added a comment - Commit d6bf08b54e467a6bdc6a5fc0edd41c51200e9da1 in flume's branch refs/heads/trunk from Johny Rufus [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d6bf08b ] FLUME-2632 : High CPU on KafkaSink (Ashish Paliwal via Johny Rufus)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit bf2495047f5cebb26f26f94904831c52057d129d in flume's branch refs/heads/flume-1.7 from Johny Rufus
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=bf24950 ]

          FLUME-2632: High CPU on KafkaSink

          (Ashish Paliwal via Johny Rufus)

          Show
          jira-bot ASF subversion and git services added a comment - Commit bf2495047f5cebb26f26f94904831c52057d129d in flume's branch refs/heads/flume-1.7 from Johny Rufus [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=bf24950 ] FLUME-2632 : High CPU on KafkaSink (Ashish Paliwal via Johny Rufus)
          Hide
          jrufus Johny Rufus added a comment -

          Thanks Ashish for the patch !!

          Show
          jrufus Johny Rufus added a comment - Thanks Ashish for the patch !!
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          Yeah - I forgot Flume had that rule. Totally different than every other project I work on. I guess that's why I never commit.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - Yeah - I forgot Flume had that rule. Totally different than every other project I work on. I guess that's why I never commit.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Flume-trunk-hbase-1 #130 (See https://builds.apache.org/job/Flume-trunk-hbase-1/130/)
          FLUME-2632: High CPU on KafkaSink (johnyrufus: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=d6bf08b54e467a6bdc6a5fc0edd41c51200e9da1)

          • flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
          • flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-1 #130 (See https://builds.apache.org/job/Flume-trunk-hbase-1/130/ ) FLUME-2632 : High CPU on KafkaSink (johnyrufus: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=d6bf08b54e467a6bdc6a5fc0edd41c51200e9da1 ) flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java

            People

            • Assignee:
              paliwalashish Ashish Paliwal
              Reporter:
              gwenshap Gwen Shapira
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development