Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.5.1
    • Fix Version/s: v1.7.0
    • Component/s: Sinks+Sources
    • Labels:

      Description

      Reported here: https://github.com/harishreedharan/flume/issues/1

      "I tried flume-ng-kafka-sink and it worked fine. But I noticed that the cpu utilization stay at 100% and never dropped down all the time even at the time the channel is empty.

      I looked into the source code and found that "process" function in KafkaSink always return Status.READY even if no events available in channel. That causes the SinkRunner keep running achieving event from channel and get nothing.

      Do we need to change to return Status.BACKOFF in "process" function in KafkaSink when it notices that there is no events processed in current round? So that the SinkRunner has a chance to take a rest when there is no event in channel. If this proposal feasible, function "testEmptyChannel" in TestKafkaSink also need to be changed. "

      1. FLUME-2632-0.patch
        1 kB
        Ashish Paliwal

        Issue Links

          Activity

          Hide
          Ashish Paliwal added a comment - - edited

          Gwen Shapira Can you please have a look at it, minor fix.

          Show
          Ashish Paliwal added a comment - - edited Gwen Shapira Can you please have a look at it, minor fix.
          Hide
          Ralph Goers added a comment -

          This was reviewed quite a while ago. Any reason it hasn't been applied? I am seeing the same issue.

          Show
          Ralph Goers added a comment - This was reviewed quite a while ago. Any reason it hasn't been applied? I am seeing the same issue.
          Hide
          Ashish Paliwal added a comment -

          Johny Rufus/Hari Shreedharan/Gwen Shapira Can anyone of you please have a look and do the needful

          Show
          Ashish Paliwal added a comment - Johny Rufus / Hari Shreedharan / Gwen Shapira Can anyone of you please have a look and do the needful
          Hide
          Mike Lerch added a comment -

          We're using the KafkaSink in a larger production environment, and this bug eats up 6 cores at 100% on each instance until the first messages arrive. Would be nice to see this fixed.

          Show
          Mike Lerch added a comment - We're using the KafkaSink in a larger production environment, and this bug eats up 6 cores at 100% on each instance until the first messages arrive. Would be nice to see this fixed.
          Hide
          Johny Rufus added a comment -

          +1,
          Looks like I +1ed this on the review request long time back, will run tests and commit

          Show
          Johny Rufus added a comment - +1, Looks like I +1ed this on the review request long time back, will run tests and commit
          Hide
          Ralph Goers added a comment -

          I am now running this in production in my environment and it works just fine. I'd be happy to commit it but that seems a bit strange since I didn't author it and wasn't part of the review.

          Show
          Ralph Goers added a comment - I am now running this in production in my environment and it works just fine. I'd be happy to commit it but that seems a bit strange since I didn't author it and wasn't part of the review.
          Hide
          Ashish Paliwal added a comment -

          It's a minor fix. Since I authored it, I cannot commit the changes. If its working fine, lets get it out of way

          Show
          Ashish Paliwal added a comment - It's a minor fix. Since I authored it, I cannot commit the changes. If its working fine, lets get it out of way
          Hide
          ASF subversion and git services added a comment -

          Commit d6bf08b54e467a6bdc6a5fc0edd41c51200e9da1 in flume's branch refs/heads/trunk from Johny Rufus
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d6bf08b ]

          FLUME-2632: High CPU on KafkaSink

          (Ashish Paliwal via Johny Rufus)

          Show
          ASF subversion and git services added a comment - Commit d6bf08b54e467a6bdc6a5fc0edd41c51200e9da1 in flume's branch refs/heads/trunk from Johny Rufus [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d6bf08b ] FLUME-2632 : High CPU on KafkaSink (Ashish Paliwal via Johny Rufus)
          Hide
          ASF subversion and git services added a comment -

          Commit bf2495047f5cebb26f26f94904831c52057d129d in flume's branch refs/heads/flume-1.7 from Johny Rufus
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=bf24950 ]

          FLUME-2632: High CPU on KafkaSink

          (Ashish Paliwal via Johny Rufus)

          Show
          ASF subversion and git services added a comment - Commit bf2495047f5cebb26f26f94904831c52057d129d in flume's branch refs/heads/flume-1.7 from Johny Rufus [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=bf24950 ] FLUME-2632 : High CPU on KafkaSink (Ashish Paliwal via Johny Rufus)
          Hide
          Johny Rufus added a comment -

          Thanks Ashish for the patch !!

          Show
          Johny Rufus added a comment - Thanks Ashish for the patch !!
          Hide
          Ralph Goers added a comment -

          Yeah - I forgot Flume had that rule. Totally different than every other project I work on. I guess that's why I never commit.

          Show
          Ralph Goers added a comment - Yeah - I forgot Flume had that rule. Totally different than every other project I work on. I guess that's why I never commit.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Flume-trunk-hbase-1 #130 (See https://builds.apache.org/job/Flume-trunk-hbase-1/130/)
          FLUME-2632: High CPU on KafkaSink (johnyrufus: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=d6bf08b54e467a6bdc6a5fc0edd41c51200e9da1)

          • flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
          • flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-1 #130 (See https://builds.apache.org/job/Flume-trunk-hbase-1/130/ ) FLUME-2632 : High CPU on KafkaSink (johnyrufus: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=d6bf08b54e467a6bdc6a5fc0edd41c51200e9da1 ) flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java

            People

            • Assignee:
              Ashish Paliwal
              Reporter:
              Gwen Shapira
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development