Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2495

Kafka Source may miss events when channel is not available

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Because the Kafka consumer itself tracks offsets, and we don't restart the consumer when we get channel errors, the consumer will skip messages that we couldn't write to channel, even though we did not advance offset in Zookeeper.

      1. FLUME-2495.0.patch
        6 kB
        Gwen Shapira
      2. FLUME-2495.1.patch
        7 kB
        Gwen Shapira
      3. FLUME-2495.2.patch
        8 kB
        Gwen Shapira

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Flume-trunk-hbase-98 #27 (See https://builds.apache.org/job/Flume-trunk-hbase-98/27/)
        FLUME-2495. Kafka Source may miss events when channel is not available (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9f0abea027c408669977aa34cd32f5e525bb508b)

        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
        • flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-98 #27 (See https://builds.apache.org/job/Flume-trunk-hbase-98/27/ ) FLUME-2495 . Kafka Source may miss events when channel is not available (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9f0abea027c408669977aa34cd32f5e525bb508b ) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in flume-trunk #667 (See https://builds.apache.org/job/flume-trunk/667/)
        FLUME-2495. Kafka Source may miss events when channel is not available (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9f0abea027c408669977aa34cd32f5e525bb508b)

        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
        • flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in flume-trunk #667 (See https://builds.apache.org/job/flume-trunk/667/ ) FLUME-2495 . Kafka Source may miss events when channel is not available (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9f0abea027c408669977aa34cd32f5e525bb508b ) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Committed! Thanks Gwen!

        Show
        hshreedharan Hari Shreedharan added a comment - Committed! Thanks Gwen!
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 6f3cb541c80b0fb3fe3faf445b52745a588d4993 in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=6f3cb54 ]

        FLUME-2495. Kafka Source may miss events when channel is not available

        (Gwen Shapira via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 6f3cb541c80b0fb3fe3faf445b52745a588d4993 in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=6f3cb54 ] FLUME-2495 . Kafka Source may miss events when channel is not available (Gwen Shapira via Hari)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 9f0abea027c408669977aa34cd32f5e525bb508b in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9f0abea ]

        FLUME-2495. Kafka Source may miss events when channel is not available

        (Gwen Shapira via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 9f0abea027c408669977aa34cd32f5e525bb508b in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9f0abea ] FLUME-2495 . Kafka Source may miss events when channel is not available (Gwen Shapira via Hari)
        Hide
        hshreedharan Hari Shreedharan added a comment -

        +1. Running tests and committing

        Show
        hshreedharan Hari Shreedharan added a comment - +1. Running tests and committing
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Correct. If the source dies, it is pretty much an error condition and Flume already causes duplicates in case of error. So it is fine.

        Show
        hshreedharan Hari Shreedharan added a comment - Correct. If the source dies, it is pretty much an error condition and Flume already causes duplicates in case of error. So it is fine.
        Hide
        gwenshap Gwen Shapira added a comment -

        ok, I think I see what you mean. Basically, clear list immediately after writing to channel. Then if we fail to commit to Kafka, most of the time we will not have duplicates (unless the source actually dies)

        Show
        gwenshap Gwen Shapira added a comment - ok, I think I see what you mean. Basically, clear list immediately after writing to channel. Then if we fail to commit to Kafka, most of the time we will not have duplicates (unless the source actually dies)
        Hide
        hshreedharan Hari Shreedharan added a comment -

        commitOffsets could throw an exception, correct? This fix would cause a lot of duplicates if commitOffsets throws an exception even if the transaction was successfully committed. Maybe the list should be cleared before the attempt to commit?

        Show
        hshreedharan Hari Shreedharan added a comment - commitOffsets could throw an exception, correct? This fix would cause a lot of duplicates if commitOffsets throws an exception even if the transaction was successfully committed. Maybe the list should be cleared before the attempt to commit?
        Hide
        gwenshap Gwen Shapira added a comment -

        Thanks a lot for the catch, Hari Shreedharan.
        Attaching a fixed version + additional tests.

        Show
        gwenshap Gwen Shapira added a comment - Thanks a lot for the catch, Hari Shreedharan . Attaching a fixed version + additional tests.
        Hide
        hshreedharan Hari Shreedharan added a comment - - edited

        There is a bug in this patch - you'd never clear the list if autocommit is enabled - this means that the list would grow indefinitely causing duplicates.

        Show
        hshreedharan Hari Shreedharan added a comment - - edited There is a bug in this patch - you'd never clear the list if autocommit is enabled - this means that the list would grow indefinitely causing duplicates.

          People

          • Assignee:
            gwenshap Gwen Shapira
            Reporter:
            gwenshap Gwen Shapira
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development