Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2920

Kafka Channel Should Not Commit Offsets When Stopping

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.7.0
    • Component/s: Channel
    • Labels:
      None

      Description

      The Kafka channel commits the consumer offsets when shutting down (via stop() -> decommissionConsumerAndRecords())

      This can lead to data loss if the channel is shut down while messages have been fetched in a transaction but the transaction has not yet been committed.

      The only time that the offsets should be committed is when a transaction is complete.

      1. FLUME-2920.patch
        6 kB
        Kevin Conaway

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user kevinconaway opened a pull request:

          https://github.com/apache/flume/pull/51

          FLUME-2920 Don't commit offsets when shutting down consumer

          Also removed some unused test code in `TestKafkaChanel`

          @jarcec is this something that you can review?

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/kevinconaway/flume flume-2920

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flume/pull/51.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #51


          commit fc31f14abf7b46680ec94014dd877e2e1f01958a
          Author: Kevin Conaway <kevin.conaway@walmart.com>
          Date: 2016-06-09T16:32:35Z

          FLUME-2920 Don't commit offsets when shutting down consumer. Remove some dead test code


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user kevinconaway opened a pull request: https://github.com/apache/flume/pull/51 FLUME-2920 Don't commit offsets when shutting down consumer Also removed some unused test code in `TestKafkaChanel` @jarcec is this something that you can review? You can merge this pull request into a Git repository by running: $ git pull https://github.com/kevinconaway/flume flume-2920 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/51.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #51 commit fc31f14abf7b46680ec94014dd877e2e1f01958a Author: Kevin Conaway <kevin.conaway@walmart.com> Date: 2016-06-09T16:32:35Z FLUME-2920 Don't commit offsets when shutting down consumer. Remove some dead test code
          Show
          kevinconaway Kevin Conaway added a comment - Attaching patch via https://patch-diff.githubusercontent.com/raw/apache/flume/pull/51.patch
          Hide
          jholoman Jeff Holoman added a comment -

          +1

          Show
          jholoman Jeff Holoman added a comment - +1
          Hide
          granthenke Grant Henke added a comment -

          +1

          Show
          granthenke Grant Henke added a comment - +1
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f46bee03eb2c9094aa192dd6b82a494cb26eef35 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=f46bee0 ]

          FLUME-2920: Kafka Channel Should Not Commit Offsets When Stopping

          (Kevin Conaway via Jarek Jarcec Cecho)

          Show
          jira-bot ASF subversion and git services added a comment - Commit f46bee03eb2c9094aa192dd6b82a494cb26eef35 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=f46bee0 ] FLUME-2920 : Kafka Channel Should Not Commit Offsets When Stopping (Kevin Conaway via Jarek Jarcec Cecho)
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Thank you for your contribution Kevin Conaway and for the review Jeff Holoman / Grant Henke!

          Show
          jarcec Jarek Jarcec Cecho added a comment - Thank you for your contribution Kevin Conaway and for the review Jeff Holoman / Grant Henke !
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kevinconaway closed the pull request at:

          https://github.com/apache/flume/pull/51

          Show
          githubbot ASF GitHub Bot added a comment - Github user kevinconaway closed the pull request at: https://github.com/apache/flume/pull/51
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Flume-trunk-hbase-1 #160 (See https://builds.apache.org/job/Flume-trunk-hbase-1/160/)
          FLUME-2920: Kafka Channel Should Not Commit Offsets When Stopping (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=f46bee03eb2c9094aa192dd6b82a494cb26eef35)

          • flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
          • flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-1 #160 (See https://builds.apache.org/job/Flume-trunk-hbase-1/160/ ) FLUME-2920 : Kafka Channel Should Not Commit Offsets When Stopping (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=f46bee03eb2c9094aa192dd6b82a494cb26eef35 ) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java

            People

            • Assignee:
              kevinconaway Kevin Conaway
              Reporter:
              kevinconaway Kevin Conaway
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development