Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3027

Kafka Channel should clear offsets map after commit

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently the channel doesn't clear the TopicPartition / OffsetsAndMetadata map after a commit.

        Issue Links

          Activity

          Hide
          jholoman Jeff Holoman added a comment -

          Yes, I think this will address that as well. It has been some time since I fixed this and I haven't looked recently but the fundamental issue was a missing line clearing the offsets and so commits would be called on partitions no longer owned by the channel.

          Show
          jholoman Jeff Holoman added a comment - Yes, I think this will address that as well. It has been some time since I fixed this and I haven't looked recently but the fundamental issue was a missing line clearing the offsets and so commits would be called on partitions no longer owned by the channel.
          Hide
          jkandula Jitendra Kandula added a comment -

          Jeff Holoman
          I have noticed an issue in 1.7 with KafkaChannel. When a node is rebalanced and gets a different partitions to work with than what it has, offset commit request includes even old partitions resulting in sending stale offset values to Kafka.
          With this change to clear the map I think that is also resolved. Could you confirm?

          Show
          jkandula Jitendra Kandula added a comment - Jeff Holoman I have noticed an issue in 1.7 with KafkaChannel. When a node is rebalanced and gets a different partitions to work with than what it has, offset commit request includes even old partitions resulting in sending stale offset values to Kafka. With this change to clear the map I think that is also resolved. Could you confirm?
          Hide
          hudson Hudson added a comment -

          UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #234 (See https://builds.apache.org/job/Flume-trunk-hbase-1/234/)
          FLUME-3027. Change Kafka Channel to clear offsets map after commit (bessbd: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=fa1ee05af38bcf08ed18ff36d4284e68836a9054)

          • (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
          Show
          hudson Hudson added a comment - UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #234 (See https://builds.apache.org/job/Flume-trunk-hbase-1/234/ ) FLUME-3027 . Change Kafka Channel to clear offsets map after commit (bessbd: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=fa1ee05af38bcf08ed18ff36d4284e68836a9054 ) (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flume/pull/92

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flume/pull/92
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit fa1ee05af38bcf08ed18ff36d4284e68836a9054 in flume's branch refs/heads/trunk from Jeff Holoman
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=fa1ee05 ]

          FLUME-3027. Change Kafka Channel to clear offsets map after commit

          This change adds a call to clear the offsets map after a commit so as to avoid repeatedly committing already-committed offsets. Also updates various debug and trace log messages/calls to help with troubleshooting.

          This closes #92

          Reviewers: Attila Simon, Bessenyei Balázs Donát

          (Jeff Holoman via Bessenyei Balázs Donát)

          Show
          jira-bot ASF subversion and git services added a comment - Commit fa1ee05af38bcf08ed18ff36d4284e68836a9054 in flume's branch refs/heads/trunk from Jeff Holoman [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=fa1ee05 ] FLUME-3027 . Change Kafka Channel to clear offsets map after commit This change adds a call to clear the offsets map after a commit so as to avoid repeatedly committing already-committed offsets. Also updates various debug and trace log messages/calls to help with troubleshooting. This closes #92 Reviewers: Attila Simon, Bessenyei Balázs Donát (Jeff Holoman via Bessenyei Balázs Donát)
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user jholoman opened a pull request:

          https://github.com/apache/flume/pull/92

          FLUME-3027 Kafka Channel should clear offsets map after commit

          Added a call to clear the offsets map after a commit so as to avoid repeatedly committing already-committed offsets. Also updated various debug and trace log messages/calls to help with troubleshooting.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/jholoman/flume FLUME-3027-3

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flume/pull/92.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #92


          commit 5756f22f815ccf58050239b8676c47ac6567a006
          Author: jholoman <jeff.holoman@gmail.com>
          Date: 2016-11-23T15:52:06Z

          FLUME-3027 Kafka Channel should clear offsets map after commit


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user jholoman opened a pull request: https://github.com/apache/flume/pull/92 FLUME-3027 Kafka Channel should clear offsets map after commit Added a call to clear the offsets map after a commit so as to avoid repeatedly committing already-committed offsets. Also updated various debug and trace log messages/calls to help with troubleshooting. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jholoman/flume FLUME-3027 -3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/92.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #92 commit 5756f22f815ccf58050239b8676c47ac6567a006 Author: jholoman <jeff.holoman@gmail.com> Date: 2016-11-23T15:52:06Z FLUME-3027 Kafka Channel should clear offsets map after commit

            People

            • Assignee:
              jholoman Jeff Holoman
              Reporter:
              jholoman Jeff Holoman
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development