Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3031

sequence source should reset its counter for event body on channel exception

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.0
    • Fix Version/s: 1.8.0
    • Component/s: Sinks+Sources, Test
    • Labels:
      None

      Description

      SequenceGeneratorSource uses a counter to construct the body of the generated Events which counter is not reseted when writing event to channel failed. This can lead to a situation that the total number of unique events at destination (if deduplication relies on msg body) is bigger than the totalEvents configuration parameter due to the retries.

      Fix should make sure that number of events at destination after filtering out the duplicates is equal to the configured value.

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #235 (See https://builds.apache.org/job/Flume-trunk-hbase-1/235/)
          FLUME-3031. Change sequence source to reset its counter for event body (bessbd: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=2ff2dbbd13db5de747c654ef132c98941cdd45dc)

          • (edit) flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java
          • (edit) flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java
          • (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst
          Show
          hudson Hudson added a comment - UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #235 (See https://builds.apache.org/job/Flume-trunk-hbase-1/235/ ) FLUME-3031 . Change sequence source to reset its counter for event body (bessbd: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=2ff2dbbd13db5de747c654ef132c98941cdd45dc ) (edit) flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java (edit) flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flume/pull/90

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flume/pull/90
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 2ff2dbbd13db5de747c654ef132c98941cdd45dc in flume's branch refs/heads/trunk from Attila Simon
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=2ff2dbb ]

          FLUME-3031. Change sequence source to reset its counter for event body on channel exception

          This patch improves rollbacks for the sequence source.
          Also, it updates tests and user documentation accordingly.

          This closes #90

          Reviewers: Denes Arvay, Jeff Holoman, Bessenyei Balázs Donát

          (Attila Simon via Bessenyei Balázs Donát)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 2ff2dbbd13db5de747c654ef132c98941cdd45dc in flume's branch refs/heads/trunk from Attila Simon [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=2ff2dbb ] FLUME-3031 . Change sequence source to reset its counter for event body on channel exception This patch improves rollbacks for the sequence source. Also, it updates tests and user documentation accordingly. This closes #90 Reviewers: Denes Arvay, Jeff Holoman, Bessenyei Balázs Donát (Attila Simon via Bessenyei Balázs Donát)
          Hide
          sati Attila Simon added a comment -

          patch and code review is in a github pull request

          Show
          sati Attila Simon added a comment - patch and code review is in a github pull request
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user satiatcloudera opened a pull request:

          https://github.com/apache/flume/pull/90

          FLUME-3031: sequence source should reset its counter for event body on channel exception

          added code fix for event body generation
          updated user guide
          reworked testing: removed original tests and introduced three new one to extend coverage

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/satiatcloudera/flume sequencesource

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flume/pull/90.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #90



          Show
          githubbot ASF GitHub Bot added a comment - GitHub user satiatcloudera opened a pull request: https://github.com/apache/flume/pull/90 FLUME-3031 : sequence source should reset its counter for event body on channel exception added code fix for event body generation updated user guide reworked testing: removed original tests and introduced three new one to extend coverage You can merge this pull request into a Git repository by running: $ git pull https://github.com/satiatcloudera/flume sequencesource Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/90.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #90

            People

            • Assignee:
              sati Attila Simon
              Reporter:
              sati Attila Simon
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development