Flume
  1. Flume
  2. FLUME-1490

Option to limit number of events sent in Stress source

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.3.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Currently the Stress source (used for testing purposes) has a 'size' prop to control the size of each event. It would be useful to also have a 'count' prop to control the number of events sent. When that limit is reached, the source's process() method can simply return Status.BACKOFF

      1. FLUME-1490.v1.patch.txt
        6 kB
        NO NAME
      2. FLUME-1490.v2.patch.txt
        7 kB
        NO NAME

        Activity

        Hide
        Hudson added a comment -

        Integrated in flume-trunk #283 (See https://builds.apache.org/job/flume-trunk/283/)
        FLUME-1490. Option to limit number of events sent in Stress source. (Revision 5da69a2ca8560b6dec660e927d0d7e0b062207ed)

        Result = SUCCESS
        will : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=5da69a2ca8560b6dec660e927d0d7e0b062207ed
        Files :

        • flume-ng-core/pom.xml
        • flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java
        • flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java
        • flume-ng-core/src/test/java/org/apache/flume/source/TestStressSource.java
        Show
        Hudson added a comment - Integrated in flume-trunk #283 (See https://builds.apache.org/job/flume-trunk/283/ ) FLUME-1490 . Option to limit number of events sent in Stress source. (Revision 5da69a2ca8560b6dec660e927d0d7e0b062207ed) Result = SUCCESS will : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=5da69a2ca8560b6dec660e927d0d7e0b062207ed Files : flume-ng-core/pom.xml flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java flume-ng-core/src/test/java/org/apache/flume/source/TestStressSource.java
        Hide
        NO NAME added a comment -

        Great! For posterity I'm including the patch that was actually committed.

        Show
        NO NAME added a comment - Great! For posterity I'm including the patch that was actually committed.
        Hide
        Will McQueen added a comment -

        Patch committed (5da69a2ca8560b6dec660e927d0d7e0b062207ed)
        Thanks, Patrick!

        Show
        Will McQueen added a comment - Patch committed (5da69a2ca8560b6dec660e927d0d7e0b062207ed) Thanks, Patrick!
        Hide
        NO NAME added a comment -

        This patch implements a count limit, both for overall (successful + failed) events and for successful events. Seems like both could be useful in testing. Includes unit test and a bit of code clean-up.

        Show
        NO NAME added a comment - This patch implements a count limit, both for overall (successful + failed) events and for successful events. Seems like both could be useful in testing. Includes unit test and a bit of code clean-up.
        Hide
        Will McQueen added a comment -

        Hi Patrick,

        As an example with the 'count' feature, a test can limit the stress test to 1M events so that later it can confirm whether exactly 1M events were received at the sink... and if not then whether there were more (duplicates) or less (sink is still catching up).

        Show
        Will McQueen added a comment - Hi Patrick, As an example with the 'count' feature, a test can limit the stress test to 1M events so that later it can confirm whether exactly 1M events were received at the sink... and if not then whether there were more (duplicates) or less (sink is still catching up).
        Hide
        NO NAME added a comment -

        I had started a discussion about a load-generating source earlier that was configurable. My thought was that rather than a fixed event count, you really wanted something that generates events at a fixed rate. For instance, to mimic a given rate you'd expect from your data source.

        Will - what is the use case for having a fixed count? Just to limit the length of an experiment?

        Show
        NO NAME added a comment - I had started a discussion about a load-generating source earlier that was configurable. My thought was that rather than a fixed event count, you really wanted something that generates events at a fixed rate . For instance, to mimic a given rate you'd expect from your data source. Will - what is the use case for having a fixed count? Just to limit the length of an experiment?

          People

          • Assignee:
            NO NAME
            Reporter:
            Will McQueen
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development