Flume
  1. Flume
  2. FLUME-867

Pollable source and sink runners should reduce poll interval after several BACKOFFs

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: NG alpha 1
    • Fix Version/s: v1.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Pollable

      {Source,Sink}

      Runner should both reduce their polling frequency after receiving several BACKOFF statuses from the underlying source or sink.

        Activity

        E. Sammer created issue -
        Hide
        E. Sammer added a comment -

        Review up.

        Show
        E. Sammer added a comment - Review up.
        E. Sammer made changes -
        Field Original Value New Value
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2999/
        -----------------------------------------------------------

        Review request for Flume, Arvind Prabhakar and Prasad Mujumdar.

        Summary
        -------

        Updated to backoff the poll interval by 1 second with a max of 5 seconds if successive BACKOFFs are returned from either a PollableSource or PollableSink.

        This addresses bug FLUME-867.
        https://issues.apache.org/jira/browse/FLUME-867

        Diffs


        flume-ng-core/src/main/java/org/apache/flume/sink/PollableSinkRunner.java 7bfb589
        flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceRunner.java 8b01da0

        Diff: https://reviews.apache.org/r/2999/diff

        Testing
        -------

        Unit tests pass, manual testing shows successful data transfer (with added delay in some cases, as expected).

        Thanks,

        Eric

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2999/ ----------------------------------------------------------- Review request for Flume, Arvind Prabhakar and Prasad Mujumdar. Summary ------- Updated to backoff the poll interval by 1 second with a max of 5 seconds if successive BACKOFFs are returned from either a PollableSource or PollableSink. This addresses bug FLUME-867 . https://issues.apache.org/jira/browse/FLUME-867 Diffs flume-ng-core/src/main/java/org/apache/flume/sink/PollableSinkRunner.java 7bfb589 flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceRunner.java 8b01da0 Diff: https://reviews.apache.org/r/2999/diff Testing ------- Unit tests pass, manual testing shows successful data transfer (with added delay in some cases, as expected). Thanks, Eric
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2999/#review3620
        -----------------------------------------------------------

        Ship it!

        I am wondering if the interval increment should be shorter, 500 or less ...
        Otherwise looks good to me.

        • Prasad

        On 2011-12-05 16:08:51, Eric Sammer wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2999/

        -----------------------------------------------------------

        (Updated 2011-12-05 16:08:51)

        Review request for Flume, Arvind Prabhakar and Prasad Mujumdar.

        Summary

        -------

        Updated to backoff the poll interval by 1 second with a max of 5 seconds if successive BACKOFFs are returned from either a PollableSource or PollableSink.

        This addresses bug FLUME-867.

        https://issues.apache.org/jira/browse/FLUME-867

        Diffs

        -----

        flume-ng-core/src/main/java/org/apache/flume/sink/PollableSinkRunner.java 7bfb589

        flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceRunner.java 8b01da0

        Diff: https://reviews.apache.org/r/2999/diff

        Testing

        -------

        Unit tests pass, manual testing shows successful data transfer (with added delay in some cases, as expected).

        Thanks,

        Eric

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2999/#review3620 ----------------------------------------------------------- Ship it! I am wondering if the interval increment should be shorter, 500 or less ... Otherwise looks good to me. Prasad On 2011-12-05 16:08:51, Eric Sammer wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2999/ ----------------------------------------------------------- (Updated 2011-12-05 16:08:51) Review request for Flume, Arvind Prabhakar and Prasad Mujumdar. Summary ------- Updated to backoff the poll interval by 1 second with a max of 5 seconds if successive BACKOFFs are returned from either a PollableSource or PollableSink. This addresses bug FLUME-867 . https://issues.apache.org/jira/browse/FLUME-867 Diffs ----- flume-ng-core/src/main/java/org/apache/flume/sink/PollableSinkRunner.java 7bfb589 flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceRunner.java 8b01da0 Diff: https://reviews.apache.org/r/2999/diff Testing ------- Unit tests pass, manual testing shows successful data transfer (with added delay in some cases, as expected). Thanks, Eric
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-12-05 17:28:25, Prasad Mujumdar wrote:

        > I am wondering if the interval increment should be shorter, 500 or less ...

        > Otherwise looks good to me.

        >

        At some point, I think it's useful for these to be configurable. The reason I went for 1000ms is because this impacts things like AvroSink's client retry interval and 500ms seemed too fast. I'm going to leave it for now but let's revisit.

        • Eric

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2999/#review3620
        -----------------------------------------------------------

        On 2011-12-05 16:08:51, Eric Sammer wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2999/

        -----------------------------------------------------------

        (Updated 2011-12-05 16:08:51)

        Review request for Flume, Arvind Prabhakar and Prasad Mujumdar.

        Summary

        -------

        Updated to backoff the poll interval by 1 second with a max of 5 seconds if successive BACKOFFs are returned from either a PollableSource or PollableSink.

        This addresses bug FLUME-867.

        https://issues.apache.org/jira/browse/FLUME-867

        Diffs

        -----

        flume-ng-core/src/main/java/org/apache/flume/sink/PollableSinkRunner.java 7bfb589

        flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceRunner.java 8b01da0

        Diff: https://reviews.apache.org/r/2999/diff

        Testing

        -------

        Unit tests pass, manual testing shows successful data transfer (with added delay in some cases, as expected).

        Thanks,

        Eric

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-12-05 17:28:25, Prasad Mujumdar wrote: > I am wondering if the interval increment should be shorter, 500 or less ... > Otherwise looks good to me. > At some point, I think it's useful for these to be configurable. The reason I went for 1000ms is because this impacts things like AvroSink's client retry interval and 500ms seemed too fast. I'm going to leave it for now but let's revisit. Eric ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2999/#review3620 ----------------------------------------------------------- On 2011-12-05 16:08:51, Eric Sammer wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2999/ ----------------------------------------------------------- (Updated 2011-12-05 16:08:51) Review request for Flume, Arvind Prabhakar and Prasad Mujumdar. Summary ------- Updated to backoff the poll interval by 1 second with a max of 5 seconds if successive BACKOFFs are returned from either a PollableSource or PollableSink. This addresses bug FLUME-867 . https://issues.apache.org/jira/browse/FLUME-867 Diffs ----- flume-ng-core/src/main/java/org/apache/flume/sink/PollableSinkRunner.java 7bfb589 flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceRunner.java 8b01da0 Diff: https://reviews.apache.org/r/2999/diff Testing ------- Unit tests pass, manual testing shows successful data transfer (with added delay in some cases, as expected). Thanks, Eric
        Hide
        Hudson added a comment -

        Integrated in flume-728 #69 (See https://builds.apache.org/job/flume-728/69/)
        FLUME-867: Pollable source and sink runners should reduce poll interval after several BACKOFFs

        esammer : http://svn.apache.org/viewvc/?view=rev&rev=1210573
        Files :

        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/PollableSinkRunner.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceRunner.java
        Show
        Hudson added a comment - Integrated in flume-728 #69 (See https://builds.apache.org/job/flume-728/69/ ) FLUME-867 : Pollable source and sink runners should reduce poll interval after several BACKOFFs esammer : http://svn.apache.org/viewvc/?view=rev&rev=1210573 Files : /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/PollableSinkRunner.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceRunner.java
        Hide
        E. Sammer added a comment -

        Committed to branch flume-728.

        Show
        E. Sammer added a comment - Committed to branch flume-728.
        E. Sammer made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        E. Sammer made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        50m 48s 1 E. Sammer 04/Dec/11 03:49
        Patch Available Patch Available Resolved Resolved
        1d 19h 11m 1 E. Sammer 05/Dec/11 23:01
        Resolved Resolved Closed Closed
        5s 1 E. Sammer 05/Dec/11 23:01

          People

          • Assignee:
            E. Sammer
            Reporter:
            E. Sammer
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development