Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2729

Allow pollableSource backoff times to be configurable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.7.0
    • Component/s: None
    • Labels:
      None

      Description

      There are use cases that require more control of the backoff times setting. This jira will allow those times to be set through config of the source

      1. FLUME_2729.patch
        7 kB
        Theodore michael Malaska
      2. FLUME-2729.patch.2
        21 kB
        Theodore michael Malaska
      3. FLUME-2729.patch.4
        22 kB
        Theodore michael Malaska
      4. FLUME-2729.patch.7
        23 kB
        Theodore michael Malaska

        Issue Links

          Activity

          Hide
          ted.m Theodore michael Malaska added a comment -

          Here is my first design.

          1. Update SourceRunner.forSource to take a Configuration object.
          2. Update AbstractConfigurationProvider to provide that object.
          3. Update PollableSourceRunner to take the configuration object through its constructer.
          4. Update SourceRunner to provide PollableSourceRunner with the config object.
          5. Added Unit Test in TestPollableSourceRunner.

          Let me know what you think

          Show
          ted.m Theodore michael Malaska added a comment - Here is my first design. 1. Update SourceRunner.forSource to take a Configuration object. 2. Update AbstractConfigurationProvider to provide that object. 3. Update PollableSourceRunner to take the configuration object through its constructer. 4. Update SourceRunner to provide PollableSourceRunner with the config object. 5. Added Unit Test in TestPollableSourceRunner. Let me know what you think
          Hide
          ted.m Theodore michael Malaska added a comment -

          First patch

          Show
          ted.m Theodore michael Malaska added a comment - First patch
          Hide
          paliwalashish Ashish Paliwal added a comment -

          I didn't look much into the patch, but I did used Guava retrying lib for a use case and it had some wonderful constructs like simple time based back-off, exponential back-off etc. Would be great if we can use something similar here. Just a suggestion

          Show
          paliwalashish Ashish Paliwal added a comment - I didn't look much into the patch, but I did used Guava retrying lib for a use case and it had some wonderful constructs like simple time based back-off, exponential back-off etc. Would be great if we can use something similar here. Just a suggestion
          Hide
          ted.m Theodore michael Malaska added a comment -

          Wow Guava looks really cool but it might be more then I intended for this Jira. Plus I would need Commiter buy in for Guava.

          Is it possible I can get this fix in then we open up another jira to discuss adding Guava retry in and what that would look like in the flume properties?

          Ted Malaska

          Show
          ted.m Theodore michael Malaska added a comment - Wow Guava looks really cool but it might be more then I intended for this Jira. Plus I would need Commiter buy in for Guava. Is it possible I can get this fix in then we open up another jira to discuss adding Guava retry in and what that would look like in the flume properties? Ted Malaska
          Hide
          paliwalashish Ashish Paliwal added a comment -

          It's just a suggestion I know Guava troubles so won't suggest it for now, but just that you know something exists

          Show
          paliwalashish Ashish Paliwal added a comment - It's just a suggestion I know Guava troubles so won't suggest it for now, but just that you know something exists
          Hide
          jrufus Johny Rufus added a comment -

          Ted Malaska, A couple of thoughts on the patch,

          1) Wouldn't it be better to store/set this parameter as part of AbstractPollableSource. (We need to make sure that all the sources that implement PollableSource extend from AbstractPollableSource)
          2) Test case is missing from the patch

          Show
          jrufus Johny Rufus added a comment - Ted Malaska , A couple of thoughts on the patch, 1) Wouldn't it be better to store/set this parameter as part of AbstractPollableSource. (We need to make sure that all the sources that implement PollableSource extend from AbstractPollableSource) 2) Test case is missing from the patch
          Hide
          ted.m Theodore michael Malaska added a comment -

          Update Johny and I talked and this is where we are:

          1. The abstractPollableSource is right in spirt but does lot possible because of how object extends other objects
          2. I will add a unit test. Johny gave me some advice on that.
          3. Johny and I will connect tomorrow to review option 1 again

          Thank Johny

          Show
          ted.m Theodore michael Malaska added a comment - Update Johny and I talked and this is where we are: 1. The abstractPollableSource is right in spirt but does lot possible because of how object extends other objects 2. I will add a unit test. Johny gave me some advice on that. 3. Johny and I will connect tomorrow to review option 1 again Thank Johny
          Hide
          ted.m Theodore michael Malaska added a comment -

          This is not 100% ready because I haven't added a new unit test. However this is the change, based on John's review.

          I will add the unit test tomorrow and resubmit

          Show
          ted.m Theodore michael Malaska added a comment - This is not 100% ready because I haven't added a new unit test. However this is the change, based on John's review. I will add the unit test tomorrow and resubmit
          Hide
          paliwalashish Ashish Paliwal added a comment -

          Will be good to create a review request

          Show
          paliwalashish Ashish Paliwal added a comment - Will be good to create a review request
          Hide
          ted.m Theodore michael Malaska added a comment -

          Sorry about that. Review Board link added.

          I have to go to a swimming met. Tomorrow I will add the unit test but also I will do a full review of the patch.

          But I would love a review to see if you agree with the over all approach.

          Thanks

          Show
          ted.m Theodore michael Malaska added a comment - Sorry about that. Review Board link added. I have to go to a swimming met. Tomorrow I will add the unit test but also I will do a full review of the patch. But I would love a review to see if you agree with the over all approach. Thanks
          Hide
          ted.m Theodore michael Malaska added a comment -

          OK unit tests added and fully ready for review

          Show
          ted.m Theodore michael Malaska added a comment - OK unit tests added and fully ready for review
          Hide
          ted.m Theodore michael Malaska added a comment -

          Hey Johny made you codes changes. Let me know if there is any more.

          Show
          ted.m Theodore michael Malaska added a comment - Hey Johny made you codes changes. Let me know if there is any more.
          Hide
          jrufus Johny Rufus added a comment -

          +1, Will run tests and commit

          Show
          jrufus Johny Rufus added a comment - +1, Will run tests and commit
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit b5e102bee4d4b2783f3f65aab403a53c1ae8e401 in flume's branch refs/heads/trunk from Johny Rufus
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=b5e102b ]

          FLUME-2729. Allow pollableSource backoff times to be configurable

          (Ted Malaska via Johny Rufus)

          Show
          jira-bot ASF subversion and git services added a comment - Commit b5e102bee4d4b2783f3f65aab403a53c1ae8e401 in flume's branch refs/heads/trunk from Johny Rufus [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=b5e102b ] FLUME-2729 . Allow pollableSource backoff times to be configurable (Ted Malaska via Johny Rufus)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit e7a8b80a0e5115e68fd545824e6b71082c89eb06 in flume's branch refs/heads/flume-1.7 from Johny Rufus
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=e7a8b80 ]

          FLUME-2729. Allow pollableSource backoff times to be configurable

          (Ted Malaska via Johny Rufus)

          Show
          jira-bot ASF subversion and git services added a comment - Commit e7a8b80a0e5115e68fd545824e6b71082c89eb06 in flume's branch refs/heads/flume-1.7 from Johny Rufus [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=e7a8b80 ] FLUME-2729 . Allow pollableSource backoff times to be configurable (Ted Malaska via Johny Rufus)
          Hide
          jrufus Johny Rufus added a comment -

          Committed. Thanks Ted!

          Show
          jrufus Johny Rufus added a comment - Committed. Thanks Ted!
          Hide
          ted.m Theodore michael Malaska added a comment -

          Wow thank u so much.

          Show
          ted.m Theodore michael Malaska added a comment - Wow thank u so much.
          Hide
          hudson Hudson added a comment -

          UNSTABLE: Integrated in Flume-trunk-hbase-1 #107 (See https://builds.apache.org/job/Flume-trunk-hbase-1/107/)
          FLUME-2729. Allow pollableSource backoff times to be configurable (johnyrufus: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=b5e102bee4d4b2783f3f65aab403a53c1ae8e401)

          • flume-ng-core/src/main/java/org/apache/flume/PollableSource.java
          • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
          • flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java
          • flume-ng-core/src/test/java/org/apache/flume/source/TestStressSource.java
          • flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceRunner.java
          • flume-ng-core/src/main/java/org/apache/flume/source/AbstractPollableSource.java
          • flume-ng-core/src/test/java/org/apache/flume/source/TestAbstractPollableSource.java
          • flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java
          • flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceConstants.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestIntegration.java
          • flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java
          • flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java
          Show
          hudson Hudson added a comment - UNSTABLE: Integrated in Flume-trunk-hbase-1 #107 (See https://builds.apache.org/job/Flume-trunk-hbase-1/107/ ) FLUME-2729 . Allow pollableSource backoff times to be configurable (johnyrufus: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=b5e102bee4d4b2783f3f65aab403a53c1ae8e401 ) flume-ng-core/src/main/java/org/apache/flume/PollableSource.java flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java flume-ng-core/src/test/java/org/apache/flume/source/TestStressSource.java flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceRunner.java flume-ng-core/src/main/java/org/apache/flume/source/AbstractPollableSource.java flume-ng-core/src/test/java/org/apache/flume/source/TestAbstractPollableSource.java flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceConstants.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestIntegration.java flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 5f6d6c3b1d0100c57c2656fcd014dc2ee8e0a607 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=5f6d6c3 ]

          FLUME-2737: Documentation for Pollable Source config parameters introduced in FLUME-2729

          (Ted Malaska via Jarek Jarcec Cecho)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 5f6d6c3b1d0100c57c2656fcd014dc2ee8e0a607 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=5f6d6c3 ] FLUME-2737 : Documentation for Pollable Source config parameters introduced in FLUME-2729 (Ted Malaska via Jarek Jarcec Cecho)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit fbf76610f84fe5e7a6b1790a6fd0a72e523cbf34 in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=fbf7661 ]

          FLUME-2737: Documentation for Pollable Source config parameters introduced in FLUME-2729

          (Ted Malaska via Jarek Jarcec Cecho)

          Show
          jira-bot ASF subversion and git services added a comment - Commit fbf76610f84fe5e7a6b1790a6fd0a72e523cbf34 in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=fbf7661 ] FLUME-2737 : Documentation for Pollable Source config parameters introduced in FLUME-2729 (Ted Malaska via Jarek Jarcec Cecho)
          Hide
          hudson Hudson added a comment -

          UNSTABLE: Integrated in Flume-trunk-hbase-1 #111 (See https://builds.apache.org/job/Flume-trunk-hbase-1/111/)
          FLUME-2737: Documentation for Pollable Source config parameters introduced in FLUME-2729 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=5f6d6c3b1d0100c57c2656fcd014dc2ee8e0a607)

          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          Show
          hudson Hudson added a comment - UNSTABLE: Integrated in Flume-trunk-hbase-1 #111 (See https://builds.apache.org/job/Flume-trunk-hbase-1/111/ ) FLUME-2737 : Documentation for Pollable Source config parameters introduced in FLUME-2729 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=5f6d6c3b1d0100c57c2656fcd014dc2ee8e0a607 ) flume-ng-doc/sphinx/FlumeUserGuide.rst

            People

            • Assignee:
              ted.m Theodore michael Malaska
              Reporter:
              ted.m Theodore michael Malaska
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development