Here is my first design.

1. Update SourceRunner.forSource to take a Configuration object.
2. Update AbstractConfigurationProvider to provide that object.
3. Update PollableSourceRunner to take the configuration object through its constructer.
4. Update SourceRunner to provide PollableSourceRunner with the config object.
5. Added Unit Test in TestPollableSourceRunner.

Let me know what you think

First patch

I didn't look much into the patch, but I did used Guava retrying lib for a use case and it had some wonderful constructs like simple time based back-off, exponential back-off etc. Would be great if we can use something similar here. Just a suggestion

Wow Guava looks really cool but it might be more then I intended for this Jira. Plus I would need Commiter buy in for Guava.

Is it possible I can get this fix in then we open up another jira to discuss adding Guava retry in and what that would look like in the flume properties?

Ted Malaska

It's just a suggestion I know Guava troubles so won't suggest it for now, but just that you know something exists

Ted Malaska, A couple of thoughts on the patch,

1) Wouldn't it be better to store/set this parameter as part of AbstractPollableSource. (We need to make sure that all the sources that implement PollableSource extend from AbstractPollableSource)
2) Test case is missing from the patch

Update Johny and I talked and this is where we are:

1. The abstractPollableSource is right in spirt but does lot possible because of how object extends other objects
2. I will add a unit test. Johny gave me some advice on that.
3. Johny and I will connect tomorrow to review option 1 again

Thank Johny

This is not 100% ready because I haven't added a new unit test. However this is the change, based on John's review.

I will add the unit test tomorrow and resubmit

Will be good to create a review request

Sorry about that. Review Board link added.

I have to go to a swimming met. Tomorrow I will add the unit test but also I will do a full review of the patch.

But I would love a review to see if you agree with the over all approach.

Thanks

OK unit tests added and fully ready for review

Hey Johny made you codes changes. Let me know if there is any more.

+1, Will run tests and commit

Commit b5e102bee4d4b2783f3f65aab403a53c1ae8e401 in flume's branch refs/heads/trunk from Johny Rufus
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=b5e102b ]

FLUME-2729. Allow pollableSource backoff times to be configurable

(Ted Malaska via Johny Rufus)

Commit e7a8b80a0e5115e68fd545824e6b71082c89eb06 in flume's branch refs/heads/flume-1.7 from Johny Rufus
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=e7a8b80 ]

FLUME-2729. Allow pollableSource backoff times to be configurable

(Ted Malaska via Johny Rufus)

Committed. Thanks Ted!

Wow thank u so much.

UNSTABLE: Integrated in Flume-trunk-hbase-1 #107 (See https://builds.apache.org/job/Flume-trunk-hbase-1/107/)
FLUME-2729. Allow pollableSource backoff times to be configurable (johnyrufus: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=b5e102bee4d4b2783f3f65aab403a53c1ae8e401)

• flume-ng-core/src/main/java/org/apache/flume/PollableSource.java
• flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
• flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java
• flume-ng-core/src/test/java/org/apache/flume/source/TestStressSource.java
• flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceRunner.java
• flume-ng-core/src/main/java/org/apache/flume/source/AbstractPollableSource.java
• flume-ng-core/src/test/java/org/apache/flume/source/TestAbstractPollableSource.java
• flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java
• flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceConstants.java
• flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestIntegration.java
• flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java
• flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java

Commit 5f6d6c3b1d0100c57c2656fcd014dc2ee8e0a607 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=5f6d6c3 ]

FLUME-2737: Documentation for Pollable Source config parameters introduced in FLUME-2729

(Ted Malaska via Jarek Jarcec Cecho)

Commit fbf76610f84fe5e7a6b1790a6fd0a72e523cbf34 in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=fbf7661 ]

FLUME-2737: Documentation for Pollable Source config parameters introduced in FLUME-2729

(Ted Malaska via Jarek Jarcec Cecho)

UNSTABLE: Integrated in Flume-trunk-hbase-1 #111 (See https://builds.apache.org/job/Flume-trunk-hbase-1/111/)
FLUME-2737: Documentation for Pollable Source config parameters introduced in FLUME-2729 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=5f6d6c3b1d0100c57c2656fcd014dc2ee8e0a607)

• flume-ng-doc/sphinx/FlumeUserGuide.rst