Patch that handles that fixes this.

LGTM, my one suggestion is to make maxBackoffMillis a user-configurable parameter.

Hari Shreedharan
The JIRA title seems very broad. it would be nice to add some more specifics in the description.

Would be nice to understand .... which exceptions we are talking about (channel/ file reading / other?) , what is the undesired behavior being exhibited etc.

Making the maxBackoff parameter configurable

Roshan Naik - As mentioned in the title, I am talking about channel exceptions.

Test based on the source counter that is deterministic.

what is the undesired behavior being exhibited ?

The source dies if the channel is full and cannot recover. You need to restart the agent to fix this issue

when the channel getting full triggers an exception... will the SpoolDir source ...

• retry with the same data later the channel frees up (no data loss) ?
• or would it essentially keep dropping events (data loss)?

Minor test fix

Roshan Naik - If you look at the patch, it is quite self-explanatory. The source will not commit the read with the event reader, so no data is lost. The same events are retried.

Hari, looks pretty good but can you take out the first sleep? If we need an arbitrary sleep() for anything other than burning too much CPU while in a loop, then it's a flaky test.

The test is also a little complicated, I wonder if instead we could do something like:

source.start();

// Make sure the Channel fills and the Source fails a put-transaction before continuing.
while (!source.hitChannelException()) {
  Thread.sleep(50);
}

// Now, make sure the source still works. Read all of the events.
while (numEventsTaken < numEventsPut) {
  // Clear the channel and increment our counters
  Thread.sleep(50);
}

Updated test based on Mike's suggestion.

Moving the while loop outside the main loop in the test.

+1 LGTM, thanks for updating the test. I plan on committing this later today.

Pushed to trunk and flume-1.5 branches. Thanks for the patch Hari!

FAILURE: Integrated in flume-trunk #523 (See https://builds.apache.org/job/flume-trunk/523/)
FLUME-2255. Correctly handle ChannelExceptions in SpoolingDirectorySource (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=c23448fc959844eece5a8ab2dbf091c2c4973a26)

• flume-ng-doc/sphinx/FlumeUserGuide.rst
• flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java
• flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java
• flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java