[FLUME-857] CollectorSink can get stuck into perpetual close() if the underlying sinks throws IOExceptions for both append and close.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.3, 0.9.4
• Fix Version/s: 0.9.5
• Assignee: Prasad Suresh Mujumdar
• Reporter: Prasad Suresh Mujumdar
• Created: 23/Nov/11 01:18
• Updated: 28/Nov/11 22:24
• Resolved: 28/Nov/11 22:24

Description

The CollectionSink create a chain of sinks on top of the underlying sink to perform roll and retries upon failures. The stubborn append reopens the downstream sink and retries the append once. The insistent append keeps retrying the append.
Now if any errors (like network or HDFS datanodes shutdown) cause the downstream sink to throw IOException on append, the stubborn append will try to close that sink and reopen. If the close also throws an IOException, then sink chain doesn't get properly closed. This results into the collectorsink skipping the open and perpetually failing on append/close.