[FLUME-2357] HDFS sink should retry closing files that previously had close errors

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.5.0
• Component/s: Sinks+Sources
• Assignee: Hari Shreedharan
• Reporter: Patrick Dvorak
• Created: 09/Apr/14 21:50
• Updated: 29/Apr/14 04:05
• Resolved: 29/Apr/14 00:21

Description

When the AbstractHDFSWriter fails to close a file (due to exceeding the callTimeout or other hdfs issues), it will leave the the file open and never try again. The only way to close the open files is to restart the flume agent. There should be a configurable option to allow the sink to retry to close files that had previously failed to close.

Issue Links

• links to: Review