[FLUME-1916] HDFS sink should poll for # of active replicas. If less than required, roll the file.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.1
• Fix Version/s: 1.4.0
• Component/s: Sinks+Sources
• Assignee: Mike Percy
• Reporter: Mike Percy
• Created: 13/Feb/13 00:46
• Updated: 02/Mar/13 16:18
• Resolved: 01/Mar/13 08:59

Description

Add functionality to the HDFS sink which constantly polls the number of replicas for files being written. If the number of replicas drops below 3 (or specified number), it should immediately close the HDFS output file and start a new file that should be able to create a pipeline with the correct number of replicas.

This is the same behavior taken by HBase for its write-ahead log (WAL) and should help us avoid hitting the more complex corner cases around failed close() calls.

Issue Links

• links to: Review Board