[FLUME-503] Use HDFS sync API instead of rolling for durability

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.9.4
• Fix Version/s: 0.9.5
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: Disabled imported user
• Created: 02/Feb/11 19:31
• Updated: 05/Nov/14 09:34
• Resolved: 05/Nov/14 09:34

Description

Some versions of Hadoop (CDH3>b2 or 0.20-append branch) support a sync() API that guarantees data has been flushed to all of the nodes in the write pipeline. This should be equally as durable as closing an HDFS file.

Flume should allow the use of sync() to make data durable on a regular basis without having to create lots of tiny files on HDFS.

Related is the ability to use the getNumCurrentReplicas() API to detect when the number of replicas falls below the desired replication factor, and roll at that point (to pick up a new DN)

Issue Links

• duplicates: FLUME-454 Use HDFS HFlush/Sync
• is duplicated by: FLUME-454 Use HDFS HFlush/Sync