[HDFS-8254] Avoid assigning a leading streamer in StripedDataStreamer to tolerate datanode failure

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: HDFS-7285
• Hadoop Flags: Reviewed
• Assignee: Tsz-wo Sze
• Reporter: Tsz-wo Sze
• Created: 25/Apr/15 21:06
• Updated: 30/Sep/15 19:38
• Resolved: 19/Jun/15 17:26

Description

StripedDataStreamer javadoc is shown below.

 * The StripedDataStreamer class is used by {@link DFSStripedOutputStream}.
 * There are two kinds of StripedDataStreamer, leading streamer and ordinary
 * stream. Leading streamer requests a block group from NameNode, unwraps
 * it to located blocks and transfers each located block to its corresponding
 * ordinary streamer via a blocking queue.

Leading streamer is the streamer with index 0. When the datanode of the leading streamer fails, the other steamers cannot continue since no one will request a block group from NameNode anymore.

Issue Links

• is related to: HDFS-7672 Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks
• supercedes: HDFS-8366 Erasure Coding: Make the timeout parameter of polling blocking queue configurable in DFSStripedOutputStream