[HDFS-11799] Introduce a config to allow setting up write pipeline with fewer nodes than replication factor

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.3, 3.1.0
• Hadoop Flags: Reviewed

Release Note

Added new configuration "dfs.client.block.write.replace-datanode-on-failure.min-replication".
     
    The minimum number of replications that are needed to not to fail
      the write pipeline if new datanodes can not be found to replace
      failed datanodes (could be due to network failure) in the write pipeline.
      If the number of the remaining datanodes in the write pipeline is greater
      than or equal to this property value, continue writing to the remaining nodes.
      Otherwise throw exception.

      If this is set to 0, an exception will be thrown, when a replacement
      can not be found.
    

• Assignee: Brahma Reddy Battula
• Reporter: Yongjun Zhang
• Created: 10/May/17 17:08
• Updated: 19/Sep/17 18:30
• Resolved: 19/Sep/17 18:30

Description

During pipeline recovery, if not enough DNs can be found, if
dfs.client.block.write.replace-datanode-on-failure.best-effort
is enabled, we let the pipeline to continue, even if there is a single DN.

Similarly, when we create the write pipeline initially, if for some reason we can't find enough DNs, we can have a similar config to enable writing with a single DN.

More study will be done.