Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11799

Introduce a config to allow setting up write pipeline with fewer nodes than replication factor

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.3, 3.1.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Added new configuration "dfs.client.block.write.replace-datanode-on-failure.min-replication".
           
          The minimum number of replications that are needed to not to fail
            the write pipeline if new datanodes can not be found to replace
            failed datanodes (could be due to network failure) in the write pipeline.
            If the number of the remaining datanodes in the write pipeline is greater
            than or equal to this property value, continue writing to the remaining nodes.
            Otherwise throw exception.

            If this is set to 0, an exception will be thrown, when a replacement
            can not be found.
          
      Show
      Added new configuration "dfs.client.block.write.replace-datanode-on-failure.min-replication".           The minimum number of replications that are needed to not to fail       the write pipeline if new datanodes can not be found to replace       failed datanodes (could be due to network failure) in the write pipeline.       If the number of the remaining datanodes in the write pipeline is greater       than or equal to this property value, continue writing to the remaining nodes.       Otherwise throw exception.       If this is set to 0, an exception will be thrown, when a replacement       can not be found.     

      Description

      During pipeline recovery, if not enough DNs can be found, if
      dfs.client.block.write.replace-datanode-on-failure.best-effort
      is enabled, we let the pipeline to continue, even if there is a single DN.

      Similarly, when we create the write pipeline initially, if for some reason we can't find enough DNs, we can have a similar config to enable writing with a single DN.

      More study will be done.

        Attachments

        1. HDFS-11799.patch
          21 kB
          Brahma Reddy Battula
        2. HDFS-11799-002.patch
          22 kB
          Brahma Reddy Battula
        3. HDFS-11799-003.patch
          22 kB
          Brahma Reddy Battula
        4. HDFS-11799-004.patch
          17 kB
          Brahma Reddy Battula
        5. HDFS-11799-005.patch
          17 kB
          Brahma Reddy Battula
        6. HDFS-11799-006.patch
          17 kB
          Brahma Reddy Battula
        7. HDFS-11799-007.patch
          17 kB
          Brahma Reddy Battula
        8. HDFS-11799-008.patch
          17 kB
          Brahma Reddy Battula
        9. HDFS-11799-009.patch
          17 kB
          Brahma Reddy Battula

          Activity

            People

            • Assignee:
              brahmareddy Brahma Reddy Battula
              Reporter:
              yzhangal Yongjun Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: