Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1606

Provide a stronger data guarantee in the write pipeline

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: datanode, hdfs-client, namenode
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Added two configuration properties, dfs.client.block.write.replace-datanode-on-failure.enable and dfs.client.block.write.replace-datanode-on-failure.policy. Added a new feature to replace datanode on failure in DataTransferProtocol. Added getAdditionalDatanode(..) in ClientProtocol.
      Show
      Added two configuration properties, dfs.client.block.write.replace-datanode-on-failure.enable and dfs.client.block.write.replace-datanode-on-failure.policy. Added a new feature to replace datanode on failure in DataTransferProtocol. Added getAdditionalDatanode(..) in ClientProtocol.

      Description

      In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes. As a result, the number of datanodes in the pipeline is decreased. Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.

      We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

      1. h1606_20110210.patch
        7 kB
        Tsz Wo Nicholas Sze
      2. h1606_20110211.patch
        20 kB
        Tsz Wo Nicholas Sze
      3. h1606_20110217.patch
        25 kB
        Tsz Wo Nicholas Sze
      4. h1606_20110228.patch
        34 kB
        Tsz Wo Nicholas Sze
      5. h1606_20110404.patch
        34 kB
        Tsz Wo Nicholas Sze
      6. h1606_20110405.patch
        35 kB
        Tsz Wo Nicholas Sze
      7. h1606_20110405b.patch
        35 kB
        Tsz Wo Nicholas Sze
      8. h1606_20110406.patch
        41 kB
        Tsz Wo Nicholas Sze
      9. h1606_20110406b.patch
        43 kB
        Tsz Wo Nicholas Sze
      10. h1606_20110407.patch
        48 kB
        Tsz Wo Nicholas Sze
      11. h1606_20110407b.patch
        48 kB
        Tsz Wo Nicholas Sze
      12. h1606_20110407c.patch
        50 kB
        Tsz Wo Nicholas Sze
      13. h1606_20110408.patch
        49 kB
        Tsz Wo Nicholas Sze
      14. h1606_20110408b.patch
        52 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development