Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1606

Provide a stronger data guarantee in the write pipeline

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: datanode, hdfs-client, namenode
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Added two configuration properties, dfs.client.block.write.replace-datanode-on-failure.enable and dfs.client.block.write.replace-datanode-on-failure.policy. Added a new feature to replace datanode on failure in DataTransferProtocol. Added getAdditionalDatanode(..) in ClientProtocol.
      Show
      Added two configuration properties, dfs.client.block.write.replace-datanode-on-failure.enable and dfs.client.block.write.replace-datanode-on-failure.policy. Added a new feature to replace datanode on failure in DataTransferProtocol. Added getAdditionalDatanode(..) in ClientProtocol.

      Description

      In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes. As a result, the number of datanodes in the pipeline is decreased. Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.

      We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

        Attachments

        1. h1606_20110210.patch
          7 kB
          Tsz Wo Nicholas Sze
        2. h1606_20110211.patch
          20 kB
          Tsz Wo Nicholas Sze
        3. h1606_20110217.patch
          25 kB
          Tsz Wo Nicholas Sze
        4. h1606_20110228.patch
          34 kB
          Tsz Wo Nicholas Sze
        5. h1606_20110404.patch
          34 kB
          Tsz Wo Nicholas Sze
        6. h1606_20110405.patch
          35 kB
          Tsz Wo Nicholas Sze
        7. h1606_20110405b.patch
          35 kB
          Tsz Wo Nicholas Sze
        8. h1606_20110406.patch
          41 kB
          Tsz Wo Nicholas Sze
        9. h1606_20110406b.patch
          43 kB
          Tsz Wo Nicholas Sze
        10. h1606_20110407.patch
          48 kB
          Tsz Wo Nicholas Sze
        11. h1606_20110407b.patch
          48 kB
          Tsz Wo Nicholas Sze
        12. h1606_20110407c.patch
          50 kB
          Tsz Wo Nicholas Sze
        13. h1606_20110408.patch
          49 kB
          Tsz Wo Nicholas Sze
        14. h1606_20110408b.patch
          52 kB
          Tsz Wo Nicholas Sze

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                szetszwo Tsz Wo Nicholas Sze
                Reporter:
                szetszwo Tsz Wo Nicholas Sze
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: