Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5434

Write resiliency for replica count 1

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      If a file has a replica count of one, the HDFS client is exposed to write failures if the data node fails during a write. With a pipeline of size of one, no recovery is possible if the sole data node dies.

      A simple fix is to force a minimum pipeline size of 2, while leaving the replication count as 1. The implementation for this is fairly non-invasive.

      Although the replica count is one, the block will be written to two data nodes instead of one. If one of the data nodes fails during the write, normal pipeline recovery will ensure that the write succeeds to the surviving data node.

      The existing code in the name node will prune the extra replica when it receives the block received reports for the finalized block from both data nodes. This results in the intended replica count of one for the block.

      This behavior should be controlled by a configuration option such as dfs.namenode.minPipelineSize.

      This behavior can be implemented in FSNameSystem.getAdditionalBlock() by ensuring that the pipeline size passed to BlockPlacementPolicy.chooseTarget() in the replication parameter is:

      max(replication, ${dfs.namenode.minPipelineSize})
      

        Attachments

        1. BlockPlacementPolicyMinPipelineSize.java
          2 kB
          Taylor, Buddy
        2. BlockPlacementPolicyMinPipelineSizeWithNodeGroup.java
          2 kB
          Taylor, Buddy
        3. HDFS_5434.patch
          2 kB
          Taylor, Buddy
        4. HDFS-5434-branch-2.patch
          2 kB
          Taylor, Buddy

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              buddytaylor0 Taylor, Buddy
            • Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: