Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14465

When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.2
    • Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 2.9.3, 3.1.3
    • Component/s: None
    • Labels:
      None

      Description

      Scenes:

      There is a small HDFS cluster with 5 DataNodes; one of them is maintained, added to the maintenance list, and set dfs.namenode.maintenance.replication.min to 1.
      When refresh Nodes, the NameNode starts checking whether the blocks on the node require a new replication.
      The replications of the MapReduce task job file is 10 by default, isNeededReplicationForMaintenance will determine to false, and isSufficientlyReplicated will determine to false, so the block of the job file needs to increase the replication.
      When adding a replication, since the cluster has only 5 DataNodes, all the nodes have the replications of the block, chooseTargetInOrder will throw a NotEnoughReplicasException, so that the replication cannot be increase, and the Entering Maintenance cannot be ended.

      This issue will cause the independent small cluster to be unable to use the maintenance mode.

       

      chooseTarget exception log

      2019-05-03 23:42:31,008 [31545331] - WARN  [ReplicationMonitor:BlockPlacementPolicyDefault@431] - Failed to place enough replicas, still in need of 1 to reach 5 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=false) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology

       

        Attachments

        1. HDFS-14465.branch-2.9.01.patch
          3 kB
          Yicong Cai
        2. HDFS-14465.02.patch
          3 kB
          Yicong Cai
        3. HDFS-14465.01.patch
          3 kB
          Yicong Cai

          Issue Links

            Activity

              People

              • Assignee:
                caiyicong Yicong Cai
                Reporter:
                caiyicong Yicong Cai
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: