[HDDS-1707] SCMContainerPlacementRackAware#chooseDatanodes throws not enough datanodes when all nodes(40) are up

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: SCM
• Target Version/s: 0.5.0
• Assignee: Sammi Chen
• Reporter: Mukul Kumar Singh
• Created: 19/Jun/19 10:19
• Updated: 26/Jul/19 03:10
• Resolved: 26/Jul/19 03:10

Description

SCMContainerPlacementRackAware#chooseDatanodes is failing with the following error repeatedly.

2019-06-17 22:15:52,455 WARN org.apache.hadoop.hdds.scm.container.ReplicationManager: Exception while replicating container 407.
org.apache.hadoop.hdds.scm.exceptions.SCMException: No enough datanodes to choose.
        at org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware.chooseDatanodes(SCMContainerPlacementRackAware.java:100)
        at org.apache.hadoop.hdds.scm.container.ReplicationManager.handleUnderReplicatedContainer(ReplicationManager.java:487)
        at org.apache.hadoop.hdds.scm.container.ReplicationManager.processContainer(ReplicationManager.java:293)
        at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649)
        at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
        at org.apache.hadoop.hdds.scm.container.ReplicationManager.run(ReplicationManager.java:205)
        at java.lang.Thread.run(Thread.java:745)