[HDDS-601] On restart, SCM throws 'No such datanode' exception

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.2.1
• Fix Version/s: 0.3.0, 0.4.0
• Component/s: SCM
• Target Version/s: 0.3.0
• Assignee: Hanisha Koneru
• Reporter: Soumitra Sulav
• Created: 09/Oct/18 10:34
• Updated: 11/Oct/18 21:11
• Resolved: 11/Oct/18 20:49

Description

Encountered below exception after I changed a configuration in ozone-site and restarted SCM and Datanode :

Ozone Cluster : 1 SCM, 1 OM, 3 DNs

2018-10-04 09:35:59,716 INFO org.apache.hadoop.hdds.server.BaseHttpServer: HTTP server of SCM is listening at http://0.0.0.0:9876
2018-10-04 09:36:03,618 WARN org.apache.hadoop.hdds.scm.node.SCMNodeManager: SCM receive heartbeat from unregistered datanode 127a8e17-b2df-4663-924c-1a6909adb293{ip: 172.22.119.19, host: hcatest-2.openstacklocal}
2018-10-04 09:36:09,063 WARN org.apache.hadoop.hdds.scm.node.SCMNodeManager: SCM receive heartbeat from unregistered datanode 82555af0-a1f9-447a-ad40-c524ba6e1317{ip: 172.22.119.190, host: hcatest-3.openstacklocal}
2018-10-04 09:36:09,083 ERROR org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Error on processing container report from datanode 82555af0-a1f9-447a-ad40-c524ba6e1317{ip: 172.22.119.190, host: hcatest-3.openstacklocal}
org.apache.hadoop.hdds.scm.exceptions.SCMException: No such datanode
 at org.apache.hadoop.hdds.scm.node.states.Node2ContainerMap.setContainersForDatanode(Node2ContainerMap.java:82)
 at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:97)
 at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:45)
 at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

Issue Links

• duplicates: HDDS-610 On restart of SCM it fails to register DataNodes