[HDFS-11041] Unable to unregister FsDatasetState MBean if DataNode is shutdown twice

• Type: Bug
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Fix Version/s: 3.4.0, 2.10.2, 3.2.4, 3.3.3
• Component/s: datanode
• Target Version/s: 3.3.3
• Hadoop Flags: Reviewed
• Assignee: Wei-Chiu Chuang
• Reporter: Wei-Chiu Chuang
• Created: 21/Oct/16 11:26
• Updated: 12/Jul/22 10:39
• Resolved: 18/Apr/22 14:15

Description

I saw error message like the following in some tests

2016-10-21 04:09:03,900 [main] WARN  util.MBeans (MBeans.java:unregister(114)) - Error unregistering Hadoop:service=DataNode,name=FSDatasetState-33cd714c-0b1a-471f-8efe-f431d7d874bc
javax.management.InstanceNotFoundException: Hadoop:service=DataNode,name=FSDatasetState-33cd714c-0b1a-471f-8efe-f431d7d874bc
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:112)
	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.shutdown(FsDatasetImpl.java:2127)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:2016)
	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:1985)
	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1962)
	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1936)
	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1929)
	at org.apache.hadoop.hdfs.TestDatanodeReport.testDatanodeReport(TestDatanodeReport.java:144)

The test shuts down datanode, and then shutdown cluster, which shuts down the a datanode twice. Resetting the FsDatasetSpi reference in DataNode to null resolves the issue.

Issue Links

• is depended upon by: HADOOP-18198 Release Hadoop 3.3.3: hadoop-3.3.2 with some fixes