Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11041

Unable to unregister FsDatasetState MBean if DataNode is shutdown twice

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Trivial
    • Resolution: Fixed
    • None
    • 3.4.0, 2.10.2, 3.2.3, 3.3.3
    • datanode
    • None
    • Reviewed

    Description

      I saw error message like the following in some tests

      2016-10-21 04:09:03,900 [main] WARN  util.MBeans (MBeans.java:unregister(114)) - Error unregistering Hadoop:service=DataNode,name=FSDatasetState-33cd714c-0b1a-471f-8efe-f431d7d874bc
      javax.management.InstanceNotFoundException: Hadoop:service=DataNode,name=FSDatasetState-33cd714c-0b1a-471f-8efe-f431d7d874bc
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
      	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
      	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:112)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.shutdown(FsDatasetImpl.java:2127)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:2016)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:1985)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1962)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1936)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1929)
      	at org.apache.hadoop.hdfs.TestDatanodeReport.testDatanodeReport(TestDatanodeReport.java:144)
      

      The test shuts down datanode, and then shutdown cluster, which shuts down the a datanode twice. Resetting the FsDatasetSpi reference in DataNode to null resolves the issue.

      Attachments

        1. HDFS-11041.03.patch
          1 kB
          Wei-Chiu Chuang
        2. HDFS-11041.02.patch
          0.7 kB
          Wei-Chiu Chuang
        3. HDFS-11041.01.patch
          0.6 kB
          Wei-Chiu Chuang

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            weichiu Wei-Chiu Chuang
            weichiu Wei-Chiu Chuang
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment