Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7240 Scaling HDFS
  3. HDFS-12899

Ozone: SCM: BlockManagerImpl close is called twice during StorageContainerManager#stop

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • HDFS-7240
    • ozone
    • None
    • Reviewed

    Description

      As part of StorageContainerManager#stop, we are calling scmBlockManager#stop which will internally do BlockManagerImpl#close and again explicitly we are calling scmBlockManager#close using IOUtils.cleanupWithLogger(LOG, scmBlockManager). This causes RocksDBStore#close to be called twice which inturn does MBeans#unregister twice, resulting in the following exception trace (WARN) during the second call

      2017-12-06 22:30:06,316 [main] WARN  util.MBeans (MBeans.java:unregister(137)) - Error unregistering Hadoop:service=Ozone,name=RocksDbStore,dbName=block.db
      javax.management.InstanceNotFoundException: Hadoop:service=Ozone,name=RocksDbStore,dbName=block.db
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
      	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
      	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:135)
      	at org.apache.hadoop.utils.RocksDBStore.close(RocksDBStore.java:368)
      	at org.apache.hadoop.ozone.scm.block.BlockManagerImpl.close(BlockManagerImpl.java:506)
      	at org.apache.hadoop.io.IOUtils.cleanupWithLogger(IOUtils.java:278)
      	at org.apache.hadoop.ozone.scm.StorageContainerManager.stop(StorageContainerManager.java:900)
      --------stacktrace truncated----------
      2017-12-06 22:30:06,317 [main] WARN  util.MBeans (MBeans.java:unregister(137)) - Error unregistering Hadoop:service=Ozone,name=RocksDbStore,dbName=deletedBlock.db
      javax.management.InstanceNotFoundException: Hadoop:service=Ozone,name=RocksDbStore,dbName=deletedBlock.db
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
      	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
      	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:135)
      	at org.apache.hadoop.utils.RocksDBStore.close(RocksDBStore.java:368)
      	at org.apache.hadoop.ozone.scm.block.DeletedBlockLogImpl.close(DeletedBlockLogImpl.java:326)
      	at org.apache.hadoop.ozone.scm.block.BlockManagerImpl.close(BlockManagerImpl.java:509)
      	at org.apache.hadoop.io.IOUtils.cleanupWithLogger(IOUtils.java:278)
      	at org.apache.hadoop.ozone.scm.StorageContainerManager.stop(StorageContainerManager.java:900)
      --------stacktrace truncated----------
      

      Attachments

        1. HDFS-12899-HDFS-7240.000.patch
          0.7 kB
          Nandakumar

        Activity

          People

            nanda Nandakumar
            nanda Nandakumar
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: