[HDDS-1777] JVM crash while shutting down Ozone datanode in ShutdownHook

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.4.0
• Component/s: Ozone Datanode
• Labels: MiniOzoneChaosCluster
• Target Version/s: 1.4.0
• Assignee: Unassigned
• Reporter: Mukul Kumar Singh
• Created: 10/Jul/19 03:20
• Updated: 09/Aug/23 09:10
• Resolved: 09/Aug/23 09:10

Description

JVM crash while shutting down Ozone datanode in ShutdownHook with the following exception.

Stack: [0x0000700087910000,0x0000700087a10000],  sp=0x0000700087a0db20,  free space=1014k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libsystem_platform.dylib+0x1d09]  _platform_memmove$VARIANT$Haswell+0x29
C  [libzip.dylib+0x3399]  newEntry+0x65b
C  [libzip.dylib+0x352d]  ZIP_GetEntry2+0xd4
C  [libzip.dylib+0x2238]  Java_java_util_zip_ZipFile_getEntry+0xcf
J 108  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00000001087d23ce [0x00000001087d2300+0xce]
J 4302 C2 java.util.jar.JarFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (22 bytes) @ 0x0000000108d659e8 [0x0000000108d65660+0x388]
J 4583 C2 sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (85 bytes) @ 0x0000000108954b24 [0x0000000108954aa0+0x84]
J 25559 C2 java.net.URLClassLoader$2.run()Ljava/lang/Object; (5 bytes) @ 0x000000010c2c04c8 [0x000000010c2c0380+0x148]
v  ~StubRoutines::call_stub
V  [libjvm.dylib+0x2ef1f6]
V  [libjvm.dylib+0x34fb24]
J 4197  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (0 bytes) @ 0x0000000108e36055 [0x0000000108e35f80+0xd5]
J 25557 C2 java.net.URLClassLoader.findResource(Ljava/lang/String;)Ljava/net/URL; (37 bytes) @ 0x0000000109c8505c [0x0000000109c84fc0+0x9c]
J 25556 C2 java.lang.ClassLoader.getResource(Ljava/lang/String;)Ljava/net/URL; (36 bytes) @ 0x000000010c2bb984 [0x000000010c2bb640+0x344]
j  org.apache.hadoop.conf.Configuration.getResource(Ljava/lang/String;)Ljava/net/URL;+5
j  org.apache.hadoop.conf.Configuration.getStreamReader(Lorg/apache/hadoop/conf/Configuration$Resource;Z)Lorg/codehaus/stax2/XMLStreamReader2;+51
J 7480 C1 org.apache.hadoop.conf.Configuration.loadResource(Ljava/util/Properties;Lorg/apache/hadoop/conf/Configuration$Resource;Z)Lorg/apache/hadoop/conf/Configuration$Resource; (322 bytes) @ 0x0000000109964bf4 [0x0000000109964700+0x4f4]
j  org.apache.hadoop.conf.Configuration.loadResources(Ljava/util/Properties;Ljava/util/ArrayList;Z)V+50
J 8094 C2 org.apache.hadoop.conf.Configuration.getProps()Ljava/util/Properties; (162 bytes) @ 0x0000000109af1fc0 [0x0000000109af1d40+0x280]
J 15086 C2 org.apache.hadoop.conf.Configuration.get(Ljava/lang/String;)Ljava/lang/String; (64 bytes) @ 0x000000010ae9ee78 [0x000000010ae9eb20+0x358]
J 21716 C1 org.apache.hadoop.conf.Configuration.getTimeDuration(Ljava/lang/String;JLjava/util/concurrent/TimeUnit;)J (25 bytes) @ 0x000000010b8ab4d4 [0x000000010b8ab3c0+0x114]
j  org.apache.hadoop.util.ShutdownHookManager.getShutdownTimeout(Lorg/apache/hadoop/conf/Configuration;)J+9
j  org.apache.hadoop.util.ShutdownHookManager$HookEntry.<init>(Ljava/lang/Runnable;I)V+10
j  org.apache.hadoop.util.ShutdownHookManager.removeShutdownHook(Ljava/lang/Runnable;)Z+30
j  org.apache.hadoop.ozone.container.common.volume.VolumeSet.shutdown()V+22
j  org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.stop()V+43
j  org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.close()V+159
j  org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.stopDaemon()V+25
j  org.apache.hadoop.ozone.HddsDatanodeService.stop()V+101
j  org.apache.hadoop.ozone.MiniOzoneClusterImpl.restartHddsDatanode(IZ)V+15
j  org.apache.hadoop.ozone.MiniOzoneChaosCluster.shutdownNodes()V+103
j  org.apache.hadoop.ozone.MiniOzoneChaosCluster.fail()V+48
j  org.apache.hadoop.ozone.MiniOzoneChaosCluster$$Lambda$507.run()V+4
J 22514 C2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (14 bytes) @ 0x000000010ba89fec [0x000000010ba89fa0+0x4c]
J 23026 C1 java.util.concurrent.FutureTask.runAndReset()Z (128 bytes) @ 0x000000010bd067ec [0x000000010bd06580+0x26c]
J 22790 C2 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V (59 bytes) @ 0x000000010bdabd8c [0x000000010bdabb20+0x26c]
J 7745 C1 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x0000000109a1c9e4 [0x0000000109a1b9c0+0x1024]
J 7178 C1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @ 0x00000001098452c4 [0x00000001098451c0+0x104]
J 6840 C1 java.lang.Thread.run()V (17 bytes) @ 0x000000010977d2c4 [0x000000010977d180+0x144]
v  ~StubRoutines::call_stub
V  [libjvm.dylib+0x2ef1f6]
V  [libjvm.dylib+0x2ef99a]
V  [libjvm.dylib+0x2efb46]
V  [libjvm.dylib+0x34a46d]
V  [libjvm.dylib+0x56eb0f]
V  [libjvm.dylib+0x57020a]
V  [libjvm.dylib+0x48d4a6]
C  [libsystem_pthread.dylib+0x3305]  _pthread_body+0x7e
C  [libsystem_pthread.dylib+0x626f]  _pthread_start+0x46
C  [libsystem_pthread.dylib+0x2415]  thread_start+0xd
C  0x0000000000000000