[HDDS-1449] JVM Exit in datanode while committing a key

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.3.0
• Fix Version/s: 0.4.1
• Component/s: Ozone Datanode
• Labels: MiniOzoneChaosCluster pull-request-available
• Target Version/s: 0.5.0
• Sprint: Done
• Assignee: Mukul Kumar Singh
• Reporter: Mukul Kumar Singh
• Created: 22/Apr/19 16:07
• Updated: 05/Aug/19 14:46
• Resolved: 22/May/19 11:49

Description

Saw the following trace in MiniOzoneChaosCluster run.

C  [librocksdbjni17271331491728127.jnilib+0x9755c]  Java_org_rocksdb_RocksDB_write0+0x1c
J 13917  org.rocksdb.RocksDB.write0(JJJ)V (0 bytes) @ 0x00000001102ff62e [0x00000001102ff580+0xae]
J 17167 C2 org.apache.hadoop.utils.RocksDBStore.writeBatch(Lorg/apache/hadoop/utils/BatchOperation;)V (260 bytes) @ 0x0000000111bbd01c [0x0000000111bbcde0+0x23c]
J 20434 C1 org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.putBlock(Lorg/apache/hadoop/ozone/container/common/interfaces/Container;Lorg/apache/hadoop/ozone/container/common/helpers/BlockData;)J (261 bytes) @ 0x0000000111c267ac [0x0000000111c25640+0x116c]
J 19262 C2 org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandRequestProto;Lorg/apache/hadoop/ozone/container/common/transport/server/ratis/DispatcherContext;)Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandResponseProto; (866 bytes) @ 0x00000001125c5aa0 [0x00000001125c1560+0x4540]
J 15095 C2 org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandRequestProto;Lorg/apache/hadoop/ozone/container/common/transport/server/ratis/DispatcherContext;)Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandResponseProto; (142 bytes) @ 0x0000000110ffc940 [0x0000000110ffc0c0+0x880]
J 19301 C2 org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandRequestProto;Lorg/apache/hadoop/ozone/container/common/transport/server/ratis/DispatcherContext;)Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandResponseProto; (146 bytes) @ 0x0000000111396144 [0x0000000111395e60+0x2e4]
J 15997 C2 org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine$$Lambda$776.get()Ljava/lang/Object; (16 bytes) @ 0x0000000110138e54 [0x0000000110138d80+0xd4]
J 15970 C2 java.util.concurrent.CompletableFuture$AsyncSupply.run()V (61 bytes) @ 0x000000010fc80094 [0x000000010fc80000+0x94]
J 17368 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x0000000110b0a7a0 [0x0000000110b0a5a0+0x200]
J 7389 C1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @ 0x000000011012a004 [0x0000000110129f00+0x104]
J 6837 C1 java.lang.Thread.run()V (17 bytes) @ 0x000000011002b144 [0x000000011002b000+0x144]
v  ~StubRoutines::call_stub
V  [libjvm.dylib+0x2ef1f6]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x6ae
V  [libjvm.dylib+0x2ef99a]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x164
V  [libjvm.dylib+0x2efb46]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x4a
V  [libjvm.dylib+0x34a46d]  thread_entry(JavaThread*, Thread*)+0x7c
V  [libjvm.dylib+0x56eb0f]  JavaThread::thread_main_inner()+0x9b
V  [libjvm.dylib+0x57020a]  JavaThread::run()+0x1c2
V  [libjvm.dylib+0x48d4a6]  java_start(Thread*)+0xf6
C  [libsystem_pthread.dylib+0x3305]  _pthread_body+0x7e
C  [libsystem_pthread.dylib+0x626f]  _pthread_start+0x46
C  [libsystem_pthread.dylib+0x2415]  thread_start+0xd
C  0x0000000000000000

Issue Links

• causes: HDDS-1602 Fix TestContainerPersistence#testDeleteBlockTwice
• links to: GitHub Pull Request #825