Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-1449

JVM Exit in datanode while committing a key

    XMLWordPrintableJSON

    Details

    • Target Version/s:
    • Sprint:
      HDDS Biscayne

      Description

      Saw the following trace in MiniOzoneChaosCluster run.

      C  [librocksdbjni17271331491728127.jnilib+0x9755c]  Java_org_rocksdb_RocksDB_write0+0x1c
      J 13917  org.rocksdb.RocksDB.write0(JJJ)V (0 bytes) @ 0x00000001102ff62e [0x00000001102ff580+0xae]
      J 17167 C2 org.apache.hadoop.utils.RocksDBStore.writeBatch(Lorg/apache/hadoop/utils/BatchOperation;)V (260 bytes) @ 0x0000000111bbd01c [0x0000000111bbcde0+0x23c]
      J 20434 C1 org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.putBlock(Lorg/apache/hadoop/ozone/container/common/interfaces/Container;Lorg/apache/hadoop/ozone/container/common/helpers/BlockData;)J (261 bytes) @ 0x0000000111c267ac [0x0000000111c25640+0x116c]
      J 19262 C2 org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandRequestProto;Lorg/apache/hadoop/ozone/container/common/transport/server/ratis/DispatcherContext;)Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandResponseProto; (866 bytes) @ 0x00000001125c5aa0 [0x00000001125c1560+0x4540]
      J 15095 C2 org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandRequestProto;Lorg/apache/hadoop/ozone/container/common/transport/server/ratis/DispatcherContext;)Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandResponseProto; (142 bytes) @ 0x0000000110ffc940 [0x0000000110ffc0c0+0x880]
      J 19301 C2 org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandRequestProto;Lorg/apache/hadoop/ozone/container/common/transport/server/ratis/DispatcherContext;)Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandResponseProto; (146 bytes) @ 0x0000000111396144 [0x0000000111395e60+0x2e4]
      J 15997 C2 org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine$$Lambda$776.get()Ljava/lang/Object; (16 bytes) @ 0x0000000110138e54 [0x0000000110138d80+0xd4]
      J 15970 C2 java.util.concurrent.CompletableFuture$AsyncSupply.run()V (61 bytes) @ 0x000000010fc80094 [0x000000010fc80000+0x94]
      J 17368 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x0000000110b0a7a0 [0x0000000110b0a5a0+0x200]
      J 7389 C1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @ 0x000000011012a004 [0x0000000110129f00+0x104]
      J 6837 C1 java.lang.Thread.run()V (17 bytes) @ 0x000000011002b144 [0x000000011002b000+0x144]
      v  ~StubRoutines::call_stub
      V  [libjvm.dylib+0x2ef1f6]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x6ae
      V  [libjvm.dylib+0x2ef99a]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x164
      V  [libjvm.dylib+0x2efb46]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x4a
      V  [libjvm.dylib+0x34a46d]  thread_entry(JavaThread*, Thread*)+0x7c
      V  [libjvm.dylib+0x56eb0f]  JavaThread::thread_main_inner()+0x9b
      V  [libjvm.dylib+0x57020a]  JavaThread::run()+0x1c2
      V  [libjvm.dylib+0x48d4a6]  java_start(Thread*)+0xf6
      C  [libsystem_pthread.dylib+0x3305]  _pthread_body+0x7e
      C  [libsystem_pthread.dylib+0x626f]  _pthread_start+0x46
      C  [libsystem_pthread.dylib+0x2415]  thread_start+0xd
      C  0x0000000000000000
      

        Attachments

        1. 2019-04-22--20-23-56-IST.MiniOzoneChaosCluster.log
          3.56 MB
          Mukul Kumar Singh
        2. hs_err_pid67466.log
          409 kB
          Mukul Kumar Singh

          Issue Links

            Activity

              People

              • Assignee:
                msingh Mukul Kumar Singh
                Reporter:
                msingh Mukul Kumar Singh
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m