[HDDS-507] EventQueue should be shutdown on SCM shutdown

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.2.1
• Target Version/s: 0.2.1
• Assignee: Xiaoyu Yao
• Reporter: Xiaoyu Yao
• Created: 19/Sep/18 02:55
• Updated: 12/Nov/18 11:10
• Resolved: 19/Sep/18 19:55

Description

This can be repro-ed by when TestNodeFailure multiple times. Jenkins sometimes also hit this. 

 

Current thread (0x00007fbe6f018800):  JavaThread "EventQueue-PipelineCloseForPipelineCloseHandler" daemon [_thread_in_native, id=58639, stack(0x0000700018009000,0x0000700018109000)]

 

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000000040000001d

 

 

 

Stack: [0x0000700018009000,0x0000700018109000],  sp=0x0000700018108128,  free space=1020k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

C  [librocksdbjni6372054043595793813.jnilib+0x163ac8]  rocksdb::GetColumnFamilyID(rocksdb::ColumnFamilyHandle*)+0x8

C  [librocksdbjni6372054043595793813.jnilib+0x228368]  rocksdb::DB::Put(rocksdb::WriteOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::Slice const&)+0x58

C  [librocksdbjni6372054043595793813.jnilib+0x2282fe]  rocksdb::DBImpl::Put(rocksdb::WriteOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::Slice const&)+0xe

C  [librocksdbjni6372054043595793813.jnilib+0x171c84]  rocksdb::CompactedDBImpl::Open(rocksdb::Options const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DB**)+0x2a4

C  [librocksdbjni6372054043595793813.jnilib+0x971f7]  rocksdb_put_helper(JNIEnv_*, rocksdb::DB*, rocksdb::WriteOptions const&, rocksdb::ColumnFamilyHandle*, _jbyteArray*, int, int, _jbyteArray*, int, int)+0x137

j  org.rocksdb.RocksDB.put(JJ[BII[BII)V+0

j  org.rocksdb.RocksDB.put(Lorg/rocksdb/WriteOptions;[B[B)V+17

j  org.apache.hadoop.utils.RocksDBStore.put([B[B)V+10

j  org.apache.hadoop.hdds.scm.pipelines.PipelineSelector.updatePipelineState(Lorg/apache/hadoop/hdds/scm/container/common/helpers/Pipeline;Lorg/apache/hadoop/hdds/protocol/proto/HddsProtos$LifeCycleEvent;)V+222

j  org.apache.hadoop.hdds.scm.pipelines.PipelineSelector.finalizePipeline(Lorg/apache/hadoop/hdds/scm/container/common/helpers/Pipeline;)V+75

j  org.apache.hadoop.hdds.scm.container.ContainerMapping.handlePipelineClose(Lorg/apache/hadoop/hdds/scm/container/common/helpers/PipelineID;)V+18

j  org.apache.hadoop.hdds.scm.pipelines.PipelineCloseHandler.onMessage(Lorg/apache/hadoop/hdds/scm/container/common/helpers/PipelineID;Lorg/apache/hadoop/hdds/server/events/EventPublisher;)V+5

j  org.apache.hadoop.hdds.scm.pipelines.PipelineCloseHandler.onMessage(Ljava/lang/Object;Lorg/apache/hadoop/hdds/server/events/EventPublisher;)V+6

J 5844 C1 org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(Lorg/apache/hadoop/hdds/server/events/EventHandler;Ljava/lang/Object;Lorg/apache/hadoop/hdds/server/events/EventPublisher;)V (41 bytes) @ 0x0000000115c80bc4 [0x0000000115c80aa0+0x124]

J 5670 C1 org.apache.hadoop.hdds.server.events.SingleThreadExecutor$$Lambda$143.run()V (20 bytes) @ 0x00000001168f625c [0x00000001168f61c0+0x9c]

j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95

J 3226 C1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @ 0x0000000116356e44 [0x0000000116356d40+0x104]

J 3107 C1 java.lang.Thread.run()V (17 bytes) @ 0x0000000115d7b0c4 [0x0000000115d7af80+0x144]

v  ~StubRoutines::call_stub

V  [libjvm.dylib+0x2ef1f6]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x6ae

V  [libjvm.dylib+0x2ef99a]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x164

V  [libjvm.dylib+0x2efb46]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x4a

V  [libjvm.dylib+0x34a46d]  thread_entry(JavaThread*, Thread*)+0x7c

V  [libjvm.dylib+0x56eb0f]  JavaThread::thread_main_inner()+0x9b

V  [libjvm.dylib+0x57020a]  JavaThread::run()+0x1c2

V  [libjvm.dylib+0x48d4a6]  java_start(Thread*)+0xf6

C  [libsystem_pthread.dylib+0x3661]  _pthread_body+0x154

C  [libsystem_pthread.dylib+0x350d]  _pthread_body+0x0

C  [libsystem_pthread.dylib+0x2bf9]  thread_start+0xd

C  0x0000000000000000

Issue Links

• is duplicated by: HDDS-474 TestNodeFailure fails in trunk