[YARN-3585] NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabled

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.6.1, 2.8.0, 2.7.1, 3.0.0-alpha1
• Labels: 2.6.1-candidate
• Target Version/s: 2.7.1
• Hadoop Flags: Reviewed
• Assignee: Rohith Sharma K S
• Reporter: Peng Zhang
• Created: 06/May/15 14:00
• Updated: 19/Nov/20 11:27
• Resolved: 03/Jun/15 19:48

Description

With NM recovery enabled, after decommission, nodemanager log show stop but process cannot end.
non daemon thread:

"DestroyJavaVM" prio=10 tid=0x00007f3460011800 nid=0x29ec waiting on condition [0x0000000000000000]
"leveldb" prio=10 tid=0x00007f3354001800 nid=0x2a97 runnable [0x0000000000000000]
"VM Thread" prio=10 tid=0x00007f3460167000 nid=0x29f8 runnable 
"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f3460020000 nid=0x29ed runnable 
"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f3460022000 nid=0x29ee runnable 
"Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f3460024000 nid=0x29ef runnable 
"Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f3460025800 nid=0x29f0 runnable 
"Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00007f3460027800 nid=0x29f1 runnable 
"Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00007f3460029000 nid=0x29f2 runnable 
"Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00007f346002b000 nid=0x29f3 runnable 
"Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00007f346002d000 nid=0x29f4 runnable 
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f3460120800 nid=0x29f7 runnable 
"Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00007f346011c800 nid=0x29f5 runnable 
"Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00007f346011e800 nid=0x29f6 runnable 
"VM Periodic Task Thread" prio=10 tid=0x00007f346019f800 nid=0x2a01 waiting on condition 

and jni leveldb thread stack

Thread 12 (Thread 0x7f33dd842700 (LWP 10903)):
#0  0x0000003d8340b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f33dfce2a3b in leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper(void*) () from /tmp/libleveldbjni-64-1-6922178968300745716.8
#2  0x0000003d83407851 in start_thread () from /lib64/libpthread.so.0
#3  0x0000003d830e811d in clone () from /lib64/libc.so.6

Issue Links

• duplicates: YARN-3640 NodeManager JVM continues to run after SHUTDOWN event is triggered.
• is related to: YARN-8120 JVM can crash with SIGSEGV when exiting due to custom leveldb logger