Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3585

NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabled

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      With NM recovery enabled, after decommission, nodemanager log show stop but process cannot end.
      non daemon thread:

      "DestroyJavaVM" prio=10 tid=0x00007f3460011800 nid=0x29ec waiting on condition [0x0000000000000000]
      "leveldb" prio=10 tid=0x00007f3354001800 nid=0x2a97 runnable [0x0000000000000000]
      "VM Thread" prio=10 tid=0x00007f3460167000 nid=0x29f8 runnable 
      "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f3460020000 nid=0x29ed runnable 
      "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f3460022000 nid=0x29ee runnable 
      "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f3460024000 nid=0x29ef runnable 
      "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f3460025800 nid=0x29f0 runnable 
      "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00007f3460027800 nid=0x29f1 runnable 
      "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00007f3460029000 nid=0x29f2 runnable 
      "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00007f346002b000 nid=0x29f3 runnable 
      "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00007f346002d000 nid=0x29f4 runnable 
      "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f3460120800 nid=0x29f7 runnable 
      "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00007f346011c800 nid=0x29f5 runnable 
      "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00007f346011e800 nid=0x29f6 runnable 
      "VM Periodic Task Thread" prio=10 tid=0x00007f346019f800 nid=0x2a01 waiting on condition 
      

      and jni leveldb thread stack

      Thread 12 (Thread 0x7f33dd842700 (LWP 10903)):
      #0  0x0000003d8340b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007f33dfce2a3b in leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper(void*) () from /tmp/libleveldbjni-64-1-6922178968300745716.8
      #2  0x0000003d83407851 in start_thread () from /lib64/libpthread.so.0
      #3  0x0000003d830e811d in clone () from /lib64/libc.so.6
      

      Attachments

        1. YARN-3585.patch
          4 kB
          Rohith Sharma K S
        2. 0001-YARN-3585.patch
          2 kB
          Rohith Sharma K S

        Issue Links

          Activity

            People

              rohithsharma Rohith Sharma K S
              peng.zhang Peng Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: