Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-23164

Server is not properly terminated because of non-daemon threads

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0-alpha-1
    • HiveServer2
    • None

    Description

      HiveServer2 which receives the deregister command is at first preparing shutdown. If there's no remaining session, HiveServer2.stop() is called to shut down. But I found the case that the HiveServer2 JVM is not terminated even if HiveServer2.stop() has been called and processed. The case is always occurred when the local(embedded) metastore is used.

      I've attached the full thread dump describing the situation.

      thread_dump_hiveserver2_is_not_terminated.txt

      In this thread dump, you can see some bunch of 'daemon' threads, NO main thread, and some 'non-daemon' thread(or user thread)s. As specified by https://www.baeldung.com/java-daemon-thread, if there is at least one user thread exists, JVM does not terminate. (Note that DestroyJavaVM thread is non-daemon but it's special.)

       

      "pool-8-thread-1" #24 prio=5 os_prio=0 tid=0x00007f52ad1fc000 nid=0x821c waiting on condition [0x00007f525c500000]
       java.lang.Thread.State: TIMED_WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for <0x00000003cfa057c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:748)
      Locked ownable synchronizers:
       - None
      

      The thread above is created by the ScheduledThreadPoolExecutor(int coreSize) constructor with default ThreadFactory which always makes a thread non-daemon. If such thread pool is not destroyed by calling ScheduledThreadPoolExecutor.shutdown() method, JVM cannot terminate! The only way to kill is TERM signal. If JVM receives TERM signal, it ignores non-daemon threads and terminates.

      So I have been digging modules which create ScheduledThreadPoolExecutor with non-daemon threads and I got it. As you may guess, it's the local(embedded) metastore. ScheduledThreadPoolExecutor is created by org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler#startAlwaysTaskThreads() and ScheduledThreadPoolExecutor.shutdown() is never called.

      Plus, I found another usage of creating such ScheduledThreadPoolExecutor and not calling its shutdown. 

      Attachments

        1. HIVE-23164.01.patch
          2 kB
          Eugene Chung
        2. HIVE-23164.02.patch
          4 kB
          Eugene Chung
        3. HIVE-23164.03.patch
          4 kB
          Eugene Chung
        4. HIVE-23164.04.patch
          4 kB
          Eugene Chung
        5. thread_dump_hiveserver2_is_not_terminated.txt
          13 kB
          Eugene Chung

        Issue Links

          Activity

            People

              euigeun_chung Eugene Chung
              euigeun_chung Eugene Chung
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: