Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16949

Leak of threads from Get-Input-Paths and Get-Input-Summary thread pool

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • HiveServer2
    • None

    Description

      The commit 20210de which was part of HIVE-15546 introduced a thread pool which is not shutdown upon completion of its threads. This leads to a leak of threads for each query which uses more than 1 partition. They are not removed automatically. When queries spanning multiple partitions are made the number of threads increases and is never reduced. On my machine hiveserver2 starts to get slower and slower once 10k threads are reached.

      Thread pools only shutdown automatically in special circumstances (see documentation section Finalization). This is not currently the case for the Get-Input-Paths thread pool. I would add a pool.shutdown() in a finally block just before returning the result to make sure the threads are really shutdown.

      My current workaround is to set hive.exec.input.listing.max.threads = 1. This prevents the the thread pool from being spawned [1] [2].

      The same issue probably also applies to the Get-Input-Summary thread pool.

      Attachments

        1. HIVE-16949.1.patch
          22 kB
          Sahil Takiar

        Issue Links

          Activity

            People

              stakiar Sahil Takiar
              birger Birger Brunswiek
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: