Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16949

Leak of threads from Get-Input-Paths and Get-Input-Summary thread pool

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: HiveServer2
    • Labels:
      None

      Description

      The commit 20210de which was part of HIVE-15546 introduced a thread pool which is not shutdown upon completion of its threads. This leads to a leak of threads for each query which uses more than 1 partition. They are not removed automatically. When queries spanning multiple partitions are made the number of threads increases and is never reduced. On my machine hiveserver2 starts to get slower and slower once 10k threads are reached.

      Thread pools only shutdown automatically in special circumstances (see documentation section Finalization). This is not currently the case for the Get-Input-Paths thread pool. I would add a pool.shutdown() in a finally block just before returning the result to make sure the threads are really shutdown.

      My current workaround is to set hive.exec.input.listing.max.threads = 1. This prevents the the thread pool from being spawned [1] [2].

      The same issue probably also applies to the Get-Input-Summary thread pool.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stakiar Sahil Takiar
                Reporter:
                birger Birger Brunswiek
              • Votes:
                1 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: