[HIVE-17586] Make HS2 BackgroundOperationPool not fixed

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.1.0
• Component/s: HiveServer2
• Assignee: Xuefu Zhang
• Reporter: Xuefu Zhang
• Created: 22/Sep/17 21:28
• Updated: 27/Sep/17 03:51

Description

Currently the threadpool for background asynchronous operatons has a fixed size controled by hive.server2.async.exec.threads. However, the thread factory supplied for this threadpool is ThreadFactoryWithGarbageCleanup which creates ThreadWithGarbageCleanup. Since this is a fixed threadpool, the thread is actually never killed, defecting the purpose of garbage cleanup as noted in the thread class name. On the other hand, since these threads never go away, significant resources such as threadlocal variables (classloaders, hiveconfs, etc) are holding up even if there is no operation running. This can lead to escalated HS2 memory usage.

Ideally, the threadpool should not be fixed, allowing thread to die out so resources can be reclaimed. The existing config hive.server2.async.exec.threads is treated as the max, and we can add a min for the threadpool hive.server2.async.exec.min.threads. Default value for this configure is -1, which keeps the existing behavior.