[HIVE-22081] Hivemetastore Performance: Compaction Initiator Thread overwhelmed if there are too many Table/partitions are eligible for compaction

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Transactions
• Assignee: Rajkumar Singh
• Reporter: Rajkumar Singh
• Created: 03/Aug/19 00:15
• Updated: 17/Nov/22 08:51
• Resolved: 21/Aug/19 07:59

Description

if Automatic Compaction is turned on, Initiator thread check for potential table/partitions which are eligible for compactions and run some checks in for loop before requesting compaction for eligibles. Though initiator thread is configured to run at interval 5 min default, in case of many objects it keeps on running as these checks are IO intensive and hog cpu.
In the proposed changes, I am planning to do
1. passing less object to for loop by filtering out the objects based on the condition which we are checking within the loop.
2. Doing Async call using future to determine compaction type(this is where we do FileSystem calls)