[HIVE-22411] Performance degradation on single row inserts

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Hive, Transactions
• Assignee: Attila Magyar
• Reporter: Attila Magyar
• Created: 28/Oct/19 08:36
• Updated: 17/Nov/22 08:50
• Resolved: 12/Nov/19 15:35

Description

Executing single insert statements on a transactional table effects write performance on a s3 file system. Each insert creates a new delta directory. After each insert hive calculates statistics like number of file in the table and total size of the table. In order to calculate these, it traverses the directory recursively. During the recursion for each path a separate listStatus call is executed. In the end the more delta directory you have the more time it takes to calculate the statistics.

Therefore insertion time goes up linearly:

The fix is to use fs.listFiles(path, /*recursive*/ true) instead the handcrafter recursive method/

Issue Links

• breaks: HIVE-25960 Fix S3a recursive listing logic
• is blocked by: HADOOP-16673 Add filter parameter to FileSystem>>listFiles
• links to: review board
• links to: review board