[HIVE-25277] Slow Hive partition deletion for Cloud object stores with expensive ListFiles

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: All Versions
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Standalone Metastore
• Labels: pull-request-available
• Target Version/s: 2.3.9, 3.1.3, 4.0.0
• Hadoop Flags: Reviewed
• Assignee: Zhou Fang
• Reporter: Zhou Fang
• Created: 22/Jun/21 17:20
• Updated: 07/Dec/22 07:27
• Resolved: 05/Sep/21 05:02

Description

Deleting a Hive partition is slow when use a Cloud object store as the warehouse for which ListFiles is expensive. A root cause is that the recursive parent dir deletion is very inefficient: there are many duplicated calls to isEmpty (ListFiles is called at the end). This fix sorts the parents to delete according to the path size, and always processes the longest one (e.g., a/b/c is always before a/b). As a result, each parent path is only needed to be checked once.

Issue Links

• links to: GitHub Pull Request #2421