[HIVE-24668] Improve FileSystem usage in dynamic partition handling

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Assignee: Peter Varga
• Reporter: Peter Varga
• Created: 20/Jan/21 14:17
• Updated: 17/Nov/22 08:48
• Resolved: 24/Feb/21 07:32

Description

Possible improvements:

• In the Movetask process both getFullDPSpecs and later Hive::getValidPartitionsInPath do a listing for dynamic partitions in the table, the result of the first can be reused
• Hive::listFilesCreatedByQuery does the recursive listing on Hive side, the native recursive listing should be used
• if we add a new partition we populate the quickstats, that will do another listing for the new partition, the files are already collected for the writeNotificationlogs, that can be used

Sub-Tasks

• Improve Filesystem usage in Hive::loadPartitionInternal: Closed
• Reuse FullDPSpecs in loadDynamicPartitions to avoid double listing: Closed
• Collect dynamic partition info in FileSink for direct insert and reuse it in Movetask: Closed
• Reuse committed filelist from directInsert manifest during loadPartition: Closed