[HUDI-3717] Avoid double-listing w/in BaseHoodieTableFileIndex

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Story Points: 3
• Assignee: Alexey Kudinkin
• Reporter: Alexey Kudinkin
• Created: 26/Mar/22 02:05
• Updated: 30/Nov/22 23:53
• Resolved: 30/Nov/22 23:53

Description

Currently in `BaseHoodieTableFileIndex::loadPartitionPathFiles` essentially does file-listing twice: 

• Once when `getAllQueryPartitionPaths` is invoked
• Second time when `getFilesInPartitions` is invoked

 

While this will not result in double-listing of the files on FS (b/c of `FIleStatusCache`, if any), this leads however to MT being queried twice: 

 

Issue Links

• is fixed by: HUDI-4812 Lazy partition listing and file groups fetching in Spark Query