[HUDI-4917] Optimized the way to get HoodieBaseFile of loadColumnRangesFromFiles of Bloom Index

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Component/s: index
• Labels: pull-request-available
• Assignee: Chuang Lee
• Reporter: Chuang Lee
• Created: 26/Sep/22 03:46
• Updated: 16/Feb/23 07:12
• Resolved: 16/Feb/23 07:12

Description

When using Bloom Index for loadColumnRangesFromFiles in the tagLocation process, the existing method is to obtain the hoodieBaseFile by requesting the Driver side. When the amount of data is large and the parallelism is high, there is a certain network performance bottleneck, resulting in very slow tagloacation.
However, hoodieBaseFile can be obtained directly through HoodieIndexUtils.getLatestBaseFilesForAllPartitions() in loadColumnRangesFromFiles(), so it can effectively improve the performance of TagLoaction of Bloom Index.

Issue Links

• links to: GitHub Pull Request #6793
• links to: GitHub Pull Request #7456