[HUDI-712] Improve exporter performance and memory usage

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Component/s: Utilities
• Labels: pull-request-available
• Assignee: Raymond Xu
• Reporter: Raymond Xu
• Created: 14/Mar/20 07:07
• Updated: 21/Dec/22 05:11
• Resolved: 22/Nov/22 11:21

Description

https://github.com/apache/incubator-hudi/blob/99b7e9eb9ef8827c1e06b7e8621b6be6403b061e/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotExporter.java#L103-L107

The way the data file list for export is collected can be improved due to

• not parallelized among partitions
• the list can be too large
• listing partition to get the latest files requires scanning all files (RFC-15 could solve this)

 

Issue Links

• relates to: HUDI-344 Hudi Dataset Snapshot Exporter
• relates to: HUDI-5261 Use proper parallelism for engine context APIs
• links to: GitHub Pull Request #7267