[HUDI-3895] Make sure Hudi relations do proper file-split packing (on par w/ Spark)

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Labels: pull-request-available
• Story Points: 1
• Epic Link: Hudi Spark Datasource
• Assignee: Alexey Kudinkin
• Reporter: Alexey Kudinkin
• Created: 16/Apr/22 20:28
• Updated: 19/Apr/22 02:55
• Resolved: 18/Apr/22 20:59

Description

While investigating on HUDI-3891, it was discovered that upon introduction of Hudi's own Spark's Relation implementations, file-split packing algorithm was inadvertently subverted: 

Spark algorithm does greedy packing which relies on the list of file-splits being ordered by the file size (descending in order).

Issue Links

• causes: HUDI-3891 Investigate Hudi vs Raw Parquet table discrepancy
• links to: GH 5337
• links to: GitHub Pull Request #5337