[HUDI-3896] Support SchemaPruning optimization for Hudi's own relations

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.12.0
• Labels: performance pull-request-available
• Story Points: 3
• Epic Link: Performance Improvements
• Assignee: Alexey Kudinkin
• Reporter: Alexey Kudinkin
• Created: 16/Apr/22 20:43
• Updated: 25/Jul/22 21:28
• Resolved: 25/Jul/22 21:28

Description

After migrating to Hudi's own Relation impls, we unfortunately broke off some of the optimizations that Spark apply exclusively for `HadoopFsRelation`.

 

While these optimizations could be perfectly implemented for any `FileRelation`, Spark is unfortunately predicating them on usage of HadoopFsRelation, therefore making them non-applicable to any of the Hudi's relations.

Proper longterm solutions would be fixing this in Spark and could be either of:

1. Generalizing such optimizations to any `FileRelation`
2. Making `HadoopFsRelation` extensible (making it non-case class)

 

One example of this is Spark's `SchemaPrunning` optimization rule (HUDI-3891): Spark 3.2.x is able to effectively reduce amount of data read via schema pruning (projecting read data) even for nested structs, however this optimization is predicated on the usage of `HadoopFsRelation`:

Issue Links

• causes: HUDI-3891 Investigate Hudi vs Raw Parquet table discrepancy
• relates to: HUDI-3882 Make sure Hudi Spark relations implementations provide similar file-scanning metrics
• links to: GitHub Pull Request #5428