[HUDI-3891] Investigate Hudi vs Raw Parquet table discrepancy

• Type: Task
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Labels: pull-request-available
• Story Points: 2
• Epic Link: Hudi Spark Datasource
• Assignee: Alexey Kudinkin
• Reporter: Alexey Kudinkin
• Created: 16/Apr/22 01:46
• Updated: 09/Jun/23 04:32
• Resolved: 16/Apr/22 20:30

Description

While benchmarking querying raw Parquet tables against Hudi tables, i've run the test against the same (Hudi) table:

1. In one query path i'm reading it as just a raw Parquet table
2. In another, i'm reading it as Hudi RO (read_optimized) table

Surprisingly enough, those 2 diverge in the # of files being read:

 
Raw Parquet

 
Hudi

Issue Links

• incorporates: HUDI-3902 Fallback to HadoopFsRelation for non-sophisticated COW use-cases
• is caused by: HUDI-3895 Make sure Hudi relations do proper file-split packing (on par w/ Spark)
• is caused by: HUDI-3896 Support SchemaPruning optimization for Hudi's own relations
• links to: GitHub Pull Request #5337