[HUDI-3338] Use custom relation instead of HadoopFsRelation

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: spark, spark-sql
• Labels: pull-request-available
• Assignee: Yann Byron
• Reporter: Yann Byron
• Created: 28/Jan/22 05:18
• Updated: 26/Mar/22 17:08
• Resolved: 15/Feb/22 14:58

Description

For HUDI-3204, COW table and MOR table in read_optimized query mode should return the 'yyyy-MM-dd' format of origin `data_date`, not yyyy/MM/dd''.

And the reason for that is because Hudi use HadoopFsRelation for the snapshot query mode of cow and the read_optimized query mode of mor.

Spark HadoopFsRelation will append the partition value of the real partition path. However, different from the normal table, Hudi will persist the partition value in the parquet file. So we just need read the partition value from the parquet file, not leave it to spark.

So we should not use `HadoopFsRelation` any more, and implement Hudi own `Relation` to deal with it.

Issue Links

• relates to: HUDI-3204 Allow original partition column value to be retrieved when using TimestampBasedKeyGen
• links to: GitHub Pull Request #4709