[HUDI-7144] Support query for tables written as partitionBy but synced as non-partitioned

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.0.0-beta2, 1.0.0
• Labels: hudi-1.0.0-beta2 pull-request-available
• Story Points: 8
• Epic Link: Indexing Functions
• Assignee: Sagar Sumit
• Reporter: Sagar Sumit

Reviewers

Ethan Guo (this is the old account; please use "yihua"), Vinoth Chandar

• Created: 27/Nov/23 16:57
• Updated: 10/Jun/24 17:56
• Resolved: 01/May/24 23:52

Description

In HUDI-7023, we added support to sync any table as non-partitioned table and yet be able to query via Spark with the same performance benefits of partitioned table.
This ticket extends the functionality end-to-end. If a user executes  `spark.write.format("hudi").options(options).partitionBy(partCol).save(basePath)`, then do logical partitioning and sync as non-partitioned table to the catalog. Yet be able to query efficiently,.

Issue Links

• causes: HUDI-7829 storage partition stats index can not effert in data skipping
• links to: GitHub Pull Request #10274
• links to: GitHub Pull Request #10352