[HUDI-7829] storage partition stats index can not effert in data skipping

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.0.0
• Component/s: spark-sql
• Assignee: Sagar Sumit
• Reporter: KnightChess
• Created: 05/Jun/24 08:34
• Updated: 05/Jun/24 18:21

Description

partition stats will not effort, the current implementation does not seem to achieve the effect of partition filtering.

• first

in this picture, I change the ut filter to trigger partition stats index.

partition_stats will not save fileName, so if reuse `CSI` logical, it will throw null point in group by key

and this will cause skip other index

• second

and have a question, I am not sure this pr is use to `partition` purge like physical partition col, which mean use other field min/max to get which physical partitions to list fileSlice. or filter fileName like `CSI`, `RLI`.
thanks

Issue Links

• is caused by: HUDI-7144 Support query for tables written as partitionBy but synced as non-partitioned