[HUDI-8037] Partition query for transformed value incorrectly prunes valid partitions

• Type: Bug
• Status: In Progress
• Priority: Critical
• Resolution: Unresolved
• Fix Version/s: 1.1.0, 0.15.1
• Component/s: spark, spark-sql
• Labels: pull-request-available
• Assignee: Jonathan Vexler
• Reporter: Jonathan Vexler
• Created: 01/Aug/24 13:50
• Updated: 13/Oct/24 23:16

Description

With timestamp keygen you can have a partition column with timestamps, but then use the keygen so it will create partitions based on days so that all records that have a timestamp on 7-31-2024 will go to the same parititon even though the values in the partition column differ by hours and minutes etc.

This causes a problem with partition pruning. lets say you query "select * from table where partition < 7-31-2024 at 7am and partition > 7-31-2024 at 6am ". Since the file structure has the partition of just 7-31-2024, that will be interpreted as 7-31-2024 at 12am. So the partition will be pruned from the search space.

Issue Links

• links to: GitHub Pull Request #11717