[HUDI-7952] Incorrect partition pruning when TimestampBasedKeyGenerator is used in partition column

• Type: Sub-task
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Fix Version/s: 1.0.1
• Assignee: Danny Chen
• Reporter: Geser Dugarov
• Created: 04/Jul/24 10:20
• Updated: 13/Dec/24 01:47

Description

Fix of ClassCastException in https://issues.apache.org/jira/browse/HUDI-7709 with nulls as partition columns values could lead to an empty query results.

HoodieFileIndex.listFiles() would return Seq of PartitionDirectory with null values.

 

But there is another problem with range filters on partition column.

For instance, we have UNIX_TIMESTAMP in column ts.

And the table is also partitioned by ts with

hoodie.keygen.timebased.output.dateformat = "yyyy-MM-dd HH"

For execution of query like:

SELECT ... WHERE ts BETWEEN 1078016000 and 1718953003 ...

it's not possible to filter rows properly.