[FLINK-27217] Support partition filter push down when there exists default_parition

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Component/s: Connectors / Hive
• Labels: pull-request-available
• Assignee: luoyuxia
• Reporter: luoyuxia
• Created: 13/Apr/22 07:09
• Updated: 06/Aug/22 01:59
• Resolved: 06/Aug/22 01:59

Description

Using Hive dialect, when the table's partition column is nullable, and when insert a row whose value of partition column is null,  the  row will be assigned to the partition with value 'HiveConf.ConfVars.DEFAULTPARTITIONNAME',  default is  "_HIVE_DEFAULT_PARTITION_".

Like this :

 

create table ptestfilter (a string) partitioned by (c int);
INSERT OVERWRITE TABLE ptestfilter PARTITION (c) select 'Col1', null;
INSERT OVERWRITE TABLE ptestfilter PARTITION (c) select 'Col2', 5;
select * from ptestfilter where c between 2 and 6 ;

It'll try to do partition filter push down, and  cast the '_HIVE_DEFAULT_PARTITION_' to int type, then it will fail with the following exception:

java.lang.NumberFormatException: For input string: "__HIVE_DEFAULT_PARTITION__"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:580)
    at java.lang.Integer.parseInt(Integer.java:615)
    at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:273)
    at scala.collection.immutable.StringOps.toInt(StringOps.scala:29)
    at org.apache.flink.table.planner.plan.utils.PartitionPruner$.org$apache$flink$table$planner$plan$utils$PartitionPruner$$convertPartitionFieldValue(PartitionPruner.scala:173) 

 

Issue Links

• links to: GitHub Pull Request #19450