[HIVE-17958] spark_dynamic_partition_pruning.q fails when hive.tez.dynamic.semijoin.reduction is false

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Spark
• Target Version/s: 3.0.0
• Assignee: Sahil Takiar
• Reporter: Sahil Takiar
• Created: 01/Nov/17 19:58
• Updated: 02/Nov/17 03:11

Description

Looks like RedundantDynamicPruningConditionsRemoval causes DPP to be disabled in a few cases (not sure why). When hive.tez.dynamic.semijoin.reduction is true (the default), then this rule is disabled so the normal tests don't hit this issue.

But when I disable hive.tez.dynamic.semijoin.reduction then the following query no longer fully triggers DPP:

EXPLAIN select count(*) from srcpart join srcpart_date on (srcpart.ds = srcpart_date.ds) join srcpart_hour on (srcpart.hr = srcpart_hour.hr)
5777 where srcpart_date.`date` = '2008-04-08' and srcpart_hour.hour = 11 and srcpart.hr = 11

There should be two DPP sinks, but when the config is set to false, there is only one.