[HIVE-17405] HoS DPP ConstantPropagate should use ConstantPropagateOption.SHORTCUT

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: Spark
• Target Version/s: 3.0.0
• Assignee: Sahil Takiar
• Reporter: Sahil Takiar
• Created: 29/Aug/17 05:48
• Updated: 22/May/18 23:14
• Resolved: 02/Sep/17 16:45

Description

In SparkCompiler#runDynamicPartitionPruning we should change new ConstantPropagate().transform(parseContext) to new ConstantPropagate(ConstantPropagateOption.SHORTCUT).transform(parseContext)

Hive-on-Tez does the same thing.

Running the full constant propagation isn't really necessary, we just want to eliminate any and true predicates that were introduced by SyntheticJoinPredicate and DynamicPartitionPruningOptimization. The SyntheticJoinPredicate will introduce dummy filter predicates into the operator tree, and DynamicPartitionPruningOptimization will replace them. The predicates introduced via SyntheticJoinPredicate are necessary to help DynamicPartitionPruningOptimization determine if DPP can be used or not.

Issue Links

• links to: RB