[HIVE-16923] Hive-on-Spark DPP Improvements

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Spark
• Target Version/s: 3.0.0
• Assignee: Sahil Takiar
• Reporter: Sahil Takiar
• Created: 20/Jun/17 22:14
• Updated: 31/Aug/17 15:01

Description

Improvements to Hive-on-Spark DPP so that it is production ready.

Hive-on-Spark DPP was implemented in HIVE-9152. However, it is disabled by default. The goal of this JIRA is to improve the DPP implementation so that it can be enabled by default.

Issue Links

• is related to: HIVE-17153 Flaky test: TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning]
• is related to: HIVE-17248 DPP isn't triggered if static pruning is done for one of the partition columns
• is related to: HIVE-17122 spark_vectorized_dynamic_partition_pruning.q is continuously failing
• is related to: HIVE-17412 Add "-- SORT_QUERY_RESULTS" for spark_vectorized_dynamic_partition_pruning.q
• is related to: HIVE-17244 DPP isn't triggered against partition columns that are added to each other
• relates to: HIVE-17090 spark.only.query.files are not being run by ptest

Sub-Tasks

• Add config to enable HoS DPP only for map-joins: Closed
• Remove unnecessary HoS DPP trees during map-join conversion: Closed
• Spark Partition Pruning Sink Operator can't target multiple Works: Closed
• Additional qtests for HoS DPP: Closed
• NPE in SparkPartitionPruningSinkOperator#closeOp for query with partitioned join in subquery: Open
• HoS DPP pruning sink ops can target parallel work objects: Closed
• DPP isn't trigger for partitioned to partitioned join within a subquery: Open
• HoS DPP: UDFs on the partition column side does not evaluate correctly: Closed
• Support DPP with map joins where the source and target belong in the same stage: Patch Available
• Support Costing/Heuristics to enable or disable DPP: Open
• HoS DPP ConstantPropagate should use ConstantPropagateOption.SHORTCUT: Closed
• HoS doesn't trigger mapjoins against subquery with union all: Open
• HoS DPP + Vectorization generates invalid explain plan due to CombineEquivalentWorkResolver: Closed
• DynamicPartitionPruningOptimization doesn't log what filter triggered DPP: Open
• SparkDynamicPartitionPruner loads all partition metadata into memory: Open
• spark_dynamic_partition_pruning.q fails when hive.tez.dynamic.semijoin.reduction is false: Open
• SparkPartitionPruningSinkOperator buffers all writes in memory: Open
• SparkPartitionPruner shouldn't be triggered by Spark tasks: Resolved
• DPP call to remove PartitionDescs from aliasToPartnInfo doesn't do anything: Open
• Set hive.spark.dynamic.partition.pruning.map.join.only to true by default: Open