[HIVE-8054] Disable hive.optimize.union.remove when hive.execution.engine=spark [Spark Branch]

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Spark
• Labels: Spark-M1 TODOC-SPARK TODOC15
• Assignee: Na Yang
• Reporter: Xuefu Zhang
• Created: 11/Sep/14 18:22
• Updated: 29/May/15 07:52
• Resolved: 16/Sep/14 02:03

Description

Option hive.optimize.union.remove introduced in HIVE-3276 removes union operators from the operator graph in certain cases as an optimization reduce the number of MR jobs. While making sense in MR, this optimization is actually harmful to an execution engine such as Spark, which natives supports union without requiring additional jobs. This is because removing union operator creates disjointed operator graphs, each graph generating a job, and thus this optimization requires more jobs to run the query. Not to mention the additional complexity handling linked FS descriptors.

I propose that we disable such optimization when the execution engine is Spark.

Issue Links

• is part of: HIVE-7292 Hive on Spark
• is related to: HIVE-8073 Go thru all operator plan optimizations and disable those that are not suitable for Spark [Spark Branch]
• relates to: HIVE-8055 Code cleanup after HIVE-8054 [Spark Branch]
• relates to: HIVE-3276 optimize union sub-queries