[HIVE-3276] optimize union sub-queries

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.10.0
• Assignee: Namit Jain
• Reporter: Namit Jain
• Created: 19/Jul/12 04:42
• Updated: 15/Sep/14 03:03
• Resolved: 30/Oct/12 23:40

Description

It might be a good idea to optimize simple union queries containing map-reduce jobs in at least one of the sub-qeuries.

For eg:

a query like:

insert overwrite table T1 partition P1
select * from
(
subq1
union all
subq2
) u;

today creates 3 map-reduce jobs, one for subq1, another for subq2 and
the final one for the union.

It might be a good idea to optimize this. Instead of creating the union
task, it might be simpler to create a move task (or something like a move
task), where the outputs of the two sub-queries will be moved to the final
directory. This can easily extend to more than 2 sub-queries in the union.

This is very useful if there is a select * followed by filesink after the
union. This can be independently useful, and also be used to optimize the
skewed joins –
https://cwiki.apache.org/confluence/display/Hive/Skewed+Join+Optimization.

If there is a select, filter between the union and the filesink, the select
and the filter can be moved before the union, and the follow-up job can
still be removed.

Issue Links

• depends upon: HIVE-3341 Making hive tests run against different MR versions
• depends upon: HIVE-3544 union involving double column with a map join subquery will fail or give wrong results
• is depended upon by: HIVE-3380 As a follow up for HIVE-3276, optimize union for dynamic partition queries
• is related to: HIVE-8054 Disable hive.optimize.union.remove when hive.execution.engine=spark [Spark Branch]
• is related to: HIVE-3451 map-reduce jobs does not work for a partition containing sub-directories
• is related to: HIVE-3643 Hive List Bucketing - set hive.mapred.supports.subdirectories