[HIVE-2566] reduce the number map-reduce jobs for union all

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Assignee: Namit Jain
• Reporter: Namit Jain
• Created: 09/Nov/11 18:16
• Updated: 22/Dec/11 00:19
• Resolved: 15/Nov/11 20:13

Description

A query like:

select s.key, s.value from (
select key, value from src2 where key < 10
union all
select key, value from src3 where key < 10
union all
select key, value from src4 where key < 10
union all
select key, count(1) as value from src5 group by key
)s;

should run the last sub-query
'select key, count(1) as value from src5 group by key'
as a map-reduce job.

And then the union should be a map-only job reading from the first 3 map-only subqueries
and the output of the last map-reduce job.

The current plan is very inefficient.

Issue Links

• relates to: HIVE-2634 revert HIVE-2566