[HIVE-3952] merge map-job followed by map-reduce job

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: Query Processor
• Hadoop Flags: Reviewed
• Assignee: Vinod Kumar Vavilapalli
• Reporter: Namit Jain
• Created: 28/Jan/13 08:50
• Updated: 03/Feb/18 20:23
• Resolved: 25/Apr/13 07:59

Description

Consider the query like:

select count（*）FROM
( select idOne, idTwo, value FROM
bigTable
JOIN
smallTableOne on (bigTable.idOne = smallTableOne.idOne)
) firstjoin
JOIN
smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo);

where smallTableOne and smallTableTwo are smaller than hive.auto.convert.join.noconditionaltask.size and
hive.auto.convert.join.noconditionaltask is set to true.

The joins are collapsed into mapjoins, and it leads to a map-only job
(for the map-joins) followed by a map-reduce job (for the group by).
Ideally, the map-only job should be merged with the following map-reduce job.

Issue Links

• blocks: HIVE-4078 Delay the serialize-deserialize pair in CommonJoinTaskDispatcher
• depends upon: HIVE-3784 de-emphasize mapjoin hint
• is blocked by: HIVE-4106 SMB joins fail in multi-way joins
• is related to: HIVE-4827 Merge a Map-only task to its child task