[HIVE-988] mapjoin should throw an error if the input is too large

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: Query Processor
• Hadoop Flags: Reviewed
• Assignee: Ning Zhang
• Reporter: Namit Jain
• Created: 14/Dec/09 22:21
• Updated: 17/Dec/11 00:06
• Resolved: 06/Jan/10 23:37

Description

If the input to the map join is larger than a specific threshold, it may lead to a very slow execution of the join.
It is better to throw an error, and let the user redo his query as a non map-join query.

However, the current map-reduce framework will retry the mapper 4 times before actually killing the job.
Based on a offline discussion with Dhruba, Ning and myself, we came up with the following algorithm:

Keep a threshold in the mapper for the number of rows to be processed for map-join. If the number of rows
exceeds that threshold, set a counter and kill that mapper.

The client (ExecDriver) monitors that job continuously - if this counter is set, it kills the job and also
shows an appropriate error message to the user, so that he can retry the query without the map join.