[HIVE-1035] limit can be optimized if the limit is happening on the reducer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: Query Processor
• Assignee: Unassigned
• Reporter: Namit Jain
• Created: 07/Jan/10 21:57
• Updated: 15/Feb/13 06:25
• Resolved: 15/Feb/13 06:25

Description

A query like:

select ... from A join B.. limit 10;

where the limit is performed on the reducer can be further optimized.

Currently, all the operators on the reduce side will be done, but the ExecReducer will un-necessarily deserialize all the rows.
The following optimizations can be done:

1. Do nothing in reduce() in ExecReducer.
2. Modify map-reduce framework so that it does not even invoke the reduce() method in ExecReducer.

2. may require some work from hadoop - but we should minimally do 1.

Issue Links

• duplicates: HIVE-3550 Early skipping for limit operator at reduce stage