[HIVE-5093] Use a combiner for LIMIT with GROUP BY and ORDER BY operators

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.12.0
• Component/s: Query Processor
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 14/Aug/13 21:10
• Updated: 28/Aug/13 16:18
• Resolved: 28/Aug/13 16:18

Description

Operator trees of the following structure can have a memory friendly combiner put in place after the sort-phase

"GBY-LIM" and "OBY-LIM"

This will cut down on I/O when spilling to disk and particularly during the merge phase of the reducer.

There are two possible combiners - LimitNKeysCombiner and LimitNValuesCombiner.

The first one would be ideal for the GROUP-BY case, while the latter would more useful for the ORDER-BY case.

The combiners are still relevant even if there are 1:1 forward operators on the reducer side and for small data items, the MR base layer does not run the combiners at all.

Issue Links

• is related to: HIVE-3562 Some limit can be pushed down to map stage
• is superceded by: HIVE-3562 Some limit can be pushed down to map stage