[HIVE-963] no out of memory errors for skewed join

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: Query Processor
• Hadoop Flags: Reviewed
• Assignee: Ning Zhang
• Reporter: Namit Jain
• Created: 02/Dec/09 02:02
• Updated: 02/May/13 02:29
• Resolved: 29/Dec/09 06:41

Description

Currently, in case of skew, hive runs out of memory.
A simpler fix would be to use JDBM to store data and use that.

It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.

Issue Links

• is depended upon by: HIVE-562 join does not work well if there is a very large skew in keys