[HIVE-1158] Introducing a new parameter for Map-side join bucket size

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.5.0, 0.6.0
• Fix Version/s: 0.5.0
• Hadoop Flags: Reviewed

Release Note

HIVE-1158. Introducing a new parameter for Map-side join bucket size. (Ning Zhang via zshao)

• Assignee: Ning Zhang
• Reporter: Ning Zhang
• Created: 11/Feb/10 23:43
• Updated: 17/Dec/11 00:05
• Resolved: 17/Feb/10 16:36

Description

Map-side join cache the small table in memory and join with the split of the large table at the mapper side. If the small table is too large, it uses RowContainer to cache a number of rows indicated by parameter hive.join.cache.size, whose default value is 25000. This parameter is also used for regular reducer-side joins to cache all input tables except the streaming table. This default value is too large for map-side join bucket size, resulting in OOM exceptions sometimes. We should define a different parameter to separate these two cache sizes.