[PIG-1709] Skewed join use fewer reducer for extreme large key

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.8.0
• Component/s: impl
• Hadoop Flags: Reviewed
• Release Note: In skewed join, if one large key requires more reducers than available, we give it all the available reducers.
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 05/Nov/10 21:11
• Updated: 17/Dec/10 22:46
• Resolved: 09/Dec/10 00:17

Description

In skewed join, we use PartitionSkewedKeys to calculate number of reducers needed for a single key. If the result if larger than the number of total reducers, we will round it with reducer#. Eg, if Pig calculates that we need 12 reducers to hold a key in memory, and total reducers for this job is 10, we then allocate 2 reducers to this key; We shall use all 10 reducers in this case.