[CRUNCH-213] Add sharded join functionality

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Assignee: Gabriel Reid
• Reporter: Gabriel Reid
• Created: 07/Jun/13 20:48
• Updated: 29/Jul/13 15:51
• Resolved: 09/Jun/13 21:42

Description

Performing joins where a large proportion of the values on one or both sides of the join are mapped to a single key can result in poor performance, as one (or a small number) of reducers end up handling most of the joining work, leaving the rest of the cluster idle.

Sharded joining should be added to allow splitting up join keys, thereby distributing values mapped to a single key over multiple reducer partitions.