[CRUNCH-72] Cartesian#cross loads all data in memory per parallel mapper

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.4.0
• Assignee: Gabriel Reid
• Reporter: Gabriel Reid
• Created: 20/Sep/12 19:00
• Updated: 16/Nov/12 20:06
• Resolved: 20/Sep/12 19:12

Description

Cartesian#cross currently uses the PTable#cogroup method to join two sets of data together; this results in all data from both sides of the join being loaded in memory at one time. This can be a real problem with cartesian joins because of the quantity of data being joined.

Using PTable#join instead of PTable#cogroup will reduce the memory usage by 50%, which can be the difference between a cartesian join working or failing with an OOME.