[CRUNCH-655] DefaultJoinStrategy full outer join failing for spark pipeline

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Spark
• Environment: Mac OSX, crunch 0.13.0 and 0.15.0 (with reproduction code), Ubuntu 14.04 (repro code not tried, but similar issue in production with 0.13.0)
• Assignee: Unassigned
• Reporter: Mikael Goldmann
• Created: 17/Aug/17 13:59
• Updated: 17/Aug/17 15:43

Description

When the left and right table in the the join have entries with the same key, they do not alway end up together. Cannot reproduce when running the join with a single reducer, and happens more commonly if there are many reducers and very few copies of each key to the left and right.

My guess is that it sometimes happens that the left value for key k ends up on a different reducer from the right value with key k.

With my production issue, it went away if I either used a single reducer or used cogroup instead.

I've attached a class to reproduce the issue.