Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Let's use an example to explain the problem:
testSkewedJoinNullKeys.pig:
A = LOAD './SkewedJoinInput5.txt' as (id,name); B = LOAD './SkewedJoinInput5.txt' as (id,name); C = join A by id, B by id using 'skewed'; store C into './testSkewedJoinNullKeys.out';
cat SkewedJoinInput5.txt
apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 100 apple2 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 100
the result of mr:
100 apple2 100 apple2 100 apple2 100 100 100 apple2 100 100
The result of spark:
cat testSkewedJoinNullKeys.out.spark/part-r-00000 100 apple2 100 apple2 100 apple2 100 100 100 apple2 100 100 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 apple1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 apple1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1 orange1