Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4615

Fix null keys join in SkewedJoin in spark mode

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      Let's use an example to explain the problem:
      testSkewedJoinNullKeys.pig:

      A = LOAD './SkewedJoinInput5.txt' as (id,name);
      B = LOAD './SkewedJoinInput5.txt' as (id,name);
      C = join A by id, B by id using 'skewed';
      store C into './testSkewedJoinNullKeys.out';
      

      cat SkewedJoinInput5.txt

      	apple1
      	apple1
      	apple1
      	apple1
      	apple1
      	apple1
      	apple1
      	apple1
      	apple1
      	apple1
      100	apple2
      	orange1
      	orange1
      	orange1
      	orange1
      	orange1
      	orange1
      	orange1
      	orange1
      	orange1
      	orange1
      
      100
      

      the result of mr:

      100	apple2	100	apple2
      100	apple2	100	
      100		100	apple2
      100		100	
      

      The result of spark:

      cat testSkewedJoinNullKeys.out.spark/part-r-00000 
      100	apple2	100	apple2
      100	apple2	100	
      100		100	apple2
      100		100	
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		apple1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		orange1
      	apple1		
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		apple1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		orange1
      	orange1		
      			apple1
      			apple1
      			apple1
      			apple1
      			apple1
      			apple1
      			apple1
      			apple1
      			apple1
      			apple1
      			orange1
      			orange1
      			orange1
      			orange1
      			orange1
      			orange1
      			orange1
      			orange1
      			orange1
      			orange1
      

        Attachments

        1. PIG-4615.patch
          6 kB
          liyunzhang

          Activity

            People

            • Assignee:
              kellyzly liyunzhang
              Reporter:
              kellyzly liyunzhang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: