Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4856 Optimization for pig on spark
  3. PIG-4797

Optimization for join/group case for spark mode

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:

      Description

      There are a big performance difference in join between spark and mr mode.

      daily = load './NYSE_daily' as (exchange:chararray, symbol:chararray,
                  date:chararray, open:float, high:float, low:float,
                  close:float, volume:int, adj_close:float);
      divs  = load './NYSE_dividends' as (exchange:chararray, symbol:chararray,
                  date:chararray, dividends:float);
      jnd   = join daily by (exchange, symbol), divs by (exchange, symbol);
      store jnd into './join.out';
      

      join.sh

      mode=$1
      start=$(date +%s)
      ./pig -x $mode  $PIG_HOME/bin/join.pig
      end=$(date +%s)
      execution_time=$(( $end - $start ))
      echo "execution_time:"$excution_time
      

      The execution time:

        mr spark
      join 20 sec 79 sec

      You can download the test data NYSE_daily and NYSE_dividends in https://github.com/alanfgates/programmingpig/blob/master/data/

        Attachments

        1. PIG-4797.patch
          26 kB
          liyunzhang
        2. PIG-4797_6.patch
          38 kB
          liyunzhang
        3. PIG-4797_5.patch
          44 kB
          liyunzhang
        4. PIG-4797_3.patch
          50 kB
          liyunzhang
        5. PIG-4797_2.patch
          52 kB
          liyunzhang
        6. Join performance analysis.pdf
          370 kB
          Pallavi Rao

          Issue Links

            Activity

              People

              • Assignee:
                kellyzly liyunzhang
                Reporter:
                pallavi.rao Pallavi Rao
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: