Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4438

Limit after sort does not work in spark mode

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      when pig script executes "order" before "limit" in spark mode, the results will be wrong.
      cat testlimit.txt
      1 orange
      3 coconut
      5 grape
      6 pear
      2 apple
      4 mango

      testlimit.pig:
      a = load './testlimit.txt' as (x:int, y:chararray);
      b = order a by x;
      c = limit b 1;
      store c into './testlimit.out';

      the result:
      1 orange
      2 apple
      3 coconut
      4 mango
      5 grape
      6 pear

      the correct result should be:
      1 orange

        Attachments

        1. PIG-4438_2.patch
          1 kB
          liyunzhang
        2. PIG-4438_1.patch
          58 kB
          liyunzhang

          Issue Links

            Activity

              People

              • Assignee:
                kellyzly liyunzhang
                Reporter:
                kellyzly liyunzhang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: