Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2498

e2e tests failing in some cases due to incorrect unix sort args

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.9.2
    • Fix Version/s: 0.11
    • Component/s: None
    • Labels:
      None

      Description

      Some e2e tests are failing for me against 23 due to what I think are incorrect arguments to unix sort. For example in Order_6:

      			'num' => 6,
      			'pig' => q\a = load ':INPATH:/singlefile/studenttab10k';
      c = order a by $0;
      store c into ':OUTPATH:';\,
      			'sortArgs' => ['-t', '	', '+0', '-1'],
      

      The pig job is sorting by the first column, however unix sort is being told to sort by the first and second columns.

      From the gnu sort manual (specifically pos2 is inclusive): http://www.gnu.org/software/coreutils/manual/html_node/sort-invocation.html

      '-k pos1[,pos2]'
      '--key=pos1[,pos2]'
      Specify a sort field that consists of the part of the line between pos1 and pos2 (or the end of the line, if pos2 is omitted), inclusive.
      
      ...
      
      On older systems, sort supports an obsolete origin-zero syntax '+pos1 [-pos2]' for specifying sort keys. The obsolete sequence 'sort +a.x -b.y' is equivalent to 'sort -k a+1.x+1,b' if y is '0' or absent, otherwise it is equivalent to 'sort -k a+1.x+1,b+1.y'.
      

      I verified this by running the sort manually with +0 -1 and +0 -0, in the first case it fails, in the second case it passes.

        Attachments

        1. PIG-2498.patch
          10 kB
          Patrick Hunt

          Issue Links

            Activity

              People

              • Assignee:
                phunt Patrick Hunt
                Reporter:
                phunt Patrick Hunt
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: