[PIG-2498] e2e tests failing in some cases due to incorrect unix sort args

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.9.2
• Fix Version/s: 0.11
• Assignee: Patrick D. Hunt
• Reporter: Patrick D. Hunt
• Created: 31/Jan/12 18:14
• Updated: 12/Nov/12 21:57
• Resolved: 10/Nov/12 21:59

Description

Some e2e tests are failing for me against 23 due to what I think are incorrect arguments to unix sort. For example in Order_6:

			'num' => 6,
			'pig' => q\a = load ':INPATH:/singlefile/studenttab10k';
c = order a by $0;
store c into ':OUTPATH:';\,
			'sortArgs' => ['-t', '	', '+0', '-1'],

The pig job is sorting by the first column, however unix sort is being told to sort by the first and second columns.

From the gnu sort manual (specifically pos2 is inclusive): http://www.gnu.org/software/coreutils/manual/html_node/sort-invocation.html

'-k pos1[,pos2]'
'--key=pos1[,pos2]'
Specify a sort field that consists of the part of the line between pos1 and pos2 (or the end of the line, if pos2 is omitted), inclusive.

...

On older systems, sort supports an obsolete origin-zero syntax '+pos1 [-pos2]' for specifying sort keys. The obsolete sequence 'sort +a.x -b.y' is equivalent to 'sort -k a+1.x+1,b' if y is '0' or absent, otherwise it is equivalent to 'sort -k a+1.x+1,b+1.y'.

I verified this by running the sort manually with +0 -1 and +0 -0, in the first case it fails, in the second case it passes.

Issue Links

• duplicates: PIG-2782 Specifying sorting field(s) at nightly.conf
• duplicates: PIG-3045 Specifying sorting field(s) at nightly.conf - fix sortArgs