[PIG-5130] Union_3 e2e test failing on Spark

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: Unassigned
• Reporter: Nándor Kollár
• Created: 14/Feb/17 11:23
• Updated: 19/Apr/17 15:07
• Resolved: 19/Apr/17 15:07

Description

Union_3 e2e test is failing on spark branch. If I execute on spark branch in MR mode (test-e2e instead of test-e2e-spark), the test is passing.

It fails while comparing the output of the second foreach (the output of the first foreach is fine):
RESULTS DIFFER: vimdiff /var/lib/hadoop-hdfs/pig-spark/pig/test/e2e/pig/testdist/./out/pigtest/hdfs/hdfs-1487068760-nightly.conf/Union_3.out/2/out_sorted ./out/pigtest/hdfs/hdfs-1487068760-nightly.conf/Union_3_benchmark.out/2/out_sorted

The first 3 lines:

diff <(head -n 3 ./Union_3_benchmark.out/2/out_sorted) <(head -n 3 ./Union_3.out/2/out_sorted)
1,3c1,3
< {(alice allen,19,1.93)}
< {(alice brown,18,2.35)}
< {(alice carson,19,1.88)}
---
> {(alice allen,77,3.51)}
> {(alice brown,77,0.74)}
> {(alice carson,74,1.73)}

Old Pig is path is ../../../.., so the benchmarks were created with the same Pig in MR mode.

Issue Links

• is duplicated by: PIG-5171 SecondarySort_7 is failing with spark exec type