[PIG-5271] StackOverflowError when compiling in Tez mode (with union and replicated join)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.18.0
• Hadoop Flags: Reviewed
• Assignee: Koji Noguchi
• Reporter: Koji Noguchi
• Created: 13/Jul/17 22:14
• Updated: 22/Sep/17 13:15
• Resolved: 20/Sep/17 16:45

Description

Sample script

a4 = LOAD 'studentnulltab10k' as (name, age:int, gpa:float);
a4_1 = filter a4 by gpa is null or gpa >= 3.9;
a4_2 = filter a4 by gpa < 1;
b4 = union a4_1, a4_2;
b4_1 = filter b4 by age < 30;
b4_2 = foreach b4 generate name, age, FLOOR(gpa) as gpa;

c4 = load 'voternulltab10k' as (name, age, registration, contributions);
d4 = join b4_2 by name, c4 by name using 'replicated';
e4 = foreach d4 generate b4_2::name as name, b4_2::age as age, gpa, registration, contributions;
f4 = order e4 by name, age DESC;
store f4 into 'tmp_table_4' ;

a5_1 = filter a4 by gpa is null or gpa <= 3.9;
a5_2 = filter a4 by gpa < 2;
b5 = union a5_1, a5_2;
d5 = join c4 by name, b5 by name using 'replicated';
store d5 into 'tmp_table_5' ;

This script fails to compile with StackOverflowError.

    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:323)
Pig Stack Trace
---------------
ERROR 2998: Unhandled internal error. null

java.lang.StackOverflowError
    at java.lang.reflect.Constructor.newInstance(Constructor.java:415)
    at java.lang.Class.newInstance(Class.java:442)
    at org.apache.pig.impl.util.Utils.mergeCollection(Utils.java:490)
    at org.apache.pig.impl.plan.DependencyOrderWalker.doAllPredecessors(DependencyOrderWalker.java:101)
    at org.apache.pig.impl.plan.DependencyOrderWalker.doAllPredecessors(DependencyOrderWalker.java:105)
    at org.apache.pig.impl.plan.DependencyOrderWalker.doAllPredecessors(DependencyOrderWalker.java:105)
    at org.apache.pig.impl.plan.DependencyOrderWalker.doAllPredecessors(DependencyOrderWalker.java:105)
    at org.apache.pig.impl.plan.DependencyOrderWalker.doAllPredecessors(DependencyOrderWalker.java:105)
    at org.apache.pig.impl.plan.DependencyOrderWalker.doAllPredecessors(DependencyOrderWalker.java:105)
...