Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-5271

StackOverflowError when compiling in Tez mode (with union and replicated join)

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Sample script

      a4 = LOAD 'studentnulltab10k' as (name, age:int, gpa:float);
      a4_1 = filter a4 by gpa is null or gpa >= 3.9;
      a4_2 = filter a4 by gpa < 1;
      b4 = union a4_1, a4_2;
      b4_1 = filter b4 by age < 30;
      b4_2 = foreach b4 generate name, age, FLOOR(gpa) as gpa;
      
      c4 = load 'voternulltab10k' as (name, age, registration, contributions);
      d4 = join b4_2 by name, c4 by name using 'replicated';
      e4 = foreach d4 generate b4_2::name as name, b4_2::age as age, gpa, registration, contributions;
      f4 = order e4 by name, age DESC;
      store f4 into 'tmp_table_4' ;
      
      a5_1 = filter a4 by gpa is null or gpa <= 3.9;
      a5_2 = filter a4 by gpa < 2;
      b5 = union a5_1, a5_2;
      d5 = join c4 by name, b5 by name using 'replicated';
      store d5 into 'tmp_table_5' ;
      

      This script fails to compile with StackOverflowError.

          at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:323)
      Pig Stack Trace
      ---------------
      ERROR 2998: Unhandled internal error. null
      
      java.lang.StackOverflowError
          at java.lang.reflect.Constructor.newInstance(Constructor.java:415)
          at java.lang.Class.newInstance(Class.java:442)
          at org.apache.pig.impl.util.Utils.mergeCollection(Utils.java:490)
          at org.apache.pig.impl.plan.DependencyOrderWalker.doAllPredecessors(DependencyOrderWalker.java:101)
          at org.apache.pig.impl.plan.DependencyOrderWalker.doAllPredecessors(DependencyOrderWalker.java:105)
          at org.apache.pig.impl.plan.DependencyOrderWalker.doAllPredecessors(DependencyOrderWalker.java:105)
          at org.apache.pig.impl.plan.DependencyOrderWalker.doAllPredecessors(DependencyOrderWalker.java:105)
          at org.apache.pig.impl.plan.DependencyOrderWalker.doAllPredecessors(DependencyOrderWalker.java:105)
          at org.apache.pig.impl.plan.DependencyOrderWalker.doAllPredecessors(DependencyOrderWalker.java:105)
      ...
      

        Attachments

        1. pig-5271-v01.patch
          10 kB
          Koji Noguchi
        2. pig-5271-v02.patch
          10 kB
          Koji Noguchi
        3. pig-5271-v03.patch
          10 kB
          Koji Noguchi

          Activity

            People

            • Assignee:
              knoguchi Koji Noguchi
              Reporter:
              knoguchi Koji Noguchi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: