Pig
  1. Pig
  2. PIG-2286

Using COR function in Piggybank results in ERROR 2018: Internal error. Unable to introduce the combiner for optimization

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.9.1, 0.10.0
    • Component/s: impl, piggybank
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Usage of the COR function in a Pig script, results in an error. The "studenttab5" contains student, age and gpa separated by "tab".

      register /home/viraj/pig-svn/trunk/contrib/piggybank/java/piggybank.jar;
      A = LOAD '/user/viraj/studenttab5' AS (name, age:double,gpa:double);
      B = group A all;
      C = foreach B generate group, COR(A.a, A.b);
      dump C;
      

      2011-09-14 17:03:22,001 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://localhost:9000
      2011-09-14 17:03:22,088 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: localhost:9001
      2011-09-14 17:03:22,960 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY
      2011-09-14 17:03:23,168 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
      2011-09-14 17:03:23,179 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
      2011-09-14 17:03:23,186 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2018: Internal error. Unable to introduce the combiner for optimization.

      Viraj

      1. PIG-2286-2.patch
        6 kB
        Daniel Dai
      2. PIG-2286-1.patch
        5 kB
        Daniel Dai

        Activity

        Viraj Bhat created issue -
        Viraj Bhat made changes -
        Field Original Value New Value
        Description Usage of the COR function in a Pig script, results in an error. The "studenttab5" contains student, age and gpa separated by "tab".
        {code}
        register /home/viraj/pig-svn/trunk/contrib/piggybank/java/piggybank.jar;
        A = LOAD '/user/viraj/studenttab5' AS (name, age:double,gpa:double);
        B = group A all;
        C = foreach B generate group, COR(A.a, A.b);
        dump C;
        {code}

        {verbatim}
        2011-09-14 17:03:22,001 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://localhost:9000
        2011-09-14 17:03:22,088 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: localhost:9001
        2011-09-14 17:03:22,960 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY
        2011-09-14 17:03:23,168 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
        2011-09-14 17:03:23,179 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
        2011-09-14 17:03:23,186 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2018: Internal error. Unable to introduce the combiner for optimization.
        {verbatim}

        Viraj
        Usage of the COR function in a Pig script, results in an error. The "studenttab5" contains student, age and gpa separated by "tab".
        {code}
        register /home/viraj/pig-svn/trunk/contrib/piggybank/java/piggybank.jar;
        A = LOAD '/user/viraj/studenttab5' AS (name, age:double,gpa:double);
        B = group A all;
        C = foreach B generate group, COR(A.a, A.b);
        dump C;
        {code}

        {quote}
        2011-09-14 17:03:22,001 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://localhost:9000
        2011-09-14 17:03:22,088 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: localhost:9001
        2011-09-14 17:03:22,960 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY
        2011-09-14 17:03:23,168 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
        2011-09-14 17:03:23,179 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
        2011-09-14 17:03:23,186 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2018: Internal error. Unable to introduce the combiner for optimization.
        {quote}

        Viraj
        Daniel Dai made changes -
        Attachment PIG-2286-1.patch [ 12494732 ]
        Daniel Dai made changes -
        Assignee Daniel Dai [ daijy ]
        Daniel Dai made changes -
        Attachment PIG-2286-2.patch [ 12495443 ]
        Daniel Dai made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.9.1 [ 12317343 ]
        Fix Version/s 0.10 [ 12316246 ]
        Resolution Fixed [ 1 ]
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Daniel Dai
            Reporter:
            Viraj Bhat
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development