Pig
  1. Pig
  2. PIG-1979

New logical plan failing with ERROR 2229: Couldn't find matching uid -1

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.9.0
    • Fix Version/s: 0.8.1
    • Component/s: impl
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The below is my script

      register myudf.jar;
      c01 = LOAD 'input'  USING org.test.MyTableLoader('');
      c02 = FILTER c01  BY result == 'OK'  AND formatted IS NOT NULL  AND formatted != '' ;
      c03 = FOREACH c02 GENERATE url, formatted, FLATTEN(usage);
      c04 = FOREACH c03 GENERATE usage::domain AS domain, url, formatted;
      doc_001 = FOREACH c04 GENERATE domain,url, FLATTEN(MyExtractor(formatted)) AS category;
      doc_004_1 = GROUP doc_001 BY (domain,url);
      doc_005 = FOREACH doc_004_1 GENERATE group.domain as domain, group.url as url, doc_001.category as category;
      STORE doc_005 INTO 'out_final' USING PigStorage();
      
      review1 = FOREACH c04 GENERATE domain,url, MyExtractor(formatted) AS rev;
      review2 = FILTER review1 BY SIZE(rev)>0;
      joinresult = JOIN review2 by (domain,url), doc_005 by (domain,url);
      finalresult = FOREACH joinresult GENERATE  doc_005::category;
      STORE finalresult INTO 'out_final' using PigStorage();
      

      The script is failing in building the plan, while applying for logical optimization rule for AddForEach.

      ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2229: Couldn't find matching uid -1 for project (Name: Project Type: bytearray Uid: 106 Input: 0 Column: 5)

      The problem is happening when I try to include doc_005::category in the projection for relation finalresult. This is field is orginated from the udf org.vivek.udfs.MyExtractor (source given below).

      
      import java.io.IOException;
      import org.apache.pig.EvalFunc;
      import org.apache.pig.data.*;
      import org.apache.pig.impl.logicalLayer.FrontendException;
      import org.apache.pig.impl.logicalLayer.schema.Schema;
      import org.apache.pig.impl.logicalLayer.schema.Schema.FieldSchema;
      
      public class MyExtractor extends EvalFunc<DataBag>
      {
        @Override
      	public Schema outputSchema(Schema arg0) {
      	  try {
      			return Schema.generateNestedSchema(DataType.BAG, DataType.CHARARRAY);
      		} catch (FrontendException e) {
      			System.err.println("Error while generating schema. "+e);
      			return new Schema(new FieldSchema(null, DataType.BAG));
      		}
      	}
      
        @Override
        public DataBag exec(Tuple inputTuple)
          throws IOException
        {
          try {
            Tuple tp2 = TupleFactory.getInstance().newTuple(1);
            tp2.set(0, (inputTuple.get(0).toString()+inputTuple.hashCode()));
            DataBag retBag = BagFactory.getInstance().newDefaultBag();
            retBag.add(tp2);
            return retBag;
          }
          catch (Exception e) {
            throw new IOException(" Caught exception", e);
          }
        }
      }
      
      

      The script goes through fine if I disable AddForEach rule by -t AddForEach

      1. PIG-1979-1-trunk.patch
        3 kB
        Daniel Dai
      2. PIG-1979-1.patch
        2 kB
        Daniel Dai

        Activity

        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/587/
        -----------------------------------------------------------

        Review request for pig and thejas.

        Summary
        -------

        See PIG-1979

        This addresses bug PIG-1979.
        https://issues.apache.org/jira/browse/PIG-1979

        Diffs


        http://svn.apache.org/repos/asf/pig/branches/branch-0.8/src/org/apache/pig/newplan/logical/expression/DereferenceExpression.java 1091122
        http://svn.apache.org/repos/asf/pig/branches/branch-0.8/test/org/apache/pig/test/TestEvalPipeline2.java 1091122

        Diff: https://reviews.apache.org/r/587/diff

        Testing
        -------

        Test-patch:
        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Unit-test:
        all pass

        Thanks,

        Daniel

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/587/ ----------------------------------------------------------- Review request for pig and thejas. Summary ------- See PIG-1979 This addresses bug PIG-1979 . https://issues.apache.org/jira/browse/PIG-1979 Diffs http://svn.apache.org/repos/asf/pig/branches/branch-0.8/src/org/apache/pig/newplan/logical/expression/DereferenceExpression.java 1091122 http://svn.apache.org/repos/asf/pig/branches/branch-0.8/test/org/apache/pig/test/TestEvalPipeline2.java 1091122 Diff: https://reviews.apache.org/r/587/diff Testing ------- Test-patch: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Unit-test: all pass Thanks, Daniel
        Hide
        Daniel Dai added a comment -

        PIG-1979-1-trunk.patch is for trunk.

        Show
        Daniel Dai added a comment - PIG-1979 -1-trunk.patch is for trunk.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/587/#review461
        -----------------------------------------------------------

        Ship it!

        +1

        • thejas

        On 2011-04-13 00:11:37, Daniel Dai wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/587/

        -----------------------------------------------------------

        (Updated 2011-04-13 00:11:37)

        Review request for pig and thejas.

        Summary

        -------

        See PIG-1979

        This addresses bug PIG-1979.

        https://issues.apache.org/jira/browse/PIG-1979

        Diffs

        -----

        http://svn.apache.org/repos/asf/pig/branches/branch-0.8/src/org/apache/pig/newplan/logical/expression/DereferenceExpression.java 1091122

        http://svn.apache.org/repos/asf/pig/branches/branch-0.8/test/org/apache/pig/test/TestEvalPipeline2.java 1091122

        Diff: https://reviews.apache.org/r/587/diff

        Testing

        -------

        Test-patch:

        [exec] +1 overall.

        [exec]

        [exec] +1 @author. The patch does not contain any @author tags.

        [exec]

        [exec] +1 tests included. The patch appears to include 3 new or modified tests.

        [exec]

        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.

        [exec]

        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        [exec]

        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.

        [exec]

        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Unit-test:

        all pass

        Thanks,

        Daniel

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/587/#review461 ----------------------------------------------------------- Ship it! +1 thejas On 2011-04-13 00:11:37, Daniel Dai wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/587/ ----------------------------------------------------------- (Updated 2011-04-13 00:11:37) Review request for pig and thejas. Summary ------- See PIG-1979 This addresses bug PIG-1979 . https://issues.apache.org/jira/browse/PIG-1979 Diffs ----- http://svn.apache.org/repos/asf/pig/branches/branch-0.8/src/org/apache/pig/newplan/logical/expression/DereferenceExpression.java 1091122 http://svn.apache.org/repos/asf/pig/branches/branch-0.8/test/org/apache/pig/test/TestEvalPipeline2.java 1091122 Diff: https://reviews.apache.org/r/587/diff Testing ------- Test-patch: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Unit-test: all pass Thanks, Daniel
        Hide
        Daniel Dai added a comment -

        Patch committed to both trunk and 0.8 branch.

        Show
        Daniel Dai added a comment - Patch committed to both trunk and 0.8 branch.

          People

          • Assignee:
            Daniel Dai
            Reporter:
            Vivek Padmanabhan
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development