Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: None
    • Component/s: impl
    • Labels:
      None

      Description

      working with version
      https://svn.apache.org/repos/asf/pig/trunk@1146777
      fetched from git
      git://git.apache.org/pig.git a7e1228a0fdfe76c3cff0e749e252dba8d387052

      using file /tmp/data.tsv
      id1 123
      id1 234
      id2 345
      id2 456

      this is the most cutdown/simplest script i can make that illustrates (no pun intended) the problem
      grunt> data = load '/tmp/data.tsv' as (id:chararray, value:long);
      grunt> cogrouped = cogroup data by id;
      grunt> exists = foreach cogrouped generate (IsEmpty(data.value) ? 0 : 1) as exists;

      grunt> dump exists
      is ok

      but
      grunt> illustrate exists
      throws
      java.lang.NullPointerException
      at org.apache.pig.pen.IllustratorAttacher.visitBinCond(IllustratorAttacher.java:360)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POBinCond.visit(POBinCond.java:145)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POBinCond.visit(POBinCond.java:36)
      at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
      at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71)
      at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71)
      at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71)
      at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:52)
      at org.apache.pig.pen.IllustratorAttacher.innerPlanAttach(IllustratorAttacher.java:417)
      at org.apache.pig.pen.IllustratorAttacher.visitPOForEach(IllustratorAttacher.java:229)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.visit(POForEach.java:117)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.visit(POForEach.java:47)
      at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
      at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71)
      at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71)
      at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71)
      at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71)
      at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71)
      at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:52)
      at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
      at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:246)
      at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238)
      at org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103)
      at org.apache.pig.pen.LineageTrimmingVisitor.<init>(LineageTrimmingVisitor.java:98)
      at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166)
      at org.apache.pig.PigServer.getExamples(PigServer.java:1201)
      at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
      at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:67)
      at org.apache.pig.Main.run(Main.java:487)
      at org.apache.pig.Main.main(Main.java:108)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

      with pig.log containing
      java.io.IOException: Exception : null
      at org.apache.pig.PigServer.getExamples(PigServer.java:1207)
      at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
      at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:67)
      at org.apache.pig.Main.run(Main.java:487)
      at org.apache.pig.Main.main(Main.java:108)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

      something to do with the IsEmpty since...
      grunt> data = load '/tmp/data.tsv' as (id:chararray, value:long);
      grunt> cogrouped = cogroup data by id;
      grunt> exists = foreach cogrouped generate data.value as exists;
      illustrates ok

      1. PIG-2170.1.patch
        10 kB
        Thejas M Nair
      2. PIG-2170.2.patch
        10 kB
        Thejas M Nair

        Activity

        Hide
        Thejas M Nair added a comment -

        PIG-2170.2.patch fixes a findbugs warning in PIG-2170.1.patch.
        Changes -

        • BinCond is not a boolean expression (ie it does not return boolean value), so equivalence classes are not created for it.
        • Boolean expressions can appear in foreach (within bincond), and every where else after boolean type is introduced. But adding equivalence classes for each boolean expression in all these places will generate too many examples, affecting the conciseness. So I have chosen not to add equivalence classes in those cases.
        • When type conversion errors happened, pig tries to increment counters for it, and was throwing NPE because the local mode simulation did not supply the object that returns counters. That has also been fixed.
        Show
        Thejas M Nair added a comment - PIG-2170 .2.patch fixes a findbugs warning in PIG-2170 .1.patch. Changes - BinCond is not a boolean expression (ie it does not return boolean value), so equivalence classes are not created for it. Boolean expressions can appear in foreach (within bincond), and every where else after boolean type is introduced. But adding equivalence classes for each boolean expression in all these places will generate too many examples, affecting the conciseness. So I have chosen not to add equivalence classes in those cases. When type conversion errors happened, pig tries to increment counters for it, and was throwing NPE because the local mode simulation did not supply the object that returns counters. That has also been fixed.
        Hide
        Thejas M Nair added a comment -

        Unit test and test-patch passed (except for a javac deprecation warning as I am using the old mapred Counters class to create counter. I didn't find a new replacement for it. )
        In addition to new test cases TestExampleGenerator.java has changes to use local mode and also generate the input files only once. (test now takes 5 sec instead of 20).

        Show
        Thejas M Nair added a comment - Unit test and test-patch passed (except for a javac deprecation warning as I am using the old mapred Counters class to create counter. I didn't find a new replacement for it. ) In addition to new test cases TestExampleGenerator.java has changes to use local mode and also generate the input files only once. (test now takes 5 sec instead of 20).
        Hide
        Daniel Dai added a comment -

        Looks good. +1

        Show
        Daniel Dai added a comment - Looks good. +1
        Hide
        Thejas M Nair added a comment -

        patch committed to trunk

        Show
        Thejas M Nair added a comment - patch committed to trunk

          People

          • Assignee:
            Thejas M Nair
            Reporter:
            Mat Kelcey
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development