Pig
  1. Pig
  2. PIG-2275

NullPointerException from ILLUSTRATE

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.10.0, 0.11
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I'm getting a NullPointerException from ILLUSTRATE on a simple script using the webcrawl data from the "Programming Pig" book: https://github.com/alanfgates/programmingpig/blob/master/data/webcrawl

      The script is:

      A = load 'webcrawl' USING PigStorage('\t') AS ( url: chararray, pagerank: float, links:{ link: ( url: chararray ) } );
      B = filter A by url is not null;
      C = store B into 'output';
      

      The stack trace I'm getting is:

      java.lang.NullPointerException
      	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFilter.illustratorMarkup(POFilter.java:200)
      	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFilter.getNext(POFilter.java:155)
      	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:290)
      	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getNext(POStore.java:138)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:261)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:256)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:58)
      	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
      	at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:194)
      	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
      	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238)
      	at org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103)
      	at org.apache.pig.pen.LineageTrimmingVisitor.<init>(LineageTrimmingVisitor.java:98)
      	at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166)
      	at org.apache.pig.PigServer.getExamples(PigServer.java:1258)
      	at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
      	at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
      	at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
      	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
      	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
      	at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
      	at org.apache.pig.Main.run(Main.java:456)
      	at org.apache.pig.Main.main(Main.java:108)
      

      I'm running this in local mode, though I don't know if it makes a difference.

      1. PIG-2275-1.patch
        4 kB
        Daniel Dai
      2. PIG-2275-2.patch
        5 kB
        Daniel Dai

        Activity

        Hide
        Alan Gates added a comment -

        The last line of your script should read "store B into 'output';" (no "C ="). If you change that do you still see the NPE?

        Show
        Alan Gates added a comment - The last line of your script should read "store B into 'output';" (no "C ="). If you change that do you still see the NPE?
        Hide
        Doug Daniels added a comment -

        Good catch. I changed to:

        <code>
        A = load 'webcrawl' USING PigStorage('\t') AS ( url: chararray, pagerank: float, links:

        { link: ( url: chararray ) }

        );
        B = filter A by url is not null;
        store B into 'output';
        </code>

        Same problem though, still an NPE in the same place.

        Show
        Doug Daniels added a comment - Good catch. I changed to: <code> A = load 'webcrawl' USING PigStorage('\t') AS ( url: chararray, pagerank: float, links: { link: ( url: chararray ) } ); B = filter A by url is not null; store B into 'output'; </code> Same problem though, still an NPE in the same place.
        Hide
        Daniel Dai added a comment -

        There is a similar test case in PIG-2302. PIG-2275-2.patch address both.

        Show
        Daniel Dai added a comment - There is a similar test case in PIG-2302 . PIG-2275 -2.patch address both.
        Hide
        Thejas M Nair added a comment -

        +1

        Show
        Thejas M Nair added a comment - +1
        Hide
        Daniel Dai added a comment -

        Unit tests pass. test-patch result:
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] -1 release audit. The applied patch generated 458 release audit warnings (more than the trunk's current 447 warnings).

        No new files added, ignore release audit warning.

        Patch committed to both trunk and 0.10 branch.

        Show
        Daniel Dai added a comment - Unit tests pass. test-patch result: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 458 release audit warnings (more than the trunk's current 447 warnings). No new files added, ignore release audit warning. Patch committed to both trunk and 0.10 branch.

          People

          • Assignee:
            Daniel Dai
            Reporter:
            Doug Daniels
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development