Pig
  1. Pig
  2. PIG-2275

NullPointerException from ILLUSTRATE

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.10.0, 0.11
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I'm getting a NullPointerException from ILLUSTRATE on a simple script using the webcrawl data from the "Programming Pig" book: https://github.com/alanfgates/programmingpig/blob/master/data/webcrawl

      The script is:

      A = load 'webcrawl' USING PigStorage('\t') AS ( url: chararray, pagerank: float, links:{ link: ( url: chararray ) } );
      B = filter A by url is not null;
      C = store B into 'output';
      

      The stack trace I'm getting is:

      java.lang.NullPointerException
      	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFilter.illustratorMarkup(POFilter.java:200)
      	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFilter.getNext(POFilter.java:155)
      	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:290)
      	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getNext(POStore.java:138)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:261)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:256)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:58)
      	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
      	at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:194)
      	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
      	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238)
      	at org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103)
      	at org.apache.pig.pen.LineageTrimmingVisitor.<init>(LineageTrimmingVisitor.java:98)
      	at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166)
      	at org.apache.pig.PigServer.getExamples(PigServer.java:1258)
      	at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
      	at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
      	at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
      	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
      	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
      	at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
      	at org.apache.pig.Main.run(Main.java:456)
      	at org.apache.pig.Main.main(Main.java:108)
      

      I'm running this in local mode, though I don't know if it makes a difference.

      1. PIG-2275-1.patch
        4 kB
        Daniel Dai
      2. PIG-2275-2.patch
        5 kB
        Daniel Dai

        Activity

        Doug Daniels created issue -
        Hide
        Alan Gates added a comment -

        The last line of your script should read "store B into 'output';" (no "C ="). If you change that do you still see the NPE?

        Show
        Alan Gates added a comment - The last line of your script should read "store B into 'output';" (no "C ="). If you change that do you still see the NPE?
        Hide
        Doug Daniels added a comment -

        Good catch. I changed to:

        <code>
        A = load 'webcrawl' USING PigStorage('\t') AS ( url: chararray, pagerank: float, links:

        { link: ( url: chararray ) }

        );
        B = filter A by url is not null;
        store B into 'output';
        </code>

        Same problem though, still an NPE in the same place.

        Show
        Doug Daniels added a comment - Good catch. I changed to: <code> A = load 'webcrawl' USING PigStorage('\t') AS ( url: chararray, pagerank: float, links: { link: ( url: chararray ) } ); B = filter A by url is not null; store B into 'output'; </code> Same problem though, still an NPE in the same place.
        Daniel Dai made changes -
        Field Original Value New Value
        Assignee Daniel Dai [ daijy ]
        Daniel Dai made changes -
        Attachment PIG-2275-1.patch [ 12495422 ]
        Hide
        Daniel Dai added a comment -

        There is a similar test case in PIG-2302. PIG-2275-2.patch address both.

        Show
        Daniel Dai added a comment - There is a similar test case in PIG-2302 . PIG-2275 -2.patch address both.
        Daniel Dai made changes -
        Attachment PIG-2275-2.patch [ 12500963 ]
        Daniel Dai made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Thejas M Nair added a comment -

        +1

        Show
        Thejas M Nair added a comment - +1
        Hide
        Daniel Dai added a comment -

        Unit tests pass. test-patch result:
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] -1 release audit. The applied patch generated 458 release audit warnings (more than the trunk's current 447 warnings).

        No new files added, ignore release audit warning.

        Patch committed to both trunk and 0.10 branch.

        Show
        Daniel Dai added a comment - Unit tests pass. test-patch result: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 458 release audit warnings (more than the trunk's current 447 warnings). No new files added, ignore release audit warning. Patch committed to both trunk and 0.10 branch.
        Daniel Dai made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.10 [ 12316246 ]
        Fix Version/s 0.11 [ 12318878 ]
        Resolution Fixed [ 1 ]
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        47d 1h 29m 1 Daniel Dai 26/Oct/11 22:54
        Patch Available Patch Available Resolved Resolved
        6d 9h 39m 1 Daniel Dai 02/Nov/11 07:34
        Resolved Resolved Closed Closed
        176d 12h 58m 1 Daniel Dai 26/Apr/12 21:33

          People

          • Assignee:
            Daniel Dai
            Reporter:
            Doug Daniels
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development