Pig
  1. Pig
  2. PIG-2527

ILLUSTRATE fails for relations LOADed with the AvroStorage UDF

    Details

    • Release Note:
      Hide
      The fix for this wasn't committed for 0.10.0, and I'm not sure where the fix is/if it was committed at all.

      Can anyone remember what happened with the fixes for AvroStorage that didn't go in 0.10.0?
      Show
      The fix for this wasn't committed for 0.10.0, and I'm not sure where the fix is/if it was committed at all. Can anyone remember what happened with the fixes for AvroStorage that didn't go in 0.10.0?

      Description

      grunt> describe emails
      emails: {message_id: chararray,from:

      {PIG_WRAPPER: (ARRAY_ELEM: chararray)}

      ,to:

      {PIG_WRAPPER: (ARRAY_ELEM: chararray)}

      ,cc:

      {PIG_WRAPPER: (ARRAY_ELEM: chararray)}

      ,bcc:

      {PIG_WRAPPER: (ARRAY_ELEM: chararray)}

      ,reply_to:

      {PIG_WRAPPER: (ARRAY_ELEM: chararray)}

      ,in_reply_to:

      {PIG_WRAPPER: (ARRAY_ELEM: chararray)}

      ,subject: chararray,body: chararray,date: chararray}

      grunt> illustrate emails
      2012-02-10 18:15:01,591 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
      2012-02-10 18:15:01,592 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
      2012-02-10 18:15:01,649 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
      2012-02-10 18:15:01,649 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
      2012-02-10 18:15:01,649 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
      2012-02-10 18:15:01,649 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
      2012-02-10 18:15:01,668 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 5
      2012-02-10 18:15:02,719 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
      2012-02-10 18:15:02,719 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
      2012-02-10 18:15:02,719 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
      2012-02-10 18:15:02,720 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
      2012-02-10 18:15:02,720 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
      2012-02-10 18:15:02,733 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
      2012-02-10 18:15:02,734 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
      2012-02-10 18:15:02,734 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
      2012-02-10 18:15:02,734 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
      2012-02-10 18:15:02,734 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
      2012-02-10 18:15:02,749 [main] ERROR org.apache.pig.pen.AugmentBaseDataVisitor - No (valid) input data found!
      java.lang.RuntimeException: No (valid) input data found!
      at org.apache.pig.pen.AugmentBaseDataVisitor.visit(AugmentBaseDataVisitor.java:579)
      at org.apache.pig.newplan.logical.relational.LOLoad.accept(LOLoad.java:218)
      at org.apache.pig.pen.util.PreOrderDepthFirstWalker.depthFirst(PreOrderDepthFirstWalker.java:82)
      at org.apache.pig.pen.util.PreOrderDepthFirstWalker.walk(PreOrderDepthFirstWalker.java:66)
      at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
      at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:180)
      at org.apache.pig.PigServer.getExamples(PigServer.java:1245)
      at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
      at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
      at org.apache.pig.Main.run(Main.java:495)
      at org.apache.pig.Main.main(Main.java:111)
      2012-02-10 18:15:02,752 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2997: Encountered IOException. Exception : No (valid) input data found!

      ================================================================================
      Pig Stack Trace
      ---------------
      ERROR 2997: Encountered IOException. Exception : No (valid) input data found!

      java.io.IOException: Exception : No (valid) input data found!
      at org.apache.pig.PigServer.getExamples(PigServer.java:1251)
      at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
      at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
      at org.apache.pig.Main.run(Main.java:495)
      at org.apache.pig.Main.main(Main.java:111)
      ================================================================================

        Activity

        Hide
        Russell Jurney added a comment -

        I don't remember how to do this. Does anyone remember how to do this?

        Show
        Russell Jurney added a comment - I don't remember how to do this. Does anyone remember how to do this?
        Hide
        Russell Jurney added a comment -

        This issue was inadvertently resolved in Pig 0.10. WOOT!

        Show
        Russell Jurney added a comment - This issue was inadvertently resolved in Pig 0.10. WOOT!
        Hide
        Russell Jurney added a comment -

        The fix for AvroStorage that fixed ILLUSTRATE didn't go in Pig 0.10.0, so this is still open.

        Show
        Russell Jurney added a comment - The fix for AvroStorage that fixed ILLUSTRATE didn't go in Pig 0.10.0, so this is still open.
        Hide
        Daniel Dai added a comment -

        Verified it is fixed in 0.10 branch. Mark it as fixed in 0.10.1.

        Show
        Daniel Dai added a comment - Verified it is fixed in 0.10 branch. Mark it as fixed in 0.10.1.
        Hide
        Rohini Palaniswamy added a comment -

        Daniel Dai,
        Do you know which jira incorporates this fix? I don't see any patch attached to this jira. One of our users encountered a similar stacktrace with illustrate and PigStorage.

        Show
        Rohini Palaniswamy added a comment - Daniel Dai , Do you know which jira incorporates this fix? I don't see any patch attached to this jira. One of our users encountered a similar stacktrace with illustrate and PigStorage.
        Hide
        Daniel Dai added a comment -

        Rohini Palaniswamy, I didn't investigate which jira fix the issue. I did a trivial illustrate and it works. If you have a script which fail, can you open a new ticket and provide more details?

        Show
        Daniel Dai added a comment - Rohini Palaniswamy , I didn't investigate which jira fix the issue. I did a trivial illustrate and it works. If you have a script which fail, can you open a new ticket and provide more details?

          People

          • Assignee:
            Jonathan Coveney
            Reporter:
            Russell Jurney
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development