Pig
  1. Pig
  2. PIG-2527

ILLUSTRATE fails for relations LOADed with the AvroStorage UDF

    Details

    • Release Note:
      Hide
      The fix for this wasn't committed for 0.10.0, and I'm not sure where the fix is/if it was committed at all.

      Can anyone remember what happened with the fixes for AvroStorage that didn't go in 0.10.0?
      Show
      The fix for this wasn't committed for 0.10.0, and I'm not sure where the fix is/if it was committed at all. Can anyone remember what happened with the fixes for AvroStorage that didn't go in 0.10.0?

      Description

      grunt> describe emails
      emails: {message_id: chararray,from:

      {PIG_WRAPPER: (ARRAY_ELEM: chararray)}

      ,to:

      {PIG_WRAPPER: (ARRAY_ELEM: chararray)}

      ,cc:

      {PIG_WRAPPER: (ARRAY_ELEM: chararray)}

      ,bcc:

      {PIG_WRAPPER: (ARRAY_ELEM: chararray)}

      ,reply_to:

      {PIG_WRAPPER: (ARRAY_ELEM: chararray)}

      ,in_reply_to:

      {PIG_WRAPPER: (ARRAY_ELEM: chararray)}

      ,subject: chararray,body: chararray,date: chararray}

      grunt> illustrate emails
      2012-02-10 18:15:01,591 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
      2012-02-10 18:15:01,592 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
      2012-02-10 18:15:01,649 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
      2012-02-10 18:15:01,649 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
      2012-02-10 18:15:01,649 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
      2012-02-10 18:15:01,649 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
      2012-02-10 18:15:01,668 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 5
      2012-02-10 18:15:02,719 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
      2012-02-10 18:15:02,719 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
      2012-02-10 18:15:02,719 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
      2012-02-10 18:15:02,720 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
      2012-02-10 18:15:02,720 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
      2012-02-10 18:15:02,733 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
      2012-02-10 18:15:02,734 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
      2012-02-10 18:15:02,734 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
      2012-02-10 18:15:02,734 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
      2012-02-10 18:15:02,734 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
      2012-02-10 18:15:02,749 [main] ERROR org.apache.pig.pen.AugmentBaseDataVisitor - No (valid) input data found!
      java.lang.RuntimeException: No (valid) input data found!
      at org.apache.pig.pen.AugmentBaseDataVisitor.visit(AugmentBaseDataVisitor.java:579)
      at org.apache.pig.newplan.logical.relational.LOLoad.accept(LOLoad.java:218)
      at org.apache.pig.pen.util.PreOrderDepthFirstWalker.depthFirst(PreOrderDepthFirstWalker.java:82)
      at org.apache.pig.pen.util.PreOrderDepthFirstWalker.walk(PreOrderDepthFirstWalker.java:66)
      at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
      at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:180)
      at org.apache.pig.PigServer.getExamples(PigServer.java:1245)
      at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
      at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
      at org.apache.pig.Main.run(Main.java:495)
      at org.apache.pig.Main.main(Main.java:111)
      2012-02-10 18:15:02,752 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2997: Encountered IOException. Exception : No (valid) input data found!

      ================================================================================
      Pig Stack Trace
      ---------------
      ERROR 2997: Encountered IOException. Exception : No (valid) input data found!

      java.io.IOException: Exception : No (valid) input data found!
      at org.apache.pig.PigServer.getExamples(PigServer.java:1251)
      at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
      at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
      at org.apache.pig.Main.run(Main.java:495)
      at org.apache.pig.Main.main(Main.java:111)
      ================================================================================

        Activity

        Russell Jurney created issue -
        Hide
        Russell Jurney added a comment -

        I don't remember how to do this. Does anyone remember how to do this?

        Show
        Russell Jurney added a comment - I don't remember how to do this. Does anyone remember how to do this?
        Hide
        Russell Jurney added a comment -

        This issue was inadvertently resolved in Pig 0.10. WOOT!

        Show
        Russell Jurney added a comment - This issue was inadvertently resolved in Pig 0.10. WOOT!
        Russell Jurney made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Russell Jurney made changes -
        Release Note The fix for this wasn't committed for 0.10.0, and I'm not sure where the fix is/if it was committed at all.

        Can anyone remember what happened with the fixes for AvroStorage that didn't go in 0.10.0?
        Fix Version/s 0.10.1 [ 12320547 ]
        Affects Version/s 0.10.0 [ 12316246 ]
        Affects Version/s 0.11 [ 12318878 ]
        Affects Version/s 0.10.1 [ 12320547 ]
        Priority Critical [ 2 ] Blocker [ 1 ]
        Hide
        Russell Jurney added a comment -

        The fix for AvroStorage that fixed ILLUSTRATE didn't go in Pig 0.10.0, so this is still open.

        Show
        Russell Jurney added a comment - The fix for AvroStorage that fixed ILLUSTRATE didn't go in Pig 0.10.0, so this is still open.
        Russell Jurney made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Assignee Jonathan Coveney [ jcoveney ]
        Daniel Dai made changes -
        Fix Version/s 0.10.1 [ 12320547 ]
        Daniel Dai made changes -
        Fix Version/s 0.10.1 [ 12320547 ]
        Hide
        Daniel Dai added a comment -

        Verified it is fixed in 0.10 branch. Mark it as fixed in 0.10.1.

        Show
        Daniel Dai added a comment - Verified it is fixed in 0.10 branch. Mark it as fixed in 0.10.1.
        Daniel Dai made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Rohini Palaniswamy added a comment -

        Daniel Dai,
        Do you know which jira incorporates this fix? I don't see any patch attached to this jira. One of our users encountered a similar stacktrace with illustrate and PigStorage.

        Show
        Rohini Palaniswamy added a comment - Daniel Dai , Do you know which jira incorporates this fix? I don't see any patch attached to this jira. One of our users encountered a similar stacktrace with illustrate and PigStorage.
        Hide
        Daniel Dai added a comment -

        Rohini Palaniswamy, I didn't investigate which jira fix the issue. I did a trivial illustrate and it works. If you have a script which fail, can you open a new ticket and provide more details?

        Show
        Daniel Dai added a comment - Rohini Palaniswamy , I didn't investigate which jira fix the issue. I did a trivial illustrate and it works. If you have a script which fail, can you open a new ticket and provide more details?
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        96d 4h 56m 1 Russell Jurney 17/May/12 08:13
        Resolved Resolved Reopened Reopened
        67d 16h 4m 1 Russell Jurney 24/Jul/12 00:18
        Reopened Reopened Resolved Resolved
        148d 17h 14m 1 Daniel Dai 19/Dec/12 16:33
        Resolved Resolved Closed Closed
        18d 7h 23m 1 Daniel Dai 06/Jan/13 23:57

          People

          • Assignee:
            Jonathan Coveney
            Reporter:
            Russell Jurney
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development