Pig
  1. Pig
  2. PIG-2540

AvroStorage can't read schema on amazon s3 in elastic mapreduce

    Details

    • Patch Info:
      Patch Available
    • Hadoop Flags:
      Reviewed

      Description

      grunt> emails = load 's3://agile.data/again_inbox' using AvroStorage();
      grunt> describe emails
      Schema for emails unknown.
      grunt> a = limit emails 10;
      grunt> dump a
      2012-02-16 22:15:58,347 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: LIMIT
      2012-02-16 22:15:58,483 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
      2012-02-16 22:15:58,542 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
      2012-02-16 22:15:58,542 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
      2012-02-16 22:15:58,632 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
      2012-02-16 22:15:58,658 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
      2012-02-16 22:15:58,665 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2017: Internal error creating job configuration.
      2012-02-16 22:15:58,665 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias a
      at org.apache.pig.PigServer.openIterator(PigServer.java:901)
      at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:652)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
      at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:67)
      at org.apache.pig.Main.run(Main.java:497)
      at org.apache.pig.Main.main(Main.java:111)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
      Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias a
      at org.apache.pig.PigServer.storeEx(PigServer.java:1000)
      at org.apache.pig.PigServer.store(PigServer.java:963)
      at org.apache.pig.PigServer.openIterator(PigServer.java:876)
      ... 12 more
      Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2017: Internal error creating job configuration.
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:731)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:263)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:149)
      at org.apache.pig.PigServer.launchPlan(PigServer.java:1314)
      at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1299)
      at org.apache.pig.PigServer.storeEx(PigServer.java:996)
      ... 14 more
      Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
      at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:352)
      at org.apache.pig.piggybank.storage.avro.AvroStorage.setLocation(AvroStorage.java:138)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:387)
      ... 19 more

      1. TEST-org.apache.pig.piggybank.test.storage.avro.TestAvroStorage.txt
        84 kB
        Russell Jurney
      2. PIG-2540.tests_fail.patch.2
        7 kB
        Russell Jurney
      3. PIG-2540.tests_fail.patch
        6 kB
        Russell Jurney
      4. PIG-2540_almost_there.patch
        5 kB
        Jonathan Coveney
      5. PIG-2540_4.patch
        5 kB
        Jonathan Coveney

        Activity

        Russell Jurney created issue -
        Russell Jurney made changes -
        Field Original Value New Value
        Affects Version/s 0.10 [ 12316246 ]
        Russell Jurney made changes -
        Attachment PIG-2540.tests_fail.patch [ 12519009 ]
        Russell Jurney made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Russell Jurney made changes -
        Fix Version/s 0.10 [ 12316246 ]
        Patch Info Patch Available [ 10042 ]
        Russell Jurney made changes -
        Jonathan Coveney made changes -
        Attachment PIG-2540_almost_there.patch [ 12519149 ]
        Jonathan Coveney made changes -
        Assignee Russell Jurney [ rjurney ]
        Russell Jurney made changes -
        Attachment PIG-2540.tests_fail.patch.2 [ 12519169 ]
        Jonathan Coveney made changes -
        Attachment PIG-2540_4.patch [ 12520052 ]
        Jonathan Coveney made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.9.3 [ 12319456 ]
        Fix Version/s 0.11 [ 12318878 ]
        Resolution Fixed [ 1 ]
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Gavin made changes -
        Assignee Russell Jurney [ rjurney ] Russell Jurney [ russell.jurney ]

          People

          • Assignee:
            Russell Jurney
            Reporter:
            Russell Jurney
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development