[PIG-2540] AvroStorage can't read schema on amazon s3 in elastic mapreduce

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.9.1, 0.10.0
• Fix Version/s: 0.10.0, 0.9.3, 0.11
• Component/s: data, piggybank
• Labels: avro avro_udf aws emr pants pig s3 sad unhappy

Environment

Amazon Elastic MapReduce

• Patch Info: Patch Available
• Hadoop Flags: Reviewed
• Assignee: Russell Jurney
• Reporter: Russell Jurney
• Created: 16/Feb/12 22:16
• Updated: 22/Jun/12 03:36
• Resolved: 27/Mar/12 02:56

Description

grunt> emails = load 's3://agile.data/again_inbox' using AvroStorage();
grunt> describe emails
Schema for emails unknown.
grunt> a = limit emails 10;
grunt> dump a
2012-02-16 22:15:58,347 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: LIMIT
2012-02-16 22:15:58,483 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2012-02-16 22:15:58,542 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2012-02-16 22:15:58,542 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2012-02-16 22:15:58,632 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2012-02-16 22:15:58,658 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2012-02-16 22:15:58,665 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2017: Internal error creating job configuration.
2012-02-16 22:15:58,665 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias a
at org.apache.pig.PigServer.openIterator(PigServer.java:901)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:652)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:67)
at org.apache.pig.Main.run(Main.java:497)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias a
at org.apache.pig.PigServer.storeEx(PigServer.java:1000)
at org.apache.pig.PigServer.store(PigServer.java:963)
at org.apache.pig.PigServer.openIterator(PigServer.java:876)
... 12 more
Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2017: Internal error creating job configuration.
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:731)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:263)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:149)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1314)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1299)
at org.apache.pig.PigServer.storeEx(PigServer.java:996)
... 14 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:352)
at org.apache.pig.piggybank.storage.avro.AvroStorage.setLocation(AvroStorage.java:138)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:387)
... 19 more