[HADOOP-857] IOException when running map reduce on S3 filesystem

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.10.1
• Component/s: fs
• Assignee: Unassigned
• Reporter: Thomas White
• Created: 04/Jan/07 21:24
• Updated: 11/Jan/07 00:27
• Resolved: 08/Jan/07 18:22

Description

Setting fs.default.name to be an S3 path causes the following exception:

java.io.IOException: Cannot create file /tmp/hadoop-tom/mapred/system/submit_habo0/job.jar since parent directory does not exist.
at org.apache.hadoop.fs.s3.S3FileSystem.createRaw(S3FileSystem.java:150)
at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(FSDataOutputStream.java:58)
at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(FSDataOutputStream.java:47)
at org.apache.hadoop.fs.FSDataOutputStream.<init>(FSDataOutputStream.java:148)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:369)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:276)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:94)
at org.apache.hadoop.fs.s3.S3FileSystem.copyFromLocalFile(S3FileSystem.java:290)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:294)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:371)
at org.apache.hadoop.examples.Grep.main(Grep.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:143)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.util.RunJar.main(RunJar.java:151)