[HADOOP-10533] S3 input stream NPEs in MapReduce job

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.0.0, 1.0.3, 2.4.0, 3.0.0-alpha1
• Fix Version/s: 2.5.0
• Component/s: fs/s3

Environment

Hadoop with default configurations

• Tags: mapreduce, s3, mr, hadoop
• Assignee: Steve Loughran
• Reporter: Benjamin Kim
• Created: 10/Oct/12 11:30
• Updated: 12/May/16 18:27
• Resolved: 03/Jul/14 13:04

Description

I'm running a wordcount MR as follows

hadoop jar WordCount.jar wordcount.WordCountDriver s3n://bucket/wordcount/input s3n://bucket/wordcount/output

s3n://bucket/wordcount/input is a s3 object that contains other input files.

However I get following NPE error

12/10/02 18:56:23 INFO mapred.JobClient: map 0% reduce 0%
12/10/02 18:56:54 INFO mapred.JobClient: map 50% reduce 0%
12/10/02 18:56:56 INFO mapred.JobClient: Task Id : attempt_201210021853_0001_m_000001_0, Status : FAILED
java.lang.NullPointerException
at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.close(NativeS3FileSystem.java:106)
at java.io.BufferedInputStream.close(BufferedInputStream.java:451)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

MR runs fine if i specify more specific input path such as s3n://bucket/wordcount/input/file.txt

MR fails if I pass s3 folder as a parameter

In summary,
This works
hadoop jar ./hadoop-examples-1.0.3.jar wordcount /user/hadoop/wordcount/input/ s3n://bucket/wordcount/output/

This doesn't work
hadoop jar ./hadoop-examples-1.0.3.jar wordcount s3n://bucket/wordcount/input/ s3n://bucket/wordcount/output/

(both input path are directories)

Issue Links

• is part of: HADOOP-9361 Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
• relates to: HADOOP-10589 NativeS3FileSystem throw NullPointerException when the file is empty