Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 1.0.3, 3.0.0, 2.4.0
    • Fix Version/s: 2.5.0
    • Component/s: fs/s3
    • Labels:
      None
    • Environment:

      Hadoop with default configurations

    • Tags:
      mapreduce, s3, mr, hadoop

      Description

      I'm running a wordcount MR as follows

      hadoop jar WordCount.jar wordcount.WordCountDriver s3n://bucket/wordcount/input s3n://bucket/wordcount/output

      s3n://bucket/wordcount/input is a s3 object that contains other input files.

      However I get following NPE error

      12/10/02 18:56:23 INFO mapred.JobClient: map 0% reduce 0%
      12/10/02 18:56:54 INFO mapred.JobClient: map 50% reduce 0%
      12/10/02 18:56:56 INFO mapred.JobClient: Task Id : attempt_201210021853_0001_m_000001_0, Status : FAILED
      java.lang.NullPointerException
      at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.close(NativeS3FileSystem.java:106)
      at java.io.BufferedInputStream.close(BufferedInputStream.java:451)
      at java.io.FilterInputStream.close(FilterInputStream.java:155)
      at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
      at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144)
      at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
      at org.apache.hadoop.mapred.Child.main(Child.java:249)

      MR runs fine if i specify more specific input path such as s3n://bucket/wordcount/input/file.txt

      MR fails if I pass s3 folder as a parameter

      In summary,
      This works
      hadoop jar ./hadoop-examples-1.0.3.jar wordcount /user/hadoop/wordcount/input/ s3n://bucket/wordcount/output/

      This doesn't work
      hadoop jar ./hadoop-examples-1.0.3.jar wordcount s3n://bucket/wordcount/input/ s3n://bucket/wordcount/output/

      (both input path are directories)

        Issue Links

          Activity

          Ted Yu made changes -
          Summary S3 input stream NPEs in MapReduce jon S3 input stream NPEs in MapReduce job
          Karthik Kambatla (Inactive) made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 2.5.0 [ 12326263 ]
          Resolution Fixed [ 1 ]
          Steve Loughran made changes -
          Assignee Steve Loughran [ stevel@apache.org ]
          Steve Loughran made changes -
          Link This issue relates to HADOOP-10589 [ HADOOP-10589 ]
          Steve Loughran made changes -
          Link This issue is duplicated by HADOOP-10589 [ HADOOP-10589 ]
          Steve Loughran made changes -
          Link This issue is part of HADOOP-9361 [ HADOOP-9361 ]
          Steve Loughran made changes -
          Link This issue is duplicated by HADOOP-10589 [ HADOOP-10589 ]
          Steve Loughran made changes -
          Affects Version/s 2.4.0 [ 12326144 ]
          Affects Version/s 3.0.0 [ 12320357 ]
          Steve Loughran made changes -
          Summary MapReduce fails If I pass a parameter as a S3 folder S3 input stream NPEs in MapReduce jon
          Priority Major [ 3 ] Minor [ 4 ]
          Steve Loughran made changes -
          Project Hadoop Map/Reduce [ 12310941 ] Hadoop Common [ 12310240 ]
          Key MAPREDUCE-4718 HADOOP-10533
          Affects Version/s 1.0.3 [ 12320248 ]
          Affects Version/s 1.0.0 [ 12318244 ]
          Affects Version/s 1.0.0 [ 12318240 ]
          Affects Version/s 1.0.3 [ 12320250 ]
          Target Version/s 1.0.3 [ 12320250 ]
          Component/s fs/s3 [ 12311814 ]
          Component/s job submission [ 12313413 ]
          Chen He made changes -
          Target Version/s 1.0.3, 0.23.3 [ 12320250, 12320060 ] 1.0.3 [ 12320250 ]
          Benjamin Kim made changes -
          Field Original Value New Value
          Target Version/s 1.0.3, 0.23.3, 2.0.0-alpha, 2.0.1-alpha [ 12320250, 12320060, 12320354, 12322466 ] 0.23.3, 1.0.3 [ 12320060, 12320250 ]
          Benjamin Kim created issue -

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Benjamin Kim
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development