Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2174

distcp throws a NullPointerException in the close() method of mapper class due to the Reporter becoming invalid

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 0.15.0
    • 0.15.1
    • util
    • None

    Description

      distcp occasionally throws a NullPointerException in the close() method of the mapper class, when the copy of the Reporter handle becomes invalid:

      java.lang.NullPointerException
      at org.apache.hadoop.util.CopyFiles$FSCopyFilesMapper.updateStatus(CopyFiles.java:242)
      at org.apache.hadoop.util.CopyFiles$FSCopyFilesMapper.close(CopyFiles.java:402)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:53)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
      at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)

      This can easily be fixed by checking whether the Reporter is null before updating the status. Also, the status could be updated once the invocation of copy(srcstat, dstpath, out, reporter); returns on line 373 of CopyFiles.java. Marking this as critical for a 0.15.1 release as Chris requested.

      Attachments

        1. 2174-1.patch
          2 kB
          Christopher Douglas

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            cdouglas Christopher Douglas
            drboscolo Riccardo Boscolo
            Votes:
            1 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment