Hadoop Common
  1. Hadoop Common
  2. HADOOP-2174

distcp throws a NullPointerException in the close() method of mapper class due to the Reporter becoming invalid

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.15.0
    • Fix Version/s: 0.15.1
    • Component/s: util
    • Labels:
      None

      Description

      distcp occasionally throws a NullPointerException in the close() method of the mapper class, when the copy of the Reporter handle becomes invalid:

      java.lang.NullPointerException
      at org.apache.hadoop.util.CopyFiles$FSCopyFilesMapper.updateStatus(CopyFiles.java:242)
      at org.apache.hadoop.util.CopyFiles$FSCopyFilesMapper.close(CopyFiles.java:402)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:53)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
      at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)

      This can easily be fixed by checking whether the Reporter is null before updating the status. Also, the status could be updated once the invocation of copy(srcstat, dstpath, out, reporter); returns on line 373 of CopyFiles.java. Marking this as critical for a 0.15.1 release as Chris requested.

      1. 2174-1.patch
        2 kB
        Chris Douglas

        Activity

        Riccardo Boscolo created issue -
        Hide
        Chris Douglas added a comment -

        Added a null check in close()

        Show
        Chris Douglas added a comment - Added a null check in close()
        Chris Douglas made changes -
        Field Original Value New Value
        Attachment 2174.patch [ 12369203 ]
        Chris Douglas made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Chris Douglas added a comment -

        On second thought, that's a pretty lame fix. The reporter isn't even necessary in close(); Riccardo's suggestion is the better fix.

        Show
        Chris Douglas added a comment - On second thought, that's a pretty lame fix. The reporter isn't even necessary in close(); Riccardo's suggestion is the better fix.
        Chris Douglas made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Chris Douglas made changes -
        Attachment 2174-1.patch [ 12369205 ]
        Chris Douglas made changes -
        Attachment 2174.patch [ 12369203 ]
        Chris Douglas made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12369203/2174.patch
        against trunk revision r592860.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests -1. The patch failed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1076/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1076/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1076/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1076/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12369203/2174.patch against trunk revision r592860. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1076/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1076/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1076/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1076/console This message is automatically generated.
        Chris Douglas made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Chris Douglas added a comment -

        Timeout in TestBlockReplacement; trying again.

        Show
        Chris Douglas added a comment - Timeout in TestBlockReplacement; trying again.
        Chris Douglas made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12369205/2174-1.patch
        against trunk revision r592860.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1078/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1078/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1078/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1078/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12369205/2174-1.patch against trunk revision r592860. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1078/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1078/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1078/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1078/console This message is automatically generated.
        Arun C Murthy committed 593748 (2 files)
        Reviews: none

        HADOOP-2174. Removed the unnecessary Reporter.setStatus call from FSCopyFilesMapper.close which led to a NPE since the reporter isn't valid in the close method. Contributed by Chris Douglas.

        Arun C Murthy committed 593750 (2 files)
        Arun C Murthy made changes -
        Assignee Chris Douglas [ chris.douglas ]
        Hide
        Arun C Murthy added a comment -

        I just committed this. Thanks, Chris!

        Show
        Arun C Murthy added a comment - I just committed this. Thanks, Chris!
        Arun C Murthy made changes -
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-Nightly #299 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/299/ )
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Chris Douglas
            Reporter:
            Riccardo Boscolo
          • Votes:
            1 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development