Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2618

MR-279: 0 map, 0 reduce job fails with Null Pointer Exception

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      mrv2

      Description

      A 0 map, 0 reduce job fails with an NPE. This case works fine on hadoop-0.20.x. The job should succeed and run setup/cleanup code - with no tasks. Below is the stacktrace:

      11/06/05 19:35:37 WARN mapred.ClientServiceDelegate:
      StackTrace: java.lang.NullPointerException
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.getTaskAttemptCompletionEvents(JobImpl.java:498)
      at
      org.apache.hadoop.mapreduce.v2.app.client.MRClientService$MRClientProtocolHandler.getTaskAttemptCompletionEvents(MRClientService.java:290)
      at
      org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:139)
      at
      org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:195)
      at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$TunnelResponder.call(ProtoOverHadoopRpcEngine.java:168)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:420)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1406)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1402)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1094)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1400)

      1. MAPREDUCE-2618.patch
        12 kB
        Jeffrey Naisbitt
      2. MAPREDUCE-2618-v2.patch
        12 kB
        Jeffrey Naisbitt

        Activity

        Hide
        Luke Lu added a comment -

        Tested against MR-279 head. Committed to MR-279. Thanks Jeff! Thanks Seth for reviewing!

        Show
        Luke Lu added a comment - Tested against MR-279 head. Committed to MR-279. Thanks Jeff! Thanks Seth for reviewing!
        Hide
        Siddharth Seth added a comment -

        Thanks Jeffrey.
        +1, looks good.

        Show
        Siddharth Seth added a comment - Thanks Jeffrey. +1, looks good.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12483631/MAPREDUCE-2618-v2.patch
        against trunk revision 1138301.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/414//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483631/MAPREDUCE-2618-v2.patch against trunk revision 1138301. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/414//console This message is automatically generated.
        Hide
        Jeffrey Naisbitt added a comment -

        New patch addresses Siddharth's comments.

        The failing unit test was actually testing the condition this patch reverses. I modified that test and removed the equivalent test I had placed in my new tests.

        Show
        Jeffrey Naisbitt added a comment - New patch addresses Siddharth's comments. The failing unit test was actually testing the condition this patch reverses. I modified that test and removed the equivalent test I had placed in my new tests.
        Hide
        Jeffrey Naisbitt added a comment -

        I thought I had run all the tests, but apparently I didn't pay enough attention to the results...

        I originally made checkJobCompleteSuccess public for testing, but I changed things so it does only need to be package-scoped now - good catch.
        I renamed logJobHistory as you recommended.

        Thanks for the review! I'll submit an updated patch soon.

        Show
        Jeffrey Naisbitt added a comment - I thought I had run all the tests, but apparently I didn't pay enough attention to the results... I originally made checkJobCompleteSuccess public for testing, but I changed things so it does only need to be package-scoped now - good catch. I renamed logJobHistory as you recommended. Thanks for the review! I'll submit an updated patch soon.
        Hide
        Siddharth Seth added a comment -

        Jeffrey, there's a unit test failure after applying the patch - TestMRApp.
        Also, does checkJobCompleteSuccess need to be public? logJobHistory() could be renamed to logJobHistoryFinishedEvent()

        Thx

        Show
        Siddharth Seth added a comment - Jeffrey, there's a unit test failure after applying the patch - TestMRApp. Also, does checkJobCompleteSuccess need to be public? logJobHistory() could be renamed to logJobHistoryFinishedEvent() Thx
        Hide
        Jeffrey Naisbitt added a comment -

        As stated, this patch is for the MR-279 branch, so the above output does not apply.

        Show
        Jeffrey Naisbitt added a comment - As stated, this patch is for the MR-279 branch, so the above output does not apply.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12483611/MAPREDUCE-2618.patch
        against trunk revision 1138301.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/412//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483611/MAPREDUCE-2618.patch against trunk revision 1138301. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/412//console This message is automatically generated.
        Hide
        Jeffrey Naisbitt added a comment -

        Patch for MR-279 branch

        Show
        Jeffrey Naisbitt added a comment - Patch for MR-279 branch

          People

          • Assignee:
            Jeffrey Naisbitt
            Reporter:
            Jeffrey Naisbitt
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development