Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4117

mapred job -status throws NullPointerException

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 3.0.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: client, mrv2
    • Labels:
      None

      Description

      dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status job_1333408894669_0001
      
      Exception in thread "main" java.lang.NullPointerException
          at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
          at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
          at java.lang.String.valueOf(String.java:2838)
          at java.io.PrintStream.println(PrintStream.java:788)
          at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
          at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
      
      1. MAPREDUCE-4117-1.patch
        4 kB
        Devaraj K
      2. MAPREDUCE-4117.patch
        4 kB
        Devaraj K

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1045 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1045/)
        MAPREDUCE-4117. mapred job -status throws NullPointerException (Devaraj K via bobby) (Revision 1311479)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311479
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1045 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1045/ ) MAPREDUCE-4117 . mapred job -status throws NullPointerException (Devaraj K via bobby) (Revision 1311479) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311479 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1010 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1010/)
        MAPREDUCE-4117. mapred job -status throws NullPointerException (Devaraj K via bobby) (Revision 1311479)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311479
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1010 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1010/ ) MAPREDUCE-4117 . mapred job -status throws NullPointerException (Devaraj K via bobby) (Revision 1311479) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311479 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #223 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/223/)
        svn merge -c 1311479 from trunk. FIXES: MAPREDUCE-4117. mapred job -status throws NullPointerException (Devaraj K via bobby) (Revision 1311482)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311482
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #223 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/223/ ) svn merge -c 1311479 from trunk. FIXES: MAPREDUCE-4117 . mapred job -status throws NullPointerException (Devaraj K via bobby) (Revision 1311482) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311482 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2042 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2042/)
        MAPREDUCE-4117. mapred job -status throws NullPointerException (Devaraj K via bobby) (Revision 1311479)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311479
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2042 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2042/ ) MAPREDUCE-4117 . mapred job -status throws NullPointerException (Devaraj K via bobby) (Revision 1311479) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311479 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2031 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2031/)
        MAPREDUCE-4117. mapred job -status throws NullPointerException (Devaraj K via bobby) (Revision 1311479)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311479
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2031 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2031/ ) MAPREDUCE-4117 . mapred job -status throws NullPointerException (Devaraj K via bobby) (Revision 1311479) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311479 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2106 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2106/)
        MAPREDUCE-4117. mapred job -status throws NullPointerException (Devaraj K via bobby) (Revision 1311479)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311479
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2106 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2106/ ) MAPREDUCE-4117 . mapred job -status throws NullPointerException (Devaraj K via bobby) (Revision 1311479) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311479 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
        Hide
        Robert Joseph Evans added a comment -

        I agree with Jon, +1. I merged this into trunk, branch-2, and branch-0.23

        Show
        Robert Joseph Evans added a comment - I agree with Jon, +1. I merged this into trunk, branch-2, and branch-0.23
        Hide
        Jonathan Eagles added a comment -

        I should have stated +1 (non-binding). A committer will need to put this in

        Show
        Jonathan Eagles added a comment - I should have stated +1 (non-binding). A committer will need to put this in
        Hide
        Jonathan Eagles added a comment -

        +1 lgtm
        np. this will be a great addition to the cli. i have verified that the test fail without the patch and succeed with the patch.

        Show
        Jonathan Eagles added a comment - +1 lgtm np. this will be a great addition to the cli. i have verified that the test fail without the patch and succeed with the patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12521978/MAPREDUCE-4117-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
        org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
        org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
        org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2175//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2175//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521978/MAPREDUCE-4117-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2175//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2175//console This message is automatically generated.
        Hide
        Devaraj K added a comment -

        I am sorry Jon. It's my bad, I have mentioned the name wrongly in the above comment.

        Thanks for looking and identifying all the cases. I have attached the patch as per your suggestion.

        Show
        Devaraj K added a comment - I am sorry Jon. It's my bad, I have mentioned the name wrongly in the above comment. Thanks for looking and identifying all the cases. I have attached the patch as per your suggestion.
        Hide
        Jonathan Eagles added a comment -

        My initial thought was the same as yours. However, in TypeConverter, you will see that both JobState.FAILED and JobState.ERROR are translated to JobStatus.State.FAILED. I believe it to be to much to assume that if JobStatus.State is FAILED, then you will be guaranteed to have either 1)a failed task completion event or 2) no task completion events at all. It is much more clear and simple to me to check for null in this case.

        Show
        Jonathan Eagles added a comment - My initial thought was the same as yours. However, in TypeConverter, you will see that both JobState.FAILED and JobState.ERROR are translated to JobStatus.State.FAILED. I believe it to be to much to assume that if JobStatus.State is FAILED, then you will be guaranteed to have either 1)a failed task completion event or 2) no task completion events at all. It is much more clear and simple to me to check for null in this case.
        Hide
        Devaraj K added a comment -

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
        org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
        org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
        org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

        These test failures are not related to the patch.

        Show
        Devaraj K added a comment - -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs These test failures are not related to the patch.
        Hide
        Devaraj K added a comment -

        Thanks Bobby for taking a look into the patch.

        When the Job is failed and it doesn't contain any task attempts(i.e Job is failed before launching any tasks itself) then the lastEvent will be null and it will lead to NPE.

        If the Job fails and job contains some attempts means the job should have failed because of task failures. This case lastEvent will not be null.

        Show
        Devaraj K added a comment - Thanks Bobby for taking a look into the patch. When the Job is failed and it doesn't contain any task attempts(i.e Job is failed before launching any tasks itself) then the lastEvent will be null and it will lead to NPE. If the Job fails and job contains some attempts means the job should have failed because of task failures. This case lastEvent will not be null.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12521957/MAPREDUCE-4117.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
        org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
        org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
        org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2173//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2173//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521957/MAPREDUCE-4117.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2173//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2173//console This message is automatically generated.
        Hide
        Jonathan Eagles added a comment -

        Thanks for the patch Devaraj. One case I'm wondering about is when getTaskFailureEventString is called on a Job with only non-TaskCompletionEvent.Status.FAILED events. Seems like lastEvent will still be null in that case.

        Show
        Jonathan Eagles added a comment - Thanks for the patch Devaraj. One case I'm wondering about is when getTaskFailureEventString is called on a Job with only non-TaskCompletionEvent.Status.FAILED events. Seems like lastEvent will still be null in that case.
        Hide
        Devaraj K added a comment -

        I have attached patch to fix this issue.

        Show
        Devaraj K added a comment - I have attached patch to fix this issue.

          People

          • Assignee:
            Devaraj K
            Reporter:
            Devaraj K
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development