Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4290

JobStatus.getState() API is giving ambiguous values

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 3.0.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      For failed job getState() API is giving status as SUCCEEDED if we use JobClient.getAllJobs() for retrieving all jobs info from RM.

      1. MAPREDUCE-4290-1.patch
        9 kB
        Siddharth Seth
      2. MAPREDUCE-4290-1.patch
        9 kB
        Devaraj K
      3. MAPREDUCE-4290.patch
        8 kB
        Devaraj K

        Activity

        Hide
        Devaraj K added a comment -

        JobClient.getAllJobs() API is giving the status as SUCCEEDED even if the job is failed. While converting from application report to Job Status it is considering only the yarn application state. If the application state is finished and final status is failed, it is giving the job status as SUCCEEDED by considering only the application state.

        I have attached patch to address this, if the yarn application status is finished and final status is succeeded then only giving the job status as succeeded.

        Show
        Devaraj K added a comment - JobClient.getAllJobs() API is giving the status as SUCCEEDED even if the job is failed. While converting from application report to Job Status it is considering only the yarn application state. If the application state is finished and final status is failed, it is giving the job status as SUCCEEDED by considering only the application state. I have attached patch to address this, if the yarn application status is finished and final status is succeeded then only giving the job status as succeeded.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12531241/MAPREDUCE-4290.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2443//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2443//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531241/MAPREDUCE-4290.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2443//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2443//console This message is automatically generated.
        Hide
        Siddharth Seth added a comment -

        There needs to be a couple additional cases within the FINISHED state - to deal with KILLED/FAILED. Other than that the patch looks good.

        Another problem with the getAllJobs() API - it gets the application list from the RM - which means it's going to convert non MapReduce apps as well. Don't believe there's any good way to differentiate between application types from the RM list.

        Show
        Siddharth Seth added a comment - There needs to be a couple additional cases within the FINISHED state - to deal with KILLED/FAILED. Other than that the patch looks good. Another problem with the getAllJobs() API - it gets the application list from the RM - which means it's going to convert non MapReduce apps as well. Don't believe there's any good way to differentiate between application types from the RM list.
        Hide
        Devaraj K added a comment -

        Thanks a lot Sid for looking into the patch.

        There needs to be a couple additional cases within the FINISHED state - to deal with KILLED/FAILED. Other than that the patch looks good.

        I have updated the patch for KILLED case. FAILED case will go to the next switch case(FAILED).

        Another problem with the getAllJobs() API - it gets the application list from the RM - which means it's going to convert non MapReduce apps as well. Don't believe there's any good way to differentiate between application types from the RM list.

        Yes Sid, there is no better way to differentiate mapred and non-mapred apps. Any way this api will be useful if we have all mapred jobs.

        Show
        Devaraj K added a comment - Thanks a lot Sid for looking into the patch. There needs to be a couple additional cases within the FINISHED state - to deal with KILLED/FAILED. Other than that the patch looks good. I have updated the patch for KILLED case. FAILED case will go to the next switch case(FAILED). Another problem with the getAllJobs() API - it gets the application list from the RM - which means it's going to convert non MapReduce apps as well. Don't believe there's any good way to differentiate between application types from the RM list. Yes Sid, there is no better way to differentiate mapred and non-mapred apps. Any way this api will be useful if we have all mapred jobs.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12532509/MAPREDUCE-4290-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2478//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532509/MAPREDUCE-4290-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2478//console This message is automatically generated.
        Hide
        Siddharth Seth added a comment -

        Re-uploading for jenkins.

        Show
        Siddharth Seth added a comment - Re-uploading for jenkins.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12533115/MAPREDUCE-4290-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 13 warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2503//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2503//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533115/MAPREDUCE-4290-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2503//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2503//console This message is automatically generated.
        Hide
        Siddharth Seth added a comment -

        +1. Looks good. The javadoc warnings are not related.

        Show
        Siddharth Seth added a comment - +1. Looks good. The javadoc warnings are not related.
        Hide
        Siddharth Seth added a comment -

        Committed to trunk and branch-2. Thanks Devaraj!

        Show
        Siddharth Seth added a comment - Committed to trunk and branch-2. Thanks Devaraj!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2454 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2454/)
        MAPREDUCE-4290. Fix Yarn Applicaiton Status to MR JobState conversion. (Contributed by Devaraj K) (Revision 1353684)

        Result = SUCCESS
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353684
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2454 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2454/ ) MAPREDUCE-4290 . Fix Yarn Applicaiton Status to MR JobState conversion. (Contributed by Devaraj K) (Revision 1353684) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353684 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2384 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2384/)
        MAPREDUCE-4290. Fix Yarn Applicaiton Status to MR JobState conversion. (Contributed by Devaraj K) (Revision 1353684)

        Result = SUCCESS
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353684
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2384 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2384/ ) MAPREDUCE-4290 . Fix Yarn Applicaiton Status to MR JobState conversion. (Contributed by Devaraj K) (Revision 1353684) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353684 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2402 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2402/)
        MAPREDUCE-4290. Fix Yarn Applicaiton Status to MR JobState conversion. (Contributed by Devaraj K) (Revision 1353684)

        Result = FAILURE
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353684
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2402 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2402/ ) MAPREDUCE-4290 . Fix Yarn Applicaiton Status to MR JobState conversion. (Contributed by Devaraj K) (Revision 1353684) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353684 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1088 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1088/)
        MAPREDUCE-4290. Fix Yarn Applicaiton Status to MR JobState conversion. (Contributed by Devaraj K) (Revision 1353684)

        Result = FAILURE
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353684
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1088 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1088/ ) MAPREDUCE-4290 . Fix Yarn Applicaiton Status to MR JobState conversion. (Contributed by Devaraj K) (Revision 1353684) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353684 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1121 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1121/)
        MAPREDUCE-4290. Fix Yarn Applicaiton Status to MR JobState conversion. (Contributed by Devaraj K) (Revision 1353684)

        Result = FAILURE
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353684
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1121 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1121/ ) MAPREDUCE-4290 . Fix Yarn Applicaiton Status to MR JobState conversion. (Contributed by Devaraj K) (Revision 1353684) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353684 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #304 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/304/)
        MAPREDUCE-4290. Fix Yarn Applicaiton Status to MR JobState conversion. (Contributed by Devaraj K) (Revision 1356919)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356919
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #304 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/304/ ) MAPREDUCE-4290 . Fix Yarn Applicaiton Status to MR JobState conversion. (Contributed by Devaraj K) (Revision 1356919) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356919 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java

          People

          • Assignee:
            Devaraj K
            Reporter:
            Nishan Shetty
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development