Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4721

Task startup time in JHS is same as job startup time.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 3.0.0, 2.0.2-alpha
    • Fix Version/s: 2.0.3-alpha, 0.23.5
    • Component/s: jobhistoryserver
    • Labels:
      None

      Description

      As Bobby pointed out in https://issues.apache.org/jira/browse/MAPREDUCE-4711?focusedCommentId=13471696&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13471696

      In the Map and Reduce tasks page, it should print the earliest task attempt launch time as TaskImpl:getLaunchTime() does.

      1. MAPREDUCE-4721.patch
        6 kB
        Ravi Prakash
      2. MAPREDUCE-4721.patch
        4 kB
        Ravi Prakash

        Activity

        Hide
        Ravi Prakash added a comment -

        The same patch applies to branch-0.23, branch-2 and trunk

        Show
        Ravi Prakash added a comment - The same patch applies to branch-0.23, branch-2 and trunk
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12548742/MAPREDUCE-4721.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2924//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2924//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12548742/MAPREDUCE-4721.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2924//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2924//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        The patch looks good, but I think I would prefer to see the default be the task start time from task info. This is what the AM reports for tasks with no task attempts.

        Show
        Robert Joseph Evans added a comment - The patch looks good, but I think I would prefer to see the default be the task start time from task info. This is what the AM reports for tasks with no task attempts.
        Hide
        Ravi Prakash added a comment -

        I would disagree with you there Bobby! If I'm on the task page, as a user, I don't want to have to figure out that a task attempt was never launched by matching with the job start time. I want to see unequivocally in readily apparent letters "N/A" like the great Ravi meant for me to see, so that I know that task attempts were never launched.

        I suggest changing the AM to do the same. I'll upload a patch for that.

        Show
        Ravi Prakash added a comment - I would disagree with you there Bobby! If I'm on the task page, as a user, I don't want to have to figure out that a task attempt was never launched by matching with the job start time. I want to see unequivocally in readily apparent letters "N/A" like the great Ravi meant for me to see, so that I know that task attempts were never launched. I suggest changing the AM to do the same. I'll upload a patch for that.
        Hide
        Ravi Prakash added a comment -

        Allow me to recant what I said! In the AM, when the task startup time is the same as the job startup time, the status says "Scheduled". So its indicative that it has been "Scheduled" at (that job) startup time. However, in a JHS, the task has already either finished, or failed/killed. In case the task attempt was never launched, I want to see N/A. If an attempt was launched (but failed), I want to see the start time of the first attempt.

        Can we please commit the patch as is?

        Show
        Ravi Prakash added a comment - Allow me to recant what I said! In the AM, when the task startup time is the same as the job startup time, the status says "Scheduled". So its indicative that it has been "Scheduled" at (that job) startup time. However, in a JHS, the task has already either finished, or failed/killed. In case the task attempt was never launched, I want to see N/A. If an attempt was launched (but failed), I want to see the start time of the first attempt. Can we please commit the patch as is?
        Hide
        Robert Joseph Evans added a comment -

        I am fine with that for the UI, showing an N/A triggered by the -1, but the web services are returning that value too. We either need to update the documentation to indicate what a -1 means or switch it to do what the AM does.

        Sorry it has taken me so long to respond.

        Show
        Robert Joseph Evans added a comment - I am fine with that for the UI, showing an N/A triggered by the -1, but the web services are returning that value too. We either need to update the documentation to indicate what a -1 means or switch it to do what the AM does. Sorry it has taken me so long to respond.
        Hide
        Ravi Prakash added a comment -

        Thanks for pointing that out Bobby! I had no idea that even existed!

        Show
        Ravi Prakash added a comment - Thanks for pointing that out Bobby! I had no idea that even existed!
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12549485/MAPREDUCE-4721.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2936//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2936//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12549485/MAPREDUCE-4721.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2936//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2936//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        Looks good +1. I'll check this in.

        Show
        Robert Joseph Evans added a comment - Looks good +1. I'll check this in.
        Hide
        Robert Joseph Evans added a comment -

        Thanks again Ravi.

        Show
        Robert Joseph Evans added a comment - Thanks again Ravi.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #2881 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2881/)
        MAPREDUCE-4721. Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #2881 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2881/ ) MAPREDUCE-4721 . Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #7 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/7/)
        MAPREDUCE-4721. Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #7 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/7/ ) MAPREDUCE-4721 . Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #408 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/408/)
        svn merge -c 1399289 FIXES: MAPREDUCE-4721. Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399293)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399293
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java
        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #408 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/408/ ) svn merge -c 1399289 FIXES: MAPREDUCE-4721 . Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399293) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399293 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1199 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1199/)
        MAPREDUCE-4721. Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1199 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1199/ ) MAPREDUCE-4721 . Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1229 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1229/)
        MAPREDUCE-4721. Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1229 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1229/ ) MAPREDUCE-4721 . Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm

          People

          • Assignee:
            Ravi Prakash
            Reporter:
            Ravi Prakash
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development