Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4721

Task startup time in JHS is same as job startup time.

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.23.3, 2.0.2-alpha, 3.0.0-alpha1
    • 2.0.3-alpha, 0.23.5
    • jobhistoryserver
    • None

    Description

      As Bobby pointed out in https://issues.apache.org/jira/browse/MAPREDUCE-4711?focusedCommentId=13471696&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13471696

      In the Map and Reduce tasks page, it should print the earliest task attempt launch time as TaskImpl:getLaunchTime() does.

      Attachments

        1. MAPREDUCE-4721.patch
          4 kB
          Ravi Prakash
        2. MAPREDUCE-4721.patch
          6 kB
          Ravi Prakash

        Activity

          raviprak Ravi Prakash added a comment -

          The same patch applies to branch-0.23, branch-2 and trunk

          raviprak Ravi Prakash added a comment - The same patch applies to branch-0.23, branch-2 and trunk
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12548742/MAPREDUCE-4721.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2924//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2924//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12548742/MAPREDUCE-4721.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2924//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2924//console This message is automatically generated.

          The patch looks good, but I think I would prefer to see the default be the task start time from task info. This is what the AM reports for tasks with no task attempts.

          revans2 Robert Joseph Evans added a comment - The patch looks good, but I think I would prefer to see the default be the task start time from task info. This is what the AM reports for tasks with no task attempts.
          raviprak Ravi Prakash added a comment -

          I would disagree with you there Bobby! If I'm on the task page, as a user, I don't want to have to figure out that a task attempt was never launched by matching with the job start time. I want to see unequivocally in readily apparent letters "N/A" like the great Ravi meant for me to see, so that I know that task attempts were never launched.

          I suggest changing the AM to do the same. I'll upload a patch for that.

          raviprak Ravi Prakash added a comment - I would disagree with you there Bobby! If I'm on the task page, as a user, I don't want to have to figure out that a task attempt was never launched by matching with the job start time. I want to see unequivocally in readily apparent letters "N/A" like the great Ravi meant for me to see, so that I know that task attempts were never launched. I suggest changing the AM to do the same. I'll upload a patch for that.
          raviprak Ravi Prakash added a comment -

          Allow me to recant what I said! In the AM, when the task startup time is the same as the job startup time, the status says "Scheduled". So its indicative that it has been "Scheduled" at (that job) startup time. However, in a JHS, the task has already either finished, or failed/killed. In case the task attempt was never launched, I want to see N/A. If an attempt was launched (but failed), I want to see the start time of the first attempt.

          Can we please commit the patch as is?

          raviprak Ravi Prakash added a comment - Allow me to recant what I said! In the AM, when the task startup time is the same as the job startup time, the status says "Scheduled". So its indicative that it has been "Scheduled" at (that job) startup time. However, in a JHS, the task has already either finished, or failed/killed. In case the task attempt was never launched, I want to see N/A. If an attempt was launched (but failed), I want to see the start time of the first attempt. Can we please commit the patch as is?

          I am fine with that for the UI, showing an N/A triggered by the -1, but the web services are returning that value too. We either need to update the documentation to indicate what a -1 means or switch it to do what the AM does.

          Sorry it has taken me so long to respond.

          revans2 Robert Joseph Evans added a comment - I am fine with that for the UI, showing an N/A triggered by the -1, but the web services are returning that value too. We either need to update the documentation to indicate what a -1 means or switch it to do what the AM does. Sorry it has taken me so long to respond.
          raviprak Ravi Prakash added a comment -

          Thanks for pointing that out Bobby! I had no idea that even existed!

          raviprak Ravi Prakash added a comment - Thanks for pointing that out Bobby! I had no idea that even existed!
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12549485/MAPREDUCE-4721.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2936//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2936//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12549485/MAPREDUCE-4721.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2936//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2936//console This message is automatically generated.

          Looks good +1. I'll check this in.

          revans2 Robert Joseph Evans added a comment - Looks good +1. I'll check this in.

          Thanks again Ravi.

          revans2 Robert Joseph Evans added a comment - Thanks again Ravi.
          hudson Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #2881 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2881/)
          MAPREDUCE-4721. Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
          hudson Hudson added a comment - Integrated in Hadoop-trunk-Commit #2881 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2881/ ) MAPREDUCE-4721 . Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
          hudson Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #7 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/7/)
          MAPREDUCE-4721. Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
          hudson Hudson added a comment - Integrated in Hadoop-Yarn-trunk #7 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/7/ ) MAPREDUCE-4721 . Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #408 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/408/)
          svn merge -c 1399289 FIXES: MAPREDUCE-4721. Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399293)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399293
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #408 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/408/ ) svn merge -c 1399289 FIXES: MAPREDUCE-4721 . Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399293) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399293 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1199 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1199/)
          MAPREDUCE-4721. Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1199 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1199/ ) MAPREDUCE-4721 . Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1229 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1229/)
          MAPREDUCE-4721. Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1229 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1229/ ) MAPREDUCE-4721 . Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399289 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm

          People

            raviprak Ravi Prakash
            raviprak Ravi Prakash
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: