Hadoop Common
  1. Hadoop Common
  2. HADOOP-1839

Link-ify the Pending/Running/Complete/Killed tasks/task-attempts on jobdetails.jsp

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.15.0
    • Fix Version/s: 0.16.0
    • Component/s: None
    • Labels:
      None

      Description

      Ensuring that we can quickly view the Pending/Running/Complete/Killed tasks and task-attempts from jobdetails.jsp helps to narrow down and hence makes it easier to debug.

      1. HADOOP-1839.patch
        5 kB
        Amar Kamat
      2. HADOOP-1839.patch
        5 kB
        Amar Kamat
      3. HADOOP-1839.patch
        5 kB
        Amar Kamat
      4. HADOOP-1839.patch
        5 kB
        Amar Kamat
      5. HADOOP-1839.patch
        24 kB
        Amar Kamat
      6. HADOOP-1839.patch
        24 kB
        Amar Kamat
      7. HADOOP-1839.patch
        25 kB
        Amar Kamat
      8. HADOOP-1839.patch
        25 kB
        Amar Kamat

        Activity

        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12366153/HADOOP-1839.patch
        against trunk revision r577171.

        @author +1. The patch does not contain any @author tags.

        javadoc -1. The javadoc tool appears to have generated messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/790/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/790/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/790/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/790/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12366153/HADOOP-1839.patch against trunk revision r577171. @author +1. The patch does not contain any @author tags. javadoc -1. The javadoc tool appears to have generated messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/790/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/790/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/790/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/790/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12366174/HADOOP-1839.patch
        against trunk revision r577171.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests -1. The patch failed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/791/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/791/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/791/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/791/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12366174/HADOOP-1839.patch against trunk revision r577171. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/791/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/791/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/791/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/791/console This message is automatically generated.
        Hide
        Enis Soztutar added a comment -

        Thanks Amar, could you please consider the following comments :

        1. You should increment JobSubmissionProtocol#versionID to 5.
        2. Instead of getTaskReports, i think we can keep getMapTaskReports and getReduceTaskReports, since at point in future we may have a separate shuffle task.
        3. i think instead of getTaskReports(String, boolean, boolean, boolean, boolean, boolean, boolean) a better way to do may be
          getTaskReports(String, TaskStatus.State).
        Show
        Enis Soztutar added a comment - Thanks Amar, could you please consider the following comments : You should increment JobSubmissionProtocol#versionID to 5. Instead of getTaskReports, i think we can keep getMapTaskReports and getReduceTaskReports, since at point in future we may have a separate shuffle task. i think instead of getTaskReports(String, boolean, boolean, boolean, boolean, boolean, boolean) a better way to do may be getTaskReports(String, TaskStatus.State).
        Hide
        Amar Kamat added a comment -

        This is because of the DFSAbort error linkhttps://issues.apache.org/jira/browse/HADOOP-1896. Last time it failed because of the javadoc error and passed all the tests.

        Show
        Amar Kamat added a comment - This is because of the DFSAbort error link https://issues.apache.org/jira/browse/HADOOP-1896 . Last time it failed because of the javadoc error and passed all the tests.
        Hide
        Enis Soztutar added a comment -

        update to my previous comments, i think there should be a way to get all the Tasks as well regardless of their status. How abut adding TaskStatus.State to TaskReport class and change getTaskReports to report all the tasks regardless of their statuses, or just the ones we are interested in.

        Show
        Enis Soztutar added a comment - update to my previous comments, i think there should be a way to get all the Tasks as well regardless of their status. How abut adding TaskStatus.State to TaskReport class and change getTaskReports to report all the tasks regardless of their statuses, or just the ones we are interested in.
        Hide
        Amar Kamat added a comment -

        Hey thanks for the comments,
        If we do this how will one fetch reports of all the tasks that are either completed or pending? You require to call it twice and then do the processing. But in the current implT we just call it once and avoid multiple calls.

        Show
        Amar Kamat added a comment - Hey thanks for the comments, If we do this how will one fetch reports of all the tasks that are either completed or pending? You require to call it twice and then do the processing. But in the current implT we just call it once and avoid multiple calls.
        Hide
        Enis Soztutar added a comment -

        If we do this how will one fetch reports of all the tasks that are either completed or pending.

        I think of something like

        getTaskReports(String, EnumSet<TaskStatus.State>) 
        

        the last parameter is an set of TaskStatuses used as a filter to select which TaskStatuses we want. In the JobClient then we may have methods like getMapTaskReports(), getFailedMapTaskReports(), getSuccessfulMapTaskReports(), etc.

        However i would really like TaskReport to have the state of the task as an enum, not a string so that we wont have multiconditional string comparison statements (such as if ("running".equals(state) ).

        Show
        Enis Soztutar added a comment - If we do this how will one fetch reports of all the tasks that are either completed or pending. I think of something like getTaskReports( String , EnumSet<TaskStatus.State>) the last parameter is an set of TaskStatuses used as a filter to select which TaskStatuses we want. In the JobClient then we may have methods like getMapTaskReports(), getFailedMapTaskReports(), getSuccessfulMapTaskReports(), etc. However i would really like TaskReport to have the state of the task as an enum, not a string so that we wont have multiconditional string comparison statements (such as if ("running".equals(state) ).
        Hide
        Arun C Murthy added a comment -

        +1 for getTaskReports(String, EnumSet<TaskStatus.State>).

        Show
        Arun C Murthy added a comment - +1 for getTaskReports(String, EnumSet<TaskStatus.State>) .
        Hide
        Amar Kamat added a comment -

        How about having a function like

         getTaskReports(String, EnumSet<TaskStatus.Phase>, EnumSet<TaskStatus.State>) 

        which looks like a better alternative. So in future an new task 'type' like sort,shuffle,... can also be handled.

        Show
        Amar Kamat added a comment - How about having a function like getTaskReports( String , EnumSet<TaskStatus.Phase>, EnumSet<TaskStatus.State>) which looks like a better alternative. So in future an new task 'type' like sort,shuffle,... can also be handled.
        Hide
        Arun C Murthy added a comment - - edited

        To clarify I'd go for:

        getTaskReports(String jobId, boolean isMap, EnumSet<TaskStatus.State> filter)
        

        Thus we could use this one api to filter any sort of tasks whose reports we are interested in... hopefully that works for everyone.

        Show
        Arun C Murthy added a comment - - edited To clarify I'd go for: getTaskReports(String jobId, boolean isMap, EnumSet<TaskStatus.State> filter) Thus we could use this one api to filter any sort of tasks whose reports we are interested in... hopefully that works for everyone.
        Hide
        Amar Kamat added a comment -

        EnumSet<TaskStatus.State> cannot be used since TIP states are different from task states. Currently there is no concept of TIP state and is computed using its task states. The TIP state information is made available using the following APIs

        isCompeted(), isRunning ...
        

        Hence separate TIP states are required so that some states like PENDING can also be included.
        In TaskInProgress

        enum State {Completed, Running, Killed, Failed, Pending}
        

        might be the solution over this problem.

        Show
        Amar Kamat added a comment - EnumSet<TaskStatus.State> cannot be used since TIP states are different from task states. Currently there is no concept of TIP state and is computed using its task states. The TIP state information is made available using the following APIs isCompeted(), isRunning ... Hence separate TIP states are required so that some states like PENDING can also be included. In TaskInProgress enum State {Completed, Running, Killed, Failed, Pending} might be the solution over this problem.
        Hide
        Enis Soztutar added a comment -

        You are right, TIP state is different than task state and is computed at runtime.
        I think we should add the enum to TaskReport, since we want to keep TIP package private.

        Show
        Enis Soztutar added a comment - You are right, TIP state is different than task state and is computed at runtime. I think we should add the enum to TaskReport, since we want to keep TIP package private.
        Hide
        Enis Soztutar added a comment -

        Amar, could you please :

        1. move TaskInProgress#State to TaskReport#State (since TIP is intended to be package local)
        1. stick with the signature
        getTaskReports(String jobId, boolean isMap, EnumSet<TaskReport.State> filter)
        

        #increment JobSubmissionProtocol#versionID to 5,

        Show
        Enis Soztutar added a comment - Amar, could you please : move TaskInProgress#State to TaskReport#State (since TIP is intended to be package local) stick with the signature getTaskReports( String jobId, boolean isMap, EnumSet<TaskReport.State> filter) #increment JobSubmissionProtocol#versionID to 5,
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12367263/HADOOP-1839.patch
        against trunk revision r582443.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/902/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/902/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/902/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/902/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12367263/HADOOP-1839.patch against trunk revision r582443. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/902/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/902/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/902/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/902/console This message is automatically generated.
        Hide
        Devaraj Das added a comment -

        Sorry, I should have looked at it earlier. Actually, I think we don't need the TIP state and other such changes for this issue. I think we can implement the issue described by doing everything in the JSPs, in the lines of how the links for FAILED/KILLED tasks are created on the job web ui.

        Show
        Devaraj Das added a comment - Sorry, I should have looked at it earlier. Actually, I think we don't need the TIP state and other such changes for this issue. I think we can implement the issue described by doing everything in the JSPs, in the lines of how the links for FAILED/KILLED tasks are created on the job web ui.
        Hide
        Arun C Murthy added a comment -

        Hmm... I can see Devaraj's point and have to agree that it is pointless to have dummy TaskInProgress.State just for jsps. Lets fix that as a separate issue, and go ahead with Devaraj's suggestion.

        Show
        Arun C Murthy added a comment - Hmm... I can see Devaraj's point and have to agree that it is pointless to have dummy TaskInProgress.State just for jsps. Lets fix that as a separate issue, and go ahead with Devaraj's suggestion.
        Hide
        Amar Kamat added a comment -

        I have submitted the patch. While testing it failed on TestLogRolling which I think is not affected by this patch. Comments ?

        Show
        Amar Kamat added a comment - I have submitted the patch. While testing it failed on TestLogRolling which I think is not affected by this patch. Comments ?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12367859/HADOOP-1839.patch
        against trunk revision r585409.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs -1. The patch appears to introduce 2 new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/962/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/962/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/962/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/962/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12367859/HADOOP-1839.patch against trunk revision r585409. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs -1. The patch appears to introduce 2 new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/962/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/962/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/962/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/962/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12367861/HADOOP-1839.patch
        against trunk revision r585409.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests -1. The patch failed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/963/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/963/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/963/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/963/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12367861/HADOOP-1839.patch against trunk revision r585409. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/963/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/963/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/963/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/963/console This message is automatically generated.
        Hide
        Amar Kamat added a comment - - edited

        The test failed on TestDFSShell. I ran ant test on a fresh trunk (with this patch) and it passed the TestDFSShell test .

        Show
        Amar Kamat added a comment - - edited The test failed on TestDFSShell. I ran ant test on a fresh trunk (with this patch) and it passed the TestDFSShell test .
        Hide
        Devaraj Das added a comment -

        Could you please check from http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/963/console which test(s) failed?

        Show
        Devaraj Das added a comment - Could you please check from http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/963/console which test(s) failed?
        Hide
        Arun C Murthy added a comment -

        I just committed this. Thanks, Amar!

        Show
        Arun C Murthy added a comment - I just committed this. Thanks, Amar!
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-Nightly #282 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/282/ )

          People

          • Assignee:
            Amar Kamat
            Reporter:
            Arun C Murthy
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development