Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-277

Job history counters should be avaible on the UI.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.21.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Modifies job history parser and Web UI to display counters

      Description

      Job history is logging counters. But they are not visible on the UI.
      Job history parser and UI should be modified to view counters.

      1. HADOOP-3200-20080915.1.txt
        13 kB
        Vinod Kumar Vavilapalli
      2. mapred-277-v1.2.patch
        11 kB
        Amar Kamat
      3. mapred-277-v1.4.patch
        11 kB
        Amar Kamat
      4. mapred-277.patch
        24 kB
        Jothi Padmanabhan
      5. mapred-277-v2.patch
        25 kB
        Jothi Padmanabhan
      6. patch-277-0.20.txt
        16 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Hide
          Vinod Kumar Vavilapalli added a comment -

          Putting up a patch for review. This assumes that JobHistory is written by JT such that counters are recoverable(HADOOP-3970).

          Show
          Vinod Kumar Vavilapalli added a comment - Putting up a patch for review. This assumes that JobHistory is written by JT such that counters are recoverable( HADOOP-3970 ).
          Hide
          Vinod Kumar Vavilapalli added a comment -

          HADOOP-3245 will address incorporating HADOOP-3970 into JobHistory written by JT. Making it a blocker for this JIRA.

          Show
          Vinod Kumar Vavilapalli added a comment - HADOOP-3245 will address incorporating HADOOP-3970 into JobHistory written by JT. Making it a blocker for this JIRA.
          Hide
          Amar Kamat added a comment -

          Few comments
          1) I think you could use the default job history parser and retrieve counters inline instead of writing a new listener which will rescan the whole history file.
          2) There is a piece of code in TestCounters that does counter comparison. You can reuse that to check counter's content equality.
          3) JobTracker.getJobCounters() can be used to get the job level counters instead of getting the counters from the task reports and adding/merging them.

          Show
          Amar Kamat added a comment - Few comments 1) I think you could use the default job history parser and retrieve counters inline instead of writing a new listener which will rescan the whole history file. 2) There is a piece of code in TestCounters that does counter comparison. You can reuse that to check counter's content equality. 3) JobTracker.getJobCounters() can be used to get the job level counters instead of getting the counters from the task reports and adding/merging them.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          3) JobTracker.getJobCounters() can be used to get the job level counters instead of getting the counters from the task reports and adding/merging them.

          I am trying to get map-tasks counters and reduce-tasks counters separately for displaying them on jobdetailshistory.jsp, just similar to jobdetails.jsp. And for getting them separately, I can't see any other way than task-reports.

          1) I think you could use the default job history parser and retrieve counters inline instead of writing a new listener which will rescan the whole history file.

          May be we could just change parseHistoryFromFs to accept multiple listeners instead of a single listener. Having a separate listener for each type of information is better IMHO, anyone can just ask for whatever information they want, and storing in memory only what is absolutely needed. This way, we can also remove (repetitve) history parsing for getting information sought by other filters that are currently used in jobdetailshistory.

          Show
          Vinod Kumar Vavilapalli added a comment - 3) JobTracker.getJobCounters() can be used to get the job level counters instead of getting the counters from the task reports and adding/merging them. I am trying to get map-tasks counters and reduce-tasks counters separately for displaying them on jobdetailshistory.jsp, just similar to jobdetails.jsp. And for getting them separately, I can't see any other way than task-reports. 1) I think you could use the default job history parser and retrieve counters inline instead of writing a new listener which will rescan the whole history file. May be we could just change parseHistoryFromFs to accept multiple listeners instead of a single listener. Having a separate listener for each type of information is better IMHO, anyone can just ask for whatever information they want, and storing in memory only what is absolutely needed. This way, we can also remove (repetitve) history parsing for getting information sought by other filters that are currently used in jobdetailshistory.
          Hide
          Amar Kamat added a comment -

          I am trying to get map-tasks counters and reduce-tasks counters separately

          I think comparing the final job counter should suffice. Job counter encompasses all the maps and reduce counters. Is there any specific reason why you are matching them separately?

          May be we could just change parseHistoryFromFs to accept multiple listeners

          I dont see any need for a separate listener, atleast in this jira. I think we should simply recover the counter wherever the counter is expected (like TASK_FINISH) and add it to the global job counter. The only change required will be

          Counters jobCounter = new Counters();
          ...
          ...
          // On a task/tip event
          Counters newCounter = Counters.fromEscapedCompactString(task.get(Keys.COUNTERS));
          jobCounter.incrAllCounters(newCounter);
          

          Introducing new code just for this is probably not worth it. I think we should reuse the available listener. I agree that we should support multiple listeners but it makes sense when the listeners are having separate, well defined functionalities. in this case one would be a subpart of other, logically.

          Show
          Amar Kamat added a comment - I am trying to get map-tasks counters and reduce-tasks counters separately I think comparing the final job counter should suffice. Job counter encompasses all the maps and reduce counters. Is there any specific reason why you are matching them separately? May be we could just change parseHistoryFromFs to accept multiple listeners I dont see any need for a separate listener, atleast in this jira. I think we should simply recover the counter wherever the counter is expected (like TASK_FINISH ) and add it to the global job counter. The only change required will be Counters jobCounter = new Counters(); ... ... // On a task/tip event Counters newCounter = Counters.fromEscapedCompactString(task.get(Keys.COUNTERS)); jobCounter.incrAllCounters(newCounter); Introducing new code just for this is probably not worth it. I think we should reuse the available listener. I agree that we should support multiple listeners but it makes sense when the listeners are having separate, well defined functionalities. in this case one would be a subpart of other, logically.
          Hide
          Amar Kamat added a comment -

          Attaching a patch that adds counters to jobhistory webui. Result of test-patch
          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
          [exec] Please justify why no new tests are needed for this patch.
          [exec] Also please list what manual steps were performed to verify this patch.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          Manually tested the patch and works as expected.

          Show
          Amar Kamat added a comment - Attaching a patch that adds counters to jobhistory webui. Result of test-patch [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Manually tested the patch and works as expected.
          Hide
          Jothi Padmanabhan added a comment -

          I think we should take this up after MAPREDUCE-157 gets committed as that jira changes the Parser code

          Show
          Jothi Padmanabhan added a comment - I think we should take this up after MAPREDUCE-157 gets committed as that jira changes the Parser code
          Hide
          Amar Kamat added a comment -

          Jothi, I think we can get this in sooner. Also the changes to be done in MAPREDUCE-277 w.r.t MAPREDUCE-157 and vice versa should be minimal. So atleast lets go ahead with review and testing.

          Show
          Amar Kamat added a comment - Jothi, I think we can get this in sooner. Also the changes to be done in MAPREDUCE-277 w.r.t MAPREDUCE-157 and vice versa should be minimal. So atleast lets go ahead with review and testing.
          Hide
          Amar Kamat added a comment -

          Attaching a patch that fixes counter count. Result of test-patch
          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
          [exec] Please justify why no new tests are needed for this patch.
          [exec] Also please list what manual steps were performed to verify this patch.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          Show
          Amar Kamat added a comment - Attaching a patch that fixes counter count. Result of test-patch [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Jothi Padmanabhan added a comment -

          Actually, we should fix the command line HistoryViewer to display counters as a part of this jira as well.

          Show
          Jothi Padmanabhan added a comment - Actually, we should fix the command line HistoryViewer to display counters as a part of this jira as well.
          Hide
          Amar Kamat added a comment -

          +1 Good catch. Will upload a new patch for that.
          @Jothi, Amareshwari : Any review comments?

          Show
          Amar Kamat added a comment - +1 Good catch. Will upload a new patch for that. @Jothi, Amareshwari : Any review comments?
          Hide
          Jothi Padmanabhan added a comment -

          Patch updated to the latest trunk.
          Also modified HistoryViewer to display counters.
          Cleaned up the JSP related changes from Amar's latest patch a little bit.

          Show
          Jothi Padmanabhan added a comment - Patch updated to the latest trunk. Also modified HistoryViewer to display counters. Cleaned up the JSP related changes from Amar's latest patch a little bit.
          Hide
          Jothi Padmanabhan added a comment -

          Modified the previous patch to handle null counters. Also, it sets correct finish times for killed and failed attempts while parsing.

          TestPatch passed. Running ant tests now

          Show
          Jothi Padmanabhan added a comment - Modified the previous patch to handle null counters. Also, it sets correct finish times for killed and failed attempts while parsing. TestPatch passed. Running ant tests now
          Hide
          Iyappan Srinivasan added a comment -

          +1 for QA

          1) Checked job for counters and if job and task numbers are displayed correctly.
          2) Checked when jobs reach 100, still the job and task numbers are displayed correctly without any erros.
          3) When a job is killed, counter info are not displayed and no exception gets thrown in task tracker logs.
          4) When JT restat happens, when some jobs are running and some jobs are waiting, still the jobs get resubmitted and
          then shows the counter information properly.
          5) When JT restart happens, the previously existing job history files, still shows the couner information properly.
          6) When a task attempt is killed/ failed, and the overall job fails, still the coutner info is shown
          7) When a task attempt is killed/ failed, and the overall job passes, still the coutner info is shown.
          8) Checked the counter info with the real info int eh done directory. it still shows it correctly
          9) Did job -history "hdfs://job" - dfs. It shows the history information correctly
          10) Did job -history with "file:///job" - local directory. It shows the history information correctly

          Show
          Iyappan Srinivasan added a comment - +1 for QA 1) Checked job for counters and if job and task numbers are displayed correctly. 2) Checked when jobs reach 100, still the job and task numbers are displayed correctly without any erros. 3) When a job is killed, counter info are not displayed and no exception gets thrown in task tracker logs. 4) When JT restat happens, when some jobs are running and some jobs are waiting, still the jobs get resubmitted and then shows the counter information properly. 5) When JT restart happens, the previously existing job history files, still shows the couner information properly. 6) When a task attempt is killed/ failed, and the overall job fails, still the coutner info is shown 7) When a task attempt is killed/ failed, and the overall job passes, still the coutner info is shown. 8) Checked the counter info with the real info int eh done directory. it still shows it correctly 9) Did job -history "hdfs://job" - dfs. It shows the history information correctly 10) Did job -history with "file:///job" - local directory. It shows the history information correctly
          Hide
          Sharad Agarwal added a comment -

          +1

          Show
          Sharad Agarwal added a comment - +1
          Hide
          Jothi Padmanabhan added a comment -

          Trying Hudson again

          Show
          Jothi Padmanabhan added a comment - Trying Hudson again
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12420022/mapred-277-v2.patch
          against trunk revision 816635.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/112/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/112/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/112/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/112/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12420022/mapred-277-v2.patch against trunk revision 816635. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/112/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/112/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/112/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/112/console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          I just committed this. Thanks, Jothi!

          Show
          Devaraj Das added a comment - I just committed this. Thanks, Jothi!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #56 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/56/)
          . Makes job history counters available on the job history viewers. Contributed by Jothi Padmanabhan.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #56 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/56/ ) . Makes job history counters available on the job history viewers. Contributed by Jothi Padmanabhan.
          Hide
          Nigel Daley added a comment -

          No automated tests for this?

          Show
          Nigel Daley added a comment - No automated tests for this?
          Hide
          Jothi Padmanabhan added a comment -

          No automated tests for this?

          This patch just displays the counters that are already available in JobHistory on the UI. The content validation of job history file is already done in TestJobHistory and TestJobHistoryParsing. Testing this particular feature (checking if counters are getting displayed on the UI), is not straight forward, we probably need to scrape the output of the jsps, which we currently do not do for any of the JSPs.

          Show
          Jothi Padmanabhan added a comment - No automated tests for this? This patch just displays the counters that are already available in JobHistory on the UI. The content validation of job history file is already done in TestJobHistory and TestJobHistoryParsing. Testing this particular feature (checking if counters are getting displayed on the UI), is not straight forward, we probably need to scrape the output of the jsps, which we currently do not do for any of the JSPs.
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch for yahoo distribution 0.20

          Show
          Amareshwari Sriramadasu added a comment - Patch for yahoo distribution 0.20
          Hide
          Iyappan Srinivasan added a comment -

          For the patch-277-0.20.txt,

          +1 for QA

          1) Checked job for counters and if job and task numbers are displayed correctly.
          2) Checked when jobs reach 100, still the job and task numbers are displayed correctly without any erros.
          3) When a job is killed, counter info are not displayed and no exception gets thrown in task tracker logs.
          4) When JT restat happens, when some jobs are running and some jobs are waiting, still the jobs get resubmitted and
          then shows the counter information properly.
          5) When JT restart happens, the previously existing job history files, still shows the couner information properly.
          6) When a task attempt is killed/ failed, and the overall job fails, still the counter info is shown
          7) When a task attempt is killed/ failed, and the overall job passes, still the counter info is shown.
          8) Checked the counter info with the real info in the done directory. it still shows it correctly.
          9) Did job -history "<path of job output directory" - It works

          Show
          Iyappan Srinivasan added a comment - For the patch-277-0.20.txt, +1 for QA 1) Checked job for counters and if job and task numbers are displayed correctly. 2) Checked when jobs reach 100, still the job and task numbers are displayed correctly without any erros. 3) When a job is killed, counter info are not displayed and no exception gets thrown in task tracker logs. 4) When JT restat happens, when some jobs are running and some jobs are waiting, still the jobs get resubmitted and then shows the counter information properly. 5) When JT restart happens, the previously existing job history files, still shows the couner information properly. 6) When a task attempt is killed/ failed, and the overall job fails, still the counter info is shown 7) When a task attempt is killed/ failed, and the overall job passes, still the counter info is shown. 8) Checked the counter info with the real info in the done directory. it still shows it correctly. 9) Did job -history "<path of job output directory" - It works
          Hide
          Jothi Padmanabhan added a comment -

          patch-277-0.20.txt looks good to me.

          Show
          Jothi Padmanabhan added a comment - patch-277-0.20.txt looks good to me.

            People

            • Assignee:
              Jothi Padmanabhan
              Reporter:
              Amareshwari Sriramadasu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development