Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It would be useful to track 'first-task-launched' time to job-summary for better reporting.

      1. mr-1545-y20s-v4.patch
        11 kB
        Luke Lu
      2. mr-1545-trunk-v2.patch
        11 kB
        Luke Lu
      3. mr-1545-trunk-v1.patch
        10 kB
        Luke Lu
      4. mr-1545-y20s-v3.patch
        11 kB
        Luke Lu
      5. mr-1545-y20s-v2.patch
        10 kB
        Luke Lu
      6. mr-1545-y20s-v1.patch
        9 kB
        Luke Lu

        Issue Links

          Activity

          Hide
          eric baldeschwieler added a comment -

          Our reporting team wants this for better wait time calculations in our reporting DB.

          Show
          eric baldeschwieler added a comment - Our reporting team wants this for better wait time calculations in our reporting DB.
          Hide
          Luke Lu added a comment -

          Hong Tang mentioned that first tasks are setup tasks and we'd better add firstMapTaskLaunchTime and firstReduceTaskLaunchTime besides the firstTaskLaunchTime while we're at it.

          Show
          Luke Lu added a comment - Hong Tang mentioned that first tasks are setup tasks and we'd better add firstMapTaskLaunchTime and firstReduceTaskLaunchTime besides the firstTaskLaunchTime while we're at it.
          Hide
          Luke Lu added a comment -

          Patch for yahoo 0.20.1xx (y20s) branch.

          Show
          Luke Lu added a comment - Patch for yahoo 0.20.1xx (y20s) branch.
          Hide
          Luke Lu added a comment -

          v3 patch for y20 branch to incorporate feedbacks from HADOOP-6657

          Show
          Luke Lu added a comment - v3 patch for y20 branch to incorporate feedbacks from HADOOP-6657
          Hide
          Luke Lu added a comment -

          v1 patch for trunk as well.

          Show
          Luke Lu added a comment - v1 patch for trunk as well.
          Hide
          Luke Lu added a comment -

          v2 trunk patch added tests to assert precise launch times.

          Show
          Luke Lu added a comment - v2 trunk patch added tests to assert precise launch times.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/221/)
          HADOOP-6657. Add a capitalization method to StringUtils for MAPREDUCE-1545

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/221/ ) HADOOP-6657 . Add a capitalization method to StringUtils for MAPREDUCE-1545
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12440269/mr-1545-trunk-v2.patch
          against trunk revision 937201.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12440269/mr-1545-trunk-v2.patch against trunk revision 937201. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/console This message is automatically generated.
          Hide
          David Ciemiewicz added a comment -

          Is this only for "first task" or are we measuring wait times in pending state for ALL tasks?

          Last week, we ran out of reduce slots. I had one job where 150 of my reduce tasks had completed but other 150 were in a pending state (not running) for over 1 hour.

          Are we measuring pending time for all tasks or only for first task?

          Show
          David Ciemiewicz added a comment - Is this only for "first task" or are we measuring wait times in pending state for ALL tasks? Last week, we ran out of reduce slots. I had one job where 150 of my reduce tasks had completed but other 150 were in a pending state (not running) for over 1 hour. Are we measuring pending time for all tasks or only for first task?
          Hide
          Luke Lu added a comment -

          @ciemo, you can find start and finish times of every task in job history. The first task launch times are for the job summary only.

          Show
          Luke Lu added a comment - @ciemo, you can find start and finish times of every task in job history. The first task launch times are for the job summary only.
          Hide
          Chris Douglas added a comment -

          +1

          I committed this, eliding the whitespace change to TaskInProgress. Thanks, Luke!

          Show
          Chris Douglas added a comment - +1 I committed this, eliding the whitespace change to TaskInProgress . Thanks, Luke!
          Hide
          Luke Lu added a comment -

          Looks like it's not committed to the y20s branch yet. Updated the y20s branch patch and committing.

          Show
          Luke Lu added a comment - Looks like it's not committed to the y20s branch yet. Updated the y20s branch patch and committing.

            People

            • Assignee:
              Luke Lu
              Reporter:
              Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development