Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It would be useful to track 'first-task-launched' time to job-summary for better reporting.

      1. mr-1545-trunk-v1.patch
        10 kB
        Luke Lu
      2. mr-1545-trunk-v2.patch
        11 kB
        Luke Lu
      3. mr-1545-y20s-v1.patch
        9 kB
        Luke Lu
      4. mr-1545-y20s-v2.patch
        10 kB
        Luke Lu
      5. mr-1545-y20s-v3.patch
        11 kB
        Luke Lu
      6. mr-1545-y20s-v4.patch
        11 kB
        Luke Lu

        Issue Links

          Activity

          Arun C Murthy created issue -
          Hide
          eric baldeschwieler added a comment -

          Our reporting team wants this for better wait time calculations in our reporting DB.

          Show
          eric baldeschwieler added a comment - Our reporting team wants this for better wait time calculations in our reporting DB.
          Hide
          Luke Lu added a comment -

          Hong Tang mentioned that first tasks are setup tasks and we'd better add firstMapTaskLaunchTime and firstReduceTaskLaunchTime besides the firstTaskLaunchTime while we're at it.

          Show
          Luke Lu added a comment - Hong Tang mentioned that first tasks are setup tasks and we'd better add firstMapTaskLaunchTime and firstReduceTaskLaunchTime besides the firstTaskLaunchTime while we're at it.
          Hide
          Luke Lu added a comment -

          Patch for yahoo 0.20.1xx (y20s) branch.

          Show
          Luke Lu added a comment - Patch for yahoo 0.20.1xx (y20s) branch.
          Luke Lu made changes -
          Field Original Value New Value
          Attachment mr-1545-y20s-v1.patch [ 12439619 ]
          Luke Lu made changes -
          Assignee Arun C Murthy [ acmurthy ] Luke Lu [ vicaya ]
          Luke Lu made changes -
          Link This issue is blocked by HADOOP-6657 [ HADOOP-6657 ]
          Luke Lu made changes -
          Attachment mr-1545-y20s-v2.patch [ 12439627 ]
          Hide
          Luke Lu added a comment -

          v3 patch for y20 branch to incorporate feedbacks from HADOOP-6657

          Show
          Luke Lu added a comment - v3 patch for y20 branch to incorporate feedbacks from HADOOP-6657
          Luke Lu made changes -
          Attachment mr-1545-y20s-v3.patch [ 12439745 ]
          Luke Lu made changes -
          Attachment mr-1545-trunk-v1.patch [ 12439746 ]
          Hide
          Luke Lu added a comment -

          v1 patch for trunk as well.

          Show
          Luke Lu added a comment - v1 patch for trunk as well.
          Hide
          Luke Lu added a comment -

          v2 trunk patch added tests to assert precise launch times.

          Show
          Luke Lu added a comment - v2 trunk patch added tests to assert precise launch times.
          Luke Lu made changes -
          Attachment mr-1545-trunk-v2.patch [ 12440269 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/221/)
          HADOOP-6657. Add a capitalization method to StringUtils for MAPREDUCE-1545

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/221/ ) HADOOP-6657 . Add a capitalization method to StringUtils for MAPREDUCE-1545
          Luke Lu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12440269/mr-1545-trunk-v2.patch
          against trunk revision 937201.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12440269/mr-1545-trunk-v2.patch against trunk revision 937201. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/console This message is automatically generated.
          Hide
          David Ciemiewicz added a comment -

          Is this only for "first task" or are we measuring wait times in pending state for ALL tasks?

          Last week, we ran out of reduce slots. I had one job where 150 of my reduce tasks had completed but other 150 were in a pending state (not running) for over 1 hour.

          Are we measuring pending time for all tasks or only for first task?

          Show
          David Ciemiewicz added a comment - Is this only for "first task" or are we measuring wait times in pending state for ALL tasks? Last week, we ran out of reduce slots. I had one job where 150 of my reduce tasks had completed but other 150 were in a pending state (not running) for over 1 hour. Are we measuring pending time for all tasks or only for first task?
          Hide
          Luke Lu added a comment -

          @ciemo, you can find start and finish times of every task in job history. The first task launch times are for the job summary only.

          Show
          Luke Lu added a comment - @ciemo, you can find start and finish times of every task in job history. The first task launch times are for the job summary only.
          Hide
          Chris Douglas added a comment -

          +1

          I committed this, eliding the whitespace change to TaskInProgress. Thanks, Luke!

          Show
          Chris Douglas added a comment - +1 I committed this, eliding the whitespace change to TaskInProgress . Thanks, Luke!
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Luke Lu added a comment -

          Looks like it's not committed to the y20s branch yet. Updated the y20s branch patch and committing.

          Show
          Luke Lu added a comment - Looks like it's not committed to the y20s branch yet. Updated the y20s branch patch and committing.
          Luke Lu made changes -
          Attachment mr-1545-y20s-v4.patch [ 12447725 ]
          Konstantin Shvachko made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          53d 1h 41m 1 Luke Lu 24/Apr/10 01:49
          Patch Available Patch Available Resolved Resolved
          43d 3h 30m 1 Chris Douglas 06/Jun/10 05:20
          Resolved Resolved Closed Closed
          554d 1h 59m 1 Konstantin Shvachko 12/Dec/11 06:19

            People

            • Assignee:
              Luke Lu
              Reporter:
              Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development