It would be useful to track 'first-task-launched' time to job-summary for better reporting.
Common portion of MAPREDUCE-1545
Our reporting team wants this for better wait time calculations in our reporting DB.
Hong Tang mentioned that first tasks are setup tasks and we'd better add firstMapTaskLaunchTime and firstReduceTaskLaunchTime besides the firstTaskLaunchTime while we're at it.
Patch for yahoo 0.20.1xx (y20s) branch.
v3 patch for y20 branch to incorporate feedbacks from HADOOP-6657
v1 patch for trunk as well.
v2 trunk patch added tests to assert precise launch times.
Integrated in Hadoop-Common-trunk-Commit #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/221/)
HADOOP-6657. Add a capitalization method to StringUtils for MAPREDUCE-1545
-1 overall. Here are the results of testing the latest attachment
against trunk revision 937201.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/console
This message is automatically generated.
Is this only for "first task" or are we measuring wait times in pending state for ALL tasks?
Last week, we ran out of reduce slots. I had one job where 150 of my reduce tasks had completed but other 150 were in a pending state (not running) for over 1 hour.
Are we measuring pending time for all tasks or only for first task?
@ciemo, you can find start and finish times of every task in job history. The first task launch times are for the job summary only.
I committed this, eliding the whitespace change to TaskInProgress. Thanks, Luke!
Looks like it's not committed to the y20s branch yet. Updated the y20s branch patch and committing.