Description
When an application finishes it should be added to an application summary log for historical purposes. jira MAPREDUCE-2649 is going to start purging applications from RM when certain limits are hit which makes this more critical. We also need to save the information early enough after the app finishes so we don't lose the info if the RM does get restarted.
Attachments
Attachments
- MAPREDUCE-2735.patch
- 15 kB
- Thomas Graves
- MAPREDUCE-2735-v2.patch
- 17 kB
- Thomas Graves
Activity
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12492471/MAPREDUCE-2735.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//console
This message is automatically generated.
the findbugs errors were pre-existing MAPREDUCE-2908.
test-patch output when I run:
+1 overall.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version ) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
Thomas, just to clarify - this is in addition (i.e. on RM) to the MR JobSummary log? JobSummary log should be done by the JobHistory server...
yes this is in addition to any job summary log. This is on the RM when an application completes, it gets recorded here.
Thomas, the patch looks good. Can you please add a descriptive example to log4j.properties so that people know how to configure this on? Thanks.
Thanks Arun. Sure I can add an example, you want it in the one at ./hadoop-common-project/hadoop-common/src/main/conf/log4j.properties?
added example to common log4j.properties. Hopefully it doesn't complain since its in common and not mapreduce directory.
Integrated in Hadoop-Hdfs-trunk-Commit #900 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/900/)
MAPREDUCE-2735. Add an applications summary log to ResourceManager. Contributed by Thomas Graves.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
Integrated in Hadoop-Mapreduce-trunk-Commit #833 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/833/)
MAPREDUCE-2735. Add an applications summary log to ResourceManager. Contributed by Thomas Graves.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
Integrated in Hadoop-Hdfs-trunk #780 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/780/)
MAPREDUCE-2735. Add an applications summary log to ResourceManager. Contributed by Thomas Graves.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
Integrated in Hadoop-Mapreduce-trunk #804 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/804/)
MAPREDUCE-2735. Add an applications summary log to ResourceManager. Contributed by Thomas Graves.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
The application summary log is formatted the same as the old job summary log. example output is:
2011-08-31 16:02:01,237 INFO resourcemanager.RMAppManager$ApplicationSummary: appId=application_1314806486924_1,name=word count,user=user1,queue=default,state=FINISHED,trackingUrl=99.99.99.99:19888/yarn/job/job_1314806486924_1_1,appMasterHost=99.99.99.99,startTime=1314806497546,finishTime=1314806521231
To configured would be something like this:
{ISO8601}rm.appsummary.logger=INFO,RMSUMMARY
log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=${rm.appsummary.logger}
log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=false
log4j.appender.RMSUMMARY=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RMSUMMARY.File=${hadoop.log.dir}/rm-appsummary.log
log4j.appender.RMSUMMARY.layout=org.apache.log4j.PatternLayout
log4j.appender.RMSUMMARY.layout.ConversionPattern=%d
%p %c
{2}: %m%n
log4j.appender.JSA.DatePattern=.yyyy-MM-dd