Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2735

MR279: finished applications should be added to an application summary log

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.23.0
    • mrv2
    • None

    Description

      When an application finishes it should be added to an application summary log for historical purposes. jira MAPREDUCE-2649 is going to start purging applications from RM when certain limits are hit which makes this more critical. We also need to save the information early enough after the app finishes so we don't lose the info if the RM does get restarted.

      Attachments

        1. MAPREDUCE-2735.patch
          15 kB
          Thomas Graves
        2. MAPREDUCE-2735-v2.patch
          17 kB
          Thomas Graves

        Activity

          tgraves Thomas Graves added a comment -

          The application summary log is formatted the same as the old job summary log. example output is:

          2011-08-31 16:02:01,237 INFO resourcemanager.RMAppManager$ApplicationSummary: appId=application_1314806486924_1,name=word count,user=user1,queue=default,state=FINISHED,trackingUrl=99.99.99.99:19888/yarn/job/job_1314806486924_1_1,appMasterHost=99.99.99.99,startTime=1314806497546,finishTime=1314806521231

          To configured would be something like this:
          rm.appsummary.logger=INFO,RMSUMMARY
          log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=${rm.appsummary.logger}
          log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=false
          log4j.appender.RMSUMMARY=org.apache.log4j.DailyRollingFileAppender
          log4j.appender.RMSUMMARY.File=${hadoop.log.dir}/rm-appsummary.log
          log4j.appender.RMSUMMARY.layout=org.apache.log4j.PatternLayout
          log4j.appender.RMSUMMARY.layout.ConversionPattern=%d

          {ISO8601}

          %p %c

          {2}

          : %m%n
          log4j.appender.JSA.DatePattern=.yyyy-MM-dd

          tgraves Thomas Graves added a comment - The application summary log is formatted the same as the old job summary log. example output is: 2011-08-31 16:02:01,237 INFO resourcemanager.RMAppManager$ApplicationSummary: appId=application_1314806486924_1,name=word count,user=user1,queue=default,state=FINISHED,trackingUrl=99.99.99.99:19888/yarn/job/job_1314806486924_1_1,appMasterHost=99.99.99.99,startTime=1314806497546,finishTime=1314806521231 To configured would be something like this: rm.appsummary.logger=INFO,RMSUMMARY log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=${rm.appsummary.logger} log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=false log4j.appender.RMSUMMARY=org.apache.log4j.DailyRollingFileAppender log4j.appender.RMSUMMARY.File=${hadoop.log.dir}/rm-appsummary.log log4j.appender.RMSUMMARY.layout=org.apache.log4j.PatternLayout log4j.appender.RMSUMMARY.layout.ConversionPattern=%d {ISO8601} %p %c {2} : %m%n log4j.appender.JSA.DatePattern=.yyyy-MM-dd
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12492471/MAPREDUCE-2735.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492471/MAPREDUCE-2735.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//console This message is automatically generated.
          tgraves Thomas Graves added a comment -

          the findbugs errors were pre-existing MAPREDUCE-2908.

          test-patch output when I run:
          +1 overall.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          tgraves Thomas Graves added a comment - the findbugs errors were pre-existing MAPREDUCE-2908 . test-patch output when I run: +1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings.
          acmurthy Arun Murthy added a comment -

          Thomas, just to clarify - this is in addition (i.e. on RM) to the MR JobSummary log? JobSummary log should be done by the JobHistory server...

          acmurthy Arun Murthy added a comment - Thomas, just to clarify - this is in addition (i.e. on RM) to the MR JobSummary log? JobSummary log should be done by the JobHistory server...
          tgraves Thomas Graves added a comment -

          yes this is in addition to any job summary log. This is on the RM when an application completes, it gets recorded here.

          tgraves Thomas Graves added a comment - yes this is in addition to any job summary log. This is on the RM when an application completes, it gets recorded here.
          acmurthy Arun Murthy added a comment -

          Thomas, the patch looks good. Can you please add a descriptive example to log4j.properties so that people know how to configure this on? Thanks.

          acmurthy Arun Murthy added a comment - Thomas, the patch looks good. Can you please add a descriptive example to log4j.properties so that people know how to configure this on? Thanks.
          tgraves Thomas Graves added a comment -

          Thanks Arun. Sure I can add an example, you want it in the one at ./hadoop-common-project/hadoop-common/src/main/conf/log4j.properties?

          tgraves Thomas Graves added a comment - Thanks Arun. Sure I can add an example, you want it in the one at ./hadoop-common-project/hadoop-common/src/main/conf/log4j.properties?
          acmurthy Arun Murthy added a comment -

          Yes, thanks.

          acmurthy Arun Murthy added a comment - Yes, thanks.
          tgraves Thomas Graves added a comment -

          added example to common log4j.properties. Hopefully it doesn't complain since its in common and not mapreduce directory.

          tgraves Thomas Graves added a comment - added example to common log4j.properties. Hopefully it doesn't complain since its in common and not mapreduce directory.
          acmurthy Arun Murthy added a comment -

          I just committed this. Thanks Tom!

          acmurthy Arun Murthy added a comment - I just committed this. Thanks Tom!
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #900 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/900/)
          MAPREDUCE-2735. Add an applications summary log to ResourceManager. Contributed by Thomas Graves.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #900 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/900/ ) MAPREDUCE-2735 . Add an applications summary log to ResourceManager. Contributed by Thomas Graves. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #833 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/833/)
          MAPREDUCE-2735. Add an applications summary log to ResourceManager. Contributed by Thomas Graves.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #833 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/833/ ) MAPREDUCE-2735 . Add an applications summary log to ResourceManager. Contributed by Thomas Graves. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #780 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/780/)
          MAPREDUCE-2735. Add an applications summary log to ResourceManager. Contributed by Thomas Graves.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #780 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/780/ ) MAPREDUCE-2735 . Add an applications summary log to ResourceManager. Contributed by Thomas Graves. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #804 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/804/)
          MAPREDUCE-2735. Add an applications summary log to ResourceManager. Contributed by Thomas Graves.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #804 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/804/ ) MAPREDUCE-2735 . Add an applications summary log to ResourceManager. Contributed by Thomas Graves. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java

          People

            tgraves Thomas Graves
            tgraves Thomas Graves
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: