Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2735

MR279: finished applications should be added to an application summary log

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None

      Description

      When an application finishes it should be added to an application summary log for historical purposes. jira MAPREDUCE-2649 is going to start purging applications from RM when certain limits are hit which makes this more critical. We also need to save the information early enough after the app finishes so we don't lose the info if the RM does get restarted.

      1. MAPREDUCE-2735.patch
        15 kB
        Thomas Graves
      2. MAPREDUCE-2735-v2.patch
        17 kB
        Thomas Graves

        Activity

        Thomas Graves created issue -
        Thomas Graves made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-2735.patch [ 12492471 ]
        Hide
        Thomas Graves added a comment -

        The application summary log is formatted the same as the old job summary log. example output is:

        2011-08-31 16:02:01,237 INFO resourcemanager.RMAppManager$ApplicationSummary: appId=application_1314806486924_1,name=word count,user=user1,queue=default,state=FINISHED,trackingUrl=99.99.99.99:19888/yarn/job/job_1314806486924_1_1,appMasterHost=99.99.99.99,startTime=1314806497546,finishTime=1314806521231

        To configured would be something like this:
        rm.appsummary.logger=INFO,RMSUMMARY
        log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=$

        {rm.appsummary.logger}

        log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=false
        log4j.appender.RMSUMMARY=org.apache.log4j.DailyRollingFileAppender
        log4j.appender.RMSUMMARY.File=$

        {hadoop.log.dir}

        /rm-appsummary.log
        log4j.appender.RMSUMMARY.layout=org.apache.log4j.PatternLayout
        log4j.appender.RMSUMMARY.layout.ConversionPattern=%d

        {ISO8601}

        %p %c

        {2}

        : %m%n
        log4j.appender.JSA.DatePattern=.yyyy-MM-dd

        Show
        Thomas Graves added a comment - The application summary log is formatted the same as the old job summary log. example output is: 2011-08-31 16:02:01,237 INFO resourcemanager.RMAppManager$ApplicationSummary: appId=application_1314806486924_1,name=word count,user=user1,queue=default,state=FINISHED,trackingUrl=99.99.99.99:19888/yarn/job/job_1314806486924_1_1,appMasterHost=99.99.99.99,startTime=1314806497546,finishTime=1314806521231 To configured would be something like this: rm.appsummary.logger=INFO,RMSUMMARY log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=$ {rm.appsummary.logger} log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=false log4j.appender.RMSUMMARY=org.apache.log4j.DailyRollingFileAppender log4j.appender.RMSUMMARY.File=$ {hadoop.log.dir} /rm-appsummary.log log4j.appender.RMSUMMARY.layout=org.apache.log4j.PatternLayout log4j.appender.RMSUMMARY.layout.ConversionPattern=%d {ISO8601} %p %c {2} : %m%n log4j.appender.JSA.DatePattern=.yyyy-MM-dd
        Thomas Graves made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Thomas Graves made changes -
        Fix Version/s 0.23.0 [ 12315570 ]
        Fix Version/s 0.24.0 [ 12317654 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12492471/MAPREDUCE-2735.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492471/MAPREDUCE-2735.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/571//console This message is automatically generated.
        Hide
        Thomas Graves added a comment -

        the findbugs errors were pre-existing MAPREDUCE-2908.

        test-patch output when I run:
        +1 overall.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Show
        Thomas Graves added a comment - the findbugs errors were pre-existing MAPREDUCE-2908 . test-patch output when I run: +1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Hide
        Arun C Murthy added a comment -

        Thomas, just to clarify - this is in addition (i.e. on RM) to the MR JobSummary log? JobSummary log should be done by the JobHistory server...

        Show
        Arun C Murthy added a comment - Thomas, just to clarify - this is in addition (i.e. on RM) to the MR JobSummary log? JobSummary log should be done by the JobHistory server...
        Hide
        Thomas Graves added a comment -

        yes this is in addition to any job summary log. This is on the RM when an application completes, it gets recorded here.

        Show
        Thomas Graves added a comment - yes this is in addition to any job summary log. This is on the RM when an application completes, it gets recorded here.
        Hide
        Arun C Murthy added a comment -

        Thomas, the patch looks good. Can you please add a descriptive example to log4j.properties so that people know how to configure this on? Thanks.

        Show
        Arun C Murthy added a comment - Thomas, the patch looks good. Can you please add a descriptive example to log4j.properties so that people know how to configure this on? Thanks.
        Hide
        Thomas Graves added a comment -

        Thanks Arun. Sure I can add an example, you want it in the one at ./hadoop-common-project/hadoop-common/src/main/conf/log4j.properties?

        Show
        Thomas Graves added a comment - Thanks Arun. Sure I can add an example, you want it in the one at ./hadoop-common-project/hadoop-common/src/main/conf/log4j.properties?
        Hide
        Arun C Murthy added a comment -

        Yes, thanks.

        Show
        Arun C Murthy added a comment - Yes, thanks.
        Hide
        Thomas Graves added a comment -

        added example to common log4j.properties. Hopefully it doesn't complain since its in common and not mapreduce directory.

        Show
        Thomas Graves added a comment - added example to common log4j.properties. Hopefully it doesn't complain since its in common and not mapreduce directory.
        Thomas Graves made changes -
        Attachment MAPREDUCE-2735-v2.patch [ 12492672 ]
        Hide
        Arun C Murthy added a comment -

        I just committed this. Thanks Tom!

        Show
        Arun C Murthy added a comment - I just committed this. Thanks Tom!
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.24.0 [ 12317654 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #900 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/900/)
        MAPREDUCE-2735. Add an applications summary log to ResourceManager. Contributed by Thomas Graves.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #900 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/900/ ) MAPREDUCE-2735 . Add an applications summary log to ResourceManager. Contributed by Thomas Graves. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #833 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/833/)
        MAPREDUCE-2735. Add an applications summary log to ResourceManager. Contributed by Thomas Graves.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #833 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/833/ ) MAPREDUCE-2735 . Add an applications summary log to ResourceManager. Contributed by Thomas Graves. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #780 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/780/)
        MAPREDUCE-2735. Add an applications summary log to ResourceManager. Contributed by Thomas Graves.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #780 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/780/ ) MAPREDUCE-2735 . Add an applications summary log to ResourceManager. Contributed by Thomas Graves. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #804 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/804/)
        MAPREDUCE-2735. Add an applications summary log to ResourceManager. Contributed by Thomas Graves.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #804 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/804/ ) MAPREDUCE-2735 . Add an applications summary log to ResourceManager. Contributed by Thomas Graves. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164301 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Thomas Graves
            Reporter:
            Thomas Graves
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development