Hadoop Common
  1. Hadoop Common
  2. HADOOP-7633

log4j.properties should be added to the hadoop conf on deploy

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.205.0, 0.23.0
    • Component/s: conf
    • Labels:
      None

      Description

      currently the log4j properties are not present in the hadoop conf dir. We should add them so that log rotation happens appropriately and also define other logs that hadoop can generate for example the audit and the auth logs as well as the mapred summary logs etc.

      1. HADOOP-7633.patch
        6 kB
        Eric Yang
      2. HADOOP-7633-trunk.patch
        9 kB
        Eric Yang

        Issue Links

          Activity

          Hide
          Arpit Gupta added a comment -

          hdfs audit and mr audit logs are not part of the defaults. They can be defined as such....

          1. hdfs audit logging

          hdfs.audit.logger=INFO,console
          log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=$

          {hdfs.audit.logger}

          log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false
          log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender
          log4j.appender.DRFAAUDIT.File=$

          {hadoop.log.dir}/hdfs-audit.log
          log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout
          log4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
          log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd

          # mapred audit logging

          mapred.audit.logger=INFO,console
          log4j.logger.org.apache.hadoop.mapred.AuditLogger=${mapred.audit.logger}
          log4j.additivity.org.apache.hadoop.mapred.AuditLogger=false
          log4j.appender.MRAUDIT=org.apache.log4j.DailyRollingFileAppender
          log4j.appender.MRAUDIT.File=${hadoop.log.dir}

          /mapred-audit.log
          log4j.appender.MRAUDIT.layout=org.apache.log4j.PatternLayout
          log4j.appender.MRAUDIT.layout.ConversionPattern=%d

          {ISO8601}

          %p %c

          {2}

          : %m%n
          log4j.appender.MRAUDIT.DatePattern=.yyyy-MM-dd

          We should add the audit logs as part of default confs.

          Show
          Arpit Gupta added a comment - hdfs audit and mr audit logs are not part of the defaults. They can be defined as such.... hdfs audit logging hdfs.audit.logger=INFO,console log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=$ {hdfs.audit.logger} log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFAAUDIT.File=$ {hadoop.log.dir}/hdfs-audit.log log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout log4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd # mapred audit logging mapred.audit.logger=INFO,console log4j.logger.org.apache.hadoop.mapred.AuditLogger=${mapred.audit.logger} log4j.additivity.org.apache.hadoop.mapred.AuditLogger=false log4j.appender.MRAUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.MRAUDIT.File=${hadoop.log.dir} /mapred-audit.log log4j.appender.MRAUDIT.layout=org.apache.log4j.PatternLayout log4j.appender.MRAUDIT.layout.ConversionPattern=%d {ISO8601} %p %c {2} : %m%n log4j.appender.MRAUDIT.DatePattern=.yyyy-MM-dd We should add the audit logs as part of default confs.
          Hide
          Eric Yang added a comment -

          Added log4j.prooperties as part of setup script.

          Show
          Eric Yang added a comment - Added log4j.prooperties as part of setup script.
          Hide
          Eric Yang added a comment -

          Same patch for trunk.

          Show
          Eric Yang added a comment - Same patch for trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12494325/HADOOP-7633-trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 10 release audit warnings (more than the trunk's current 0 warnings).

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/179//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/179//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/179//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494325/HADOOP-7633-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 10 release audit warnings (more than the trunk's current 0 warnings). +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/179//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/179//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/179//console This message is automatically generated.
          Hide
          Steve Loughran added a comment -

          Assuming those warnings are unrelated and they'd have to be I'm +1 for this.

          Logging at the log4 settings file I can see that it differs from my normal files by not listing threads, with a [%t] field. This is pretty low cost, and if you give your threads useful names (NN-worker-1), is very good for debugging race conditions and extracting traces from specific threads in a noisy log. That should be a separate JIRA though

          Show
          Steve Loughran added a comment - Assuming those warnings are unrelated and they'd have to be I'm +1 for this. Logging at the log4 settings file I can see that it differs from my normal files by not listing threads, with a [%t] field. This is pretty low cost, and if you give your threads useful names (NN-worker-1), is very good for debugging race conditions and extracting traces from specific threads in a noisy log. That should be a separate JIRA though
          Hide
          Devaraj Das added a comment -

          Committed this. Thanks, Eric!

          Show
          Devaraj Das added a comment - Committed this. Thanks, Eric!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #905 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/905/)
          HADOOP-7633. Adds log4j.properties to the hadoop-conf dir on deploy. Contributed by Eric Yang.

          ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171909
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #905 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/905/ ) HADOOP-7633 . Adds log4j.properties to the hadoop-conf dir on deploy. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171909 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #982 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/982/)
          HADOOP-7633. Adds log4j.properties to the hadoop-conf dir on deploy. Contributed by Eric Yang.

          ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171909
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #982 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/982/ ) HADOOP-7633 . Adds log4j.properties to the hadoop-conf dir on deploy. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171909 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #920 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/920/)
          HADOOP-7633. Adds log4j.properties to the hadoop-conf dir on deploy. Contributed by Eric Yang.

          ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171909
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #920 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/920/ ) HADOOP-7633 . Adds log4j.properties to the hadoop-conf dir on deploy. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171909 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #803 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/803/)
          HADOOP-7633. Adds log4j.properties to the hadoop-conf dir on deploy. Contributed by Eric Yang.

          ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171909
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #803 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/803/ ) HADOOP-7633 . Adds log4j.properties to the hadoop-conf dir on deploy. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171909 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #833 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/833/)
          HADOOP-7633. Adds log4j.properties to the hadoop-conf dir on deploy. Contributed by Eric Yang.

          ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171909
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #833 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/833/ ) HADOOP-7633 . Adds log4j.properties to the hadoop-conf dir on deploy. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171909 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #9 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/9/)
          Merge -r 1171908:1171909 from trunk onto branch-0.23. Fixes HADOOP-7633.

          ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171911
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #9 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/9/ ) Merge -r 1171908:1171909 from trunk onto branch-0.23. Fixes HADOOP-7633 . ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171911 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #11 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/11/)
          Merge -r 1171908:1171909 from trunk onto branch-0.23. Fixes HADOOP-7633.

          ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171911
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #11 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/11/ ) Merge -r 1171908:1171909 from trunk onto branch-0.23. Fixes HADOOP-7633 . ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171911 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #922 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/922/)
          HADOOP-7633. Missed this file in previous commit.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173411
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #922 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/922/ ) HADOOP-7633 . Missed this file in previous commit. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173411 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #999 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/999/)
          HADOOP-7633. Missed this file in previous commit.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173411
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #999 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/999/ ) HADOOP-7633 . Missed this file in previous commit. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173411 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Hide
          Matt Foley added a comment -

          File log4j.properties was left out of the commit to trunk and v0.23 branches.

          Show
          Matt Foley added a comment - File log4j.properties was left out of the commit to trunk and v0.23 branches.
          Hide
          Matt Foley added a comment -

          Committed to trunk and v0.23.

          Show
          Matt Foley added a comment - Committed to trunk and v0.23.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #939 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/939/)
          HADOOP-7633. Missed this file in previous commit.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173411
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #939 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/939/ ) HADOOP-7633 . Missed this file in previous commit. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173411 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #15 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/15/)
          HADOOP-7633. Missed this file in previous commit.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173412
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #15 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/15/ ) HADOOP-7633 . Missed this file in previous commit. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173412 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #837 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/837/)
          HADOOP-7633. Missed this file in previous commit.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173411
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #837 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/837/ ) HADOOP-7633 . Missed this file in previous commit. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173411 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #807 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/807/)
          HADOOP-7633. Missed this file in previous commit.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173411
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #807 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/807/ ) HADOOP-7633 . Missed this file in previous commit. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1173411 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Hide
          Matt Foley added a comment -

          Closed upon release of 0.20.205.0

          Show
          Matt Foley added a comment - Closed upon release of 0.20.205.0
          Hide
          Eli Collins added a comment -

          This change enabled audit logging by default for the packaging log4j.properties but not the normal one. fyi I'm disabling audit logging by default in HADOOP-8316 for 2.x

          Show
          Eli Collins added a comment - This change enabled audit logging by default for the packaging log4j.properties but not the normal one. fyi I'm disabling audit logging by default in HADOOP-8316 for 2.x

            People

            • Assignee:
              Eric Yang
              Reporter:
              Arpit Gupta
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development