Hadoop Common
  1. Hadoop Common
  2. HADOOP-8316

Audit logging should be disabled by default

        Details

        • Type: Bug Bug
        • Status: Closed
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: 2.0.0-alpha
        • Fix Version/s: 2.0.2-alpha
        • Component/s: conf
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          HADOOP-7633 made hdfs, mr and security audit logging on by default (INFO level) in log4j.properties used for the packages, this then got copied over to the non-packaging log4j.properties in HADOOP-8216 (which made them consistent).

          Seems like we should keep with the v1.x setting which is disabled (WARNING level) by default. There's a performance overhead to audit logging, and HADOOP-7633 provided not rationale (just "We should add the audit logs as part of default confs") as to why they were enabled for the packages.

            Issue Links

            is related to

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-7633 log4j.properties should be added to the hadoop conf on deploy

            • Major - Major loss of function.
            • Closed

              Activity

              Ascending order - Click to sort in descending order
              Eli Collins created issue -
              Eli Collins made changes -
              Field Original Value New Value
              Description HADOOP-7633 made hdfs audit logging on by default (INFO level) in log4j.properties used for the packages, this then got copied over to the non-packaging log4j.properties in HADOOP-8216 (which made them consistent).

              Seems like we should keep with the v1.x setting which is disabled (WARNING level) by default. There's a performance overhead to audit logging, and HADOOP-7633 provided not rationale (just "We should add the audit logs as part of default confs") as to why they were enabled for the packages. Note that as of HADOOP-8224 you'll be able to configure this.               		HADOOP-7633 made hdfs, mr and security audit logging on by default (INFO level) in log4j.properties used for the packages, this then got copied over to the non-packaging log4j.properties in HADOOP-8216 (which made them consistent).

              Seems like we should keep with the v1.x setting which is disabled (WARNING level) by default. There's a performance overhead to audit logging, and HADOOP-7633 provided not rationale (just "We should add the audit logs as part of default confs") as to why they were enabled for the packages.
              Eli Collins made changes -
              Link This issue is related to HADOOP-7633 [ HADOOP-7633 ]
              Hide
              Permalink
              Eli Collins added a comment -

              Patch attached.

              • update hadoop.security.logger to NullAppender in log4j.properties, it is set explicitly by default in the bin and env scripts, so this is mostly a nop
              • hdfs/mapred.audit.logger and now default to the NullAppender in log4j.properties. Update hdfs.audit.logger in hadoop-env.sh to match. This is being made configurable in HADOOP-8224. mapred.audit.logger is not set in the bin or env scripts and is dead code, filed HADOOP-8392 for that (and to hookup RM/NM).

              Testing, verified the hdfs audit log is no longer automatically created and logged to when run from a tarball install.

              Show
              Eli Collins added a comment - Patch attached. update hadoop.security.logger to NullAppender in log4j.properties, it is set explicitly by default in the bin and env scripts, so this is mostly a nop hdfs/mapred.audit.logger and now default to the NullAppender in log4j.properties. Update hdfs.audit.logger in hadoop-env.sh to match. This is being made configurable in HADOOP-8224 . mapred.audit.logger is not set in the bin or env scripts and is dead code, filed HADOOP-8392 for that (and to hookup RM/NM). Testing, verified the hdfs audit log is no longer automatically created and logged to when run from a tarball install.
              Eli Collins made changes -
              Attachment hadoop-8316.txt [ 12526538 ]
              Eli Collins made changes -
              Status Open [ 1 ] Patch Available [ 10002 ]
              Hide
              Permalink
              Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12526538/hadoop-8316.txt
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/984//testReport/
              Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/984//console

              This message is automatically generated.

              Show
              Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526538/hadoop-8316.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/984//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/984//console This message is automatically generated.
              Hide
              Permalink
              Patrick Hunt added a comment -

              Reviewed the patch, looks good to me.

              Show
              Patrick Hunt added a comment - Reviewed the patch, looks good to me.
              Hide
              Permalink
              Patrick Hunt added a comment -

              +1

              Show
              Patrick Hunt added a comment - +1
              Hide
              Permalink
              Eli Collins added a comment -

              I've committed this and merged to branch-2.

              Show
              Eli Collins added a comment - I've committed this and merged to branch-2.
              Eli Collins made changes -
              Status Patch Available [ 10002 ] Resolved [ 5 ]
              Hadoop Flags Reviewed [ 10343 ]
              Target Version/s 2.0.0 [ 12320352 ]
              Fix Version/s 2.0.0 [ 12320352 ]
              Resolution Fixed [ 1 ]
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Hdfs-trunk-Commit #2307 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2307/)
              HADOOP-8316. Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334)

              Result = SUCCESS
              eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334
              Files :

              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
              Show
              Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2307 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2307/ ) HADOOP-8316 . Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Common-trunk-Commit #2233 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2233/)
              HADOOP-8316. Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334)

              Result = SUCCESS
              eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334
              Files :

              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
              Show
              Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2233 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2233/ ) HADOOP-8316 . Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Mapreduce-trunk-Commit #2250 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2250/)
              HADOOP-8316. Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334)

              Result = ABORTED
              eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334
              Files :

              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
              Show
              Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2250 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2250/ ) HADOOP-8316 . Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334) Result = ABORTED eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Hdfs-trunk #1041 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/)
              HADOOP-8316. Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334)

              Result = FAILURE
              eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334
              Files :

              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
              Show
              Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1041 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/ ) HADOOP-8316 . Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Mapreduce-trunk #1077 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/)
              HADOOP-8316. Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334)

              Result = SUCCESS
              eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334
              Files :

              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
              • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
              Show
              Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1077 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/ ) HADOOP-8316 . Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
              Todd Lipcon made changes -
              Fix Version/s 2.0.1 [ 12321441 ]
              Fix Version/s 2.0.0 [ 12320352 ]
              Eli Collins made changes -
              Assignee Eli Collins [ eli2 ] Eli Collins [ eli ]
              Arun C Murthy made changes -
              Fix Version/s 2.0.2-alpha [ 12322473 ]
              Fix Version/s 2.1.0-alpha [ 12321441 ]
              Arun C Murthy made changes -
              Status Resolved [ 5 ] Closed [ 6 ]

                People

                • Assignee:
                  Eli Collins
                  Reporter:
                  Eli Collins
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development