Hadoop Common
  1. Hadoop Common
  2. HADOOP-8316

Audit logging should be disabled by default

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: conf
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-7633 made hdfs, mr and security audit logging on by default (INFO level) in log4j.properties used for the packages, this then got copied over to the non-packaging log4j.properties in HADOOP-8216 (which made them consistent).

      Seems like we should keep with the v1.x setting which is disabled (WARNING level) by default. There's a performance overhead to audit logging, and HADOOP-7633 provided not rationale (just "We should add the audit logs as part of default confs") as to why they were enabled for the packages.

        Issue Links

          Activity

          Eli Collins created issue -
          Eli Collins made changes -
          Field Original Value New Value
          Description HADOOP-7633 made hdfs audit logging on by default (INFO level) in log4j.properties used for the packages, this then got copied over to the non-packaging log4j.properties in HADOOP-8216 (which made them consistent).

          Seems like we should keep with the v1.x setting which is disabled (WARNING level) by default. There's a performance overhead to audit logging, and HADOOP-7633 provided not rationale (just "We should add the audit logs as part of default confs") as to why they were enabled for the packages. Note that as of HADOOP-8224 you'll be able to configure this.
          HADOOP-7633 made hdfs, mr and security audit logging on by default (INFO level) in log4j.properties used for the packages, this then got copied over to the non-packaging log4j.properties in HADOOP-8216 (which made them consistent).

          Seems like we should keep with the v1.x setting which is disabled (WARNING level) by default. There's a performance overhead to audit logging, and HADOOP-7633 provided not rationale (just "We should add the audit logs as part of default confs") as to why they were enabled for the packages.
          Eli Collins made changes -
          Link This issue is related to HADOOP-7633 [ HADOOP-7633 ]
          Hide
          Eli Collins added a comment -

          Patch attached.

          • update hadoop.security.logger to NullAppender in log4j.properties, it is set explicitly by default in the bin and env scripts, so this is mostly a nop
          • hdfs/mapred.audit.logger and now default to the NullAppender in log4j.properties. Update hdfs.audit.logger in hadoop-env.sh to match. This is being made configurable in HADOOP-8224. mapred.audit.logger is not set in the bin or env scripts and is dead code, filed HADOOP-8392 for that (and to hookup RM/NM).

          Testing, verified the hdfs audit log is no longer automatically created and logged to when run from a tarball install.

          Show
          Eli Collins added a comment - Patch attached. update hadoop.security.logger to NullAppender in log4j.properties, it is set explicitly by default in the bin and env scripts, so this is mostly a nop hdfs/mapred.audit.logger and now default to the NullAppender in log4j.properties. Update hdfs.audit.logger in hadoop-env.sh to match. This is being made configurable in HADOOP-8224 . mapred.audit.logger is not set in the bin or env scripts and is dead code, filed HADOOP-8392 for that (and to hookup RM/NM). Testing, verified the hdfs audit log is no longer automatically created and logged to when run from a tarball install.
          Eli Collins made changes -
          Attachment hadoop-8316.txt [ 12526538 ]
          Eli Collins made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12526538/hadoop-8316.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/984//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/984//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526538/hadoop-8316.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/984//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/984//console This message is automatically generated.
          Hide
          Patrick Hunt added a comment -

          Reviewed the patch, looks good to me.

          Show
          Patrick Hunt added a comment - Reviewed the patch, looks good to me.
          Hide
          Patrick Hunt added a comment -

          +1

          Show
          Patrick Hunt added a comment - +1
          Hide
          Eli Collins added a comment -

          I've committed this and merged to branch-2.

          Show
          Eli Collins added a comment - I've committed this and merged to branch-2.
          Eli Collins made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Target Version/s 2.0.0 [ 12320352 ]
          Fix Version/s 2.0.0 [ 12320352 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2307 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2307/)
          HADOOP-8316. Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2307 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2307/ ) HADOOP-8316 . Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2233 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2233/)
          HADOOP-8316. Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2233 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2233/ ) HADOOP-8316 . Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2250 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2250/)
          HADOOP-8316. Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334)

          Result = ABORTED
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2250 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2250/ ) HADOOP-8316 . Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334) Result = ABORTED eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1041 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/)
          HADOOP-8316. Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1041 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/ ) HADOOP-8316 . Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1077 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/)
          HADOOP-8316. Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1077 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/ ) HADOOP-8316 . Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Todd Lipcon made changes -
          Fix Version/s 2.0.1 [ 12321441 ]
          Fix Version/s 2.0.0 [ 12320352 ]
          Eli Collins made changes -
          Assignee Eli Collins [ eli2 ] Eli Collins [ eli ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322473 ]
          Fix Version/s 2.1.0-alpha [ 12321441 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Eli Collins
              Reporter:
              Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development