Hadoop Common
  1. Hadoop Common
  2. HADOOP-8224

Don't hardcode hdfs.audit.logger in the scripts

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: conf
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The HADOOP_*OPTS defined for HDFS in hadoop-env.sh hard-code the hdfs.audit.logger (is explicitly set via "-Dhdfs.audit.logger=INFO,RFAAUDIT") so it's not overridable. Let's allow someone to override it as we do the other parameters by introducing HADOOP_AUDIT_LOGGER.

      1. HADOOP-8224.txt
        1 kB
        Tomohiko Kinebuchi
      2. HADOOP-8224.txt
        2 kB
        Tomohiko Kinebuchi
      3. hadoop-8224.txt
        2 kB
        Eli Collins

        Issue Links

          Activity

          Hide
          Tomohiko Kinebuchi added a comment -

          Attached a patch.
          Could someone give me contributor permission and assign to me?

          Show
          Tomohiko Kinebuchi added a comment - Attached a patch. Could someone give me contributor permission and assign to me?
          Hide
          Eli Collins added a comment -

          Thanks Tomohiko, I've added you as a contributor and assigned this to you.

          How did you test this change? Looks like your change should be sufficient, it's not necessary to update hadoop-daemon.sh with an export similar to the other _LOGGER variables.

          Show
          Eli Collins added a comment - Thanks Tomohiko, I've added you as a contributor and assigned this to you. How did you test this change? Looks like your change should be sufficient, it's not necessary to update hadoop-daemon.sh with an export similar to the other _LOGGER variables.
          Hide
          Eli Collins added a comment -

          Actually, it does look like the export in hadoop-daemon.sh is necessary, this is for the case where people don't have hadoop-env.sh in their conf dir.

          Show
          Eli Collins added a comment - Actually, it does look like the export in hadoop-daemon.sh is necessary, this is for the case where people don't have hadoop-env.sh in their conf dir.
          Hide
          Tomohiko Kinebuchi added a comment -

          Thank you for your comment.

          I created and attached a new patch including first fix and a fix for hadoop-daemon.sh.

          I am thinking how do I test these fixes.

          1. Kick a Hadoop cluster with this shell, and check existence of log messages in hdfs-audit.log.
          2. Kick a Hadoop cluster with this shell, and get configuration values from Configuration class.

          Do you have some other testing processes?

          Show
          Tomohiko Kinebuchi added a comment - Thank you for your comment. I created and attached a new patch including first fix and a fix for hadoop-daemon.sh. I am thinking how do I test these fixes. 1. Kick a Hadoop cluster with this shell, and check existence of log messages in hdfs-audit.log. 2. Kick a Hadoop cluster with this shell, and get configuration values from Configuration class. Do you have some other testing processes?
          Hide
          Eli Collins added a comment -

          To test you'll want to build the tarball and test that HDFS_AUDIT_LOGGER is respected when set in hadoop-env.sh, eg if set to the WARNING level you don't see audit logging, and if set to INFO you do.

          Btw I noticed while reviewing this that audit logging is now enabled by default, will address that in HADOOP-8316.

          Show
          Eli Collins added a comment - To test you'll want to build the tarball and test that HDFS_AUDIT_LOGGER is respected when set in hadoop-env.sh, eg if set to the WARNING level you don't see audit logging, and if set to INFO you do. Btw I noticed while reviewing this that audit logging is now enabled by default, will address that in HADOOP-8316 .
          Hide
          Eli Collins added a comment -

          Patch attached. Updated Tomohiko's patch to apply atop HADOOP-8316. Running the testing I mentioned in the previous comment now.

          Show
          Eli Collins added a comment - Patch attached. Updated Tomohiko's patch to apply atop HADOOP-8316 . Running the testing I mentioned in the previous comment now.
          Hide
          Eli Collins added a comment -

          Verified from a tarball install that there is no hdfs audit log by default and that exporting HDFS_AUDIT_LOGGER to INFO,RFAAUDIT in hadoop-env.sh enables it.

          Show
          Eli Collins added a comment - Verified from a tarball install that there is no hdfs audit log by default and that exporting HDFS_AUDIT_LOGGER to INFO,RFAAUDIT in hadoop-env.sh enables it.
          Hide
          Patrick Hunt added a comment -

          Reviewed the patch, looks good to me.

          Show
          Patrick Hunt added a comment - Reviewed the patch, looks good to me.
          Hide
          Patrick Hunt added a comment -

          +1

          Show
          Patrick Hunt added a comment - +1
          Hide
          Eli Collins added a comment -

          I've committed this and merged to branch-2. Thanks Tomohiko!

          Show
          Eli Collins added a comment - I've committed this and merged to branch-2. Thanks Tomohiko!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2307 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2307/)
          HADOOP-8224. Don't hardcode hdfs.audit.logger in the scripts. Contributed by Tomohiko Kinebuchi (Revision 1337339)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337339
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2307 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2307/ ) HADOOP-8224 . Don't hardcode hdfs.audit.logger in the scripts. Contributed by Tomohiko Kinebuchi (Revision 1337339) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337339 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2233 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2233/)
          HADOOP-8224. Don't hardcode hdfs.audit.logger in the scripts. Contributed by Tomohiko Kinebuchi (Revision 1337339)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337339
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2233 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2233/ ) HADOOP-8224 . Don't hardcode hdfs.audit.logger in the scripts. Contributed by Tomohiko Kinebuchi (Revision 1337339) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337339 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2250 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2250/)
          HADOOP-8224. Don't hardcode hdfs.audit.logger in the scripts. Contributed by Tomohiko Kinebuchi (Revision 1337339)

          Result = ABORTED
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337339
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2250 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2250/ ) HADOOP-8224 . Don't hardcode hdfs.audit.logger in the scripts. Contributed by Tomohiko Kinebuchi (Revision 1337339) Result = ABORTED eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337339 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1041 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/)
          HADOOP-8224. Don't hardcode hdfs.audit.logger in the scripts. Contributed by Tomohiko Kinebuchi (Revision 1337339)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337339
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1041 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/ ) HADOOP-8224 . Don't hardcode hdfs.audit.logger in the scripts. Contributed by Tomohiko Kinebuchi (Revision 1337339) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337339 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1077 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/)
          HADOOP-8224. Don't hardcode hdfs.audit.logger in the scripts. Contributed by Tomohiko Kinebuchi (Revision 1337339)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337339
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1077 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/ ) HADOOP-8224 . Don't hardcode hdfs.audit.logger in the scripts. Contributed by Tomohiko Kinebuchi (Revision 1337339) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337339 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh

            People

            • Assignee:
              Tomohiko Kinebuchi
              Reporter:
              Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development