Hadoop Common
  1. Hadoop Common
  2. HADOOP-8149

cap space usage of default log4j rolling policy

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0-alpha
    • Component/s: conf
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings of hadoop.root.logger and hadoop.security.logger can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER environment variables to use DRFA and DRFAS respectively.
      Show
      Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings of hadoop.root.logger and hadoop.security.logger can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER environment variables to use DRFA and DRFAS respectively.

      Description

      I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.

      I recommend we move to an appender which:

      1) caps the max file size (configurable)
      2) caps the max number of files to keep (configurable)
      3) uses rolling file appender rather than DRFA, see the warning here:
      http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
      Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."

      We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.

      1. HADOOP-8149.patch
        21 kB
        Patrick Hunt
      2. HADOOP-8149.patch
        19 kB
        Patrick Hunt
      3. HADOOP-8149.patch
        18 kB
        Patrick Hunt
      4. HADOOP-8149.patch
        17 kB
        Patrick Hunt
      5. HADOOP-8149.patch
        17 kB
        Patrick Hunt

        Issue Links

          Activity

          Patrick Hunt created issue -
          Hide
          Patrick Hunt added a comment -

          The attached patch demonstrates the changes I'm suggesting.

          I also noticed that the package/template log4j file was out of sync somewhat with the main file, I attempted to rectify this (lmk if that's not correct).

          Show
          Patrick Hunt added a comment - The attached patch demonstrates the changes I'm suggesting. I also noticed that the package/template log4j file was out of sync somewhat with the main file, I attempted to rectify this (lmk if that's not correct).
          Patrick Hunt made changes -
          Field Original Value New Value
          Attachment HADOOP-8149.patch [ 12517443 ]
          Patrick Hunt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12517443/HADOOP-8149.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/683//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517443/HADOOP-8149.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/683//console This message is automatically generated.
          Hide
          Patrick Hunt added a comment -

          Jenkins doesn't like me, but I'd still like to get some feedback if this approach is acceptable or not.

          Show
          Patrick Hunt added a comment - Jenkins doesn't like me, but I'd still like to get some feedback if this approach is acceptable or not.
          Hide
          Philip Zeyliger added a comment -

          +1 (from a non-committer).

          I think this would be a great improvement. The previous defaults easily led to HDFS and MR eating up your disk if they're stuck in an error loop, and I've seen that happen quite often.

          – Philip

          Show
          Philip Zeyliger added a comment - +1 (from a non-committer). I think this would be a great improvement. The previous defaults easily led to HDFS and MR eating up your disk if they're stuck in an error loop, and I've seen that happen quite often. – Philip
          Hide
          Eli Collins added a comment -

          +1 Great change, and the patch looks good. Pat mentioned to me offline that he tested via running a Hadoop tarball and verifying that it rolls when the log is at the configurable amount, and that it keeps the configurable of files, ie 100mb and 10, which seem like good defaults.

          Jenkins wants the patches to be split, per project, so you'll need a MR and HDFS jira for those changes. If you create those I'll re-review the splitup patches.

          Thanks,
          Eli

          Show
          Eli Collins added a comment - +1 Great change, and the patch looks good. Pat mentioned to me offline that he tested via running a Hadoop tarball and verifying that it rolls when the log is at the configurable amount, and that it keeps the configurable of files, ie 100mb and 10, which seem like good defaults. Jenkins wants the patches to be split, per project, so you'll need a MR and HDFS jira for those changes. If you create those I'll re-review the splitup patches. Thanks, Eli
          Hide
          Patrick Hunt added a comment -

          Yes, will do. Canceling the patch while I work on splitting it out.

          Show
          Patrick Hunt added a comment - Yes, will do. Canceling the patch while I work on splitting it out.
          Patrick Hunt made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Patrick Hunt made changes -
          Link This issue is required by HDFS-3066 [ HDFS-3066 ]
          Patrick Hunt made changes -
          Link This issue is required by MAPREDUCE-3989 [ MAPREDUCE-3989 ]
          Hide
          Patrick Hunt added a comment -

          Updated patch with common only in this patch. I re-verified by running from tarball with DEBUG turned on and the maxfilesize set to just 10k.

          Show
          Patrick Hunt added a comment - Updated patch with common only in this patch. I re-verified by running from tarball with DEBUG turned on and the maxfilesize set to just 10k.
          Patrick Hunt made changes -
          Attachment HADOOP-8149.patch [ 12517633 ]
          Patrick Hunt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Eli Collins made changes -
          Assignee Patrick Hunt [ phunt ]
          Hide
          Patrick Hunt added a comment -

          Notice that I did change the appender name from DRFA to RFA, this will likely require doc changes somewhere...

          Show
          Patrick Hunt added a comment - Notice that I did change the appender name from DRFA to RFA, this will likely require doc changes somewhere...
          Hide
          Kihwal Lee added a comment -

          This is a nice change. There is one thing I want to point out though. There can be external tools and processes that depend on the daily rolling of logs. It will be better if we let them continue until they are ready to transition and modify the tools.

          There are couple of places in the scripts where values are hardcoded (e.g. hadoop-daemon.sh and hadoop-env.sh). If we can change these to be dynamic while making RFA to be the default, we can accommodate the people who want to retain the old behavior by simply letting them have custom configs.

          Show
          Kihwal Lee added a comment - This is a nice change. There is one thing I want to point out though. There can be external tools and processes that depend on the daily rolling of logs. It will be better if we let them continue until they are ready to transition and modify the tools. There are couple of places in the scripts where values are hardcoded (e.g. hadoop-daemon.sh and hadoop-env.sh). If we can change these to be dynamic while making RFA to be the default, we can accommodate the people who want to retain the old behavior by simply letting them have custom configs.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12517633/HADOOP-8149.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.ipc.TestRPCCallBenchmark

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/690//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/690//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517633/HADOOP-8149.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.ipc.TestRPCCallBenchmark +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/690//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/690//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          Kihwal,

          Good point. Given that the 23.x releases are currently alpha/beta, introduce new daemons and logs, etc, isn't this a good time for people to update? People can update their log4j.properties and env variables if they'd like to stay on the current behavior, no code changes should be required.

          Thanks,
          Eli

          Show
          Eli Collins added a comment - Kihwal, Good point. Given that the 23.x releases are currently alpha/beta, introduce new daemons and logs, etc, isn't this a good time for people to update? People can update their log4j.properties and env variables if they'd like to stay on the current behavior, no code changes should be required. Thanks, Eli
          Eli Collins made changes -
          Target Version/s 0.23.3 [ 12320059 ]
          Hide
          Eli Collins added a comment -

          +1 to the latest patch. I've updated the release notes field with information on how to configure the old behavior for users who don't want to switch.

          Show
          Eli Collins added a comment - +1 to the latest patch. I've updated the release notes field with information on how to configure the old behavior for users who don't want to switch.
          Eli Collins made changes -
          Hadoop Flags Incompatible change,Reviewed [ 10342,10343 ]
          Release Note Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER environment variables to use DRFA and DRFAS respectively.
          Hide
          Suresh Srinivas added a comment -

          Guys, can you please give me some time to look at this. I like the idea of capping the space used for logs. However, changing the appender and its impact on the file names etc. could break the existing tools/management around logs.

          Show
          Suresh Srinivas added a comment - Guys, can you please give me some time to look at this. I like the idea of capping the space used for logs. However, changing the appender and its impact on the file names etc. could break the existing tools/management around logs.
          Hide
          Rajiv Chittajallu added a comment -

          This change should be an option not the default. This is going to break lot of current tools.

          Show
          Rajiv Chittajallu added a comment - This change should be an option not the default. This is going to break lot of current tools.
          Hide
          Eli Collins added a comment -

          How about we add back the DRFA section in the log4j.properties as it currently is? This way:

          1. People can keep the current behavior by just updating the HADOOP_*_LOGGER env vars to "DRFA"
          2. We don't ship with a default logger that fills up partitions, has known data loss issues
          Show
          Eli Collins added a comment - How about we add back the DRFA section in the log4j.properties as it currently is? This way: People can keep the current behavior by just updating the HADOOP_*_LOGGER env vars to "DRFA" We don't ship with a default logger that fills up partitions, has known data loss issues
          Hide
          Kihwal Lee added a comment -

          I am fine with the default being RFA. But the latest patch still hardcodes the logger names in hadoop-env.sh and hadoop-daemons.sh, which will unconditionally override custom env var. They should set the logger to RFA only when HADOOP_*_LOGGER is not already set. I am not very concerned with the changes in log4j.properties, as it can be easily replaced by users, but leaving the DRFA settings won't hurt either and make some people's life easier.

          Show
          Kihwal Lee added a comment - I am fine with the default being RFA. But the latest patch still hardcodes the logger names in hadoop-env.sh and hadoop-daemons.sh, which will unconditionally override custom env var. They should set the logger to RFA only when HADOOP_*_LOGGER is not already set. I am not very concerned with the changes in log4j.properties, as it can be easily replaced by users, but leaving the DRFA settings won't hurt either and make some people's life easier.
          Hide
          Suresh Srinivas added a comment -

          We don't ship with a default logger that fills up partitions, has known data loss issues

          Have you noticed cases of data loss issues. Because I have not heard of these instances.

          As regards to filling up partitions, many of the installations have tools to handle this and not let either logs or some other files to fill up the partitions.

          As I said, I like the idea of providing this as a default option during fresh install. But during upgrade, changing the existing appender is problematic.

          Show
          Suresh Srinivas added a comment - We don't ship with a default logger that fills up partitions, has known data loss issues Have you noticed cases of data loss issues. Because I have not heard of these instances. As regards to filling up partitions, many of the installations have tools to handle this and not let either logs or some other files to fill up the partitions. As I said, I like the idea of providing this as a default option during fresh install. But during upgrade, changing the existing appender is problematic.
          Hide
          Eli Collins added a comment -

          Have you noticed cases of data loss issues. Because I have not heard of these instances.

          No, me neither. We do see the issue with partitions filling up frequently, people expect this to work out of the box.

          They should set the logger to RFA only when HADOOP_*_LOGGER is not already set

          Sounds like a plan. Pat, how about updating the patch to (a) add back the DRFA log4j.properties section and (b) on set HADOOP_*_LOGGER if they're already unset so they can be overriden in hadoop-env.sh?

          Show
          Eli Collins added a comment - Have you noticed cases of data loss issues. Because I have not heard of these instances. No, me neither. We do see the issue with partitions filling up frequently, people expect this to work out of the box. They should set the logger to RFA only when HADOOP_*_LOGGER is not already set Sounds like a plan. Pat, how about updating the patch to (a) add back the DRFA log4j.properties section and (b) on set HADOOP_*_LOGGER if they're already unset so they can be overriden in hadoop-env.sh?
          Hide
          Patrick Hunt added a comment -

          Sounds reasonable. Notice that in this patch DRFAS was renamed to RFAS and some of the appenders (DRFAS/JSA/MRAUDIT/RMSUMMARY) are changed from DailyRolling to just Rolling. These are fine or should some action be taken wrt backward compatibility?

          Show
          Patrick Hunt added a comment - Sounds reasonable. Notice that in this patch DRFAS was renamed to RFAS and some of the appenders (DRFAS/JSA/MRAUDIT/RMSUMMARY) are changed from DailyRolling to just Rolling. These are fine or should some action be taken wrt backward compatibility?
          Patrick Hunt made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Eli Collins added a comment -

          Per Kihwal above (correct me if I'm wrong!) they're OK with the log4.properties changes since they can easily set these with their own file, it's just the unconditional export of _LOGGER in hadoop-daemon.sh that we need to address.

          Show
          Eli Collins added a comment - Per Kihwal above (correct me if I'm wrong!) they're OK with the log4.properties changes since they can easily set these with their own file, it's just the unconditional export of _LOGGER in hadoop-daemon.sh that we need to address.
          Hide
          Suresh Srinivas added a comment -

          Admins probably do not want on upgrade this change. At least in release notes we should cover this. Not sure your above comment addresses that, Eli. Also don't you guys think the cap size of 1GB is way too small.

          Show
          Suresh Srinivas added a comment - Admins probably do not want on upgrade this change. At least in release notes we should cover this. Not sure your above comment addresses that, Eli. Also don't you guys think the cap size of 1GB is way too small.
          Hide
          Patrick Hunt added a comment -

          Updated per the review comments.

          Show
          Patrick Hunt added a comment - Updated per the review comments.
          Patrick Hunt made changes -
          Attachment HADOOP-8149.patch [ 12518569 ]
          Hide
          Patrick Hunt added a comment -

          Hm. Well not all of them. Snuck in while I was testing.

          Suresh et al. suggestions on cap?

          Show
          Patrick Hunt added a comment - Hm. Well not all of them. Snuck in while I was testing. Suresh et al. suggestions on cap?
          Hide
          Kihwal Lee added a comment -

          Please make hadoop-env.sh honor user-specified log appender.

          For example, when the name node is started through scripts, the last logger setting on the command line will be from $HADOOP_NAMENODE_OPTS, which is currently hardcoded in hadoop-env.sh. This will override whatever user specified. (last one wins)

          As a side note, we need to fix the shell scripts so that they don't source the same thing again and again and duplicate or triplicate the same command line options. Users can easily be confused about when something is set, eval'ed or overridden. But this is beyond the scope of this jira.

          As for the cap, we need to keep in mind that hadoop components are very noisy. I see hadoop service processes generating multiple gigabytes of log messages per day. So 1 GB feels too small, but, for the same reason, the default may need to be small enough to keep single-node desktop users from running into trouble. I will throw a number to get it going. How about 5GB? 5 GB * 5-8 processes <= 40 GB.

          Show
          Kihwal Lee added a comment - Please make hadoop-env.sh honor user-specified log appender. For example, when the name node is started through scripts, the last logger setting on the command line will be from $HADOOP_NAMENODE_OPTS, which is currently hardcoded in hadoop-env.sh. This will override whatever user specified. (last one wins) As a side note, we need to fix the shell scripts so that they don't source the same thing again and again and duplicate or triplicate the same command line options. Users can easily be confused about when something is set, eval'ed or overridden. But this is beyond the scope of this jira. As for the cap, we need to keep in mind that hadoop components are very noisy. I see hadoop service processes generating multiple gigabytes of log messages per day. So 1 GB feels too small, but, for the same reason, the default may need to be small enough to keep single-node desktop users from running into trouble. I will throw a number to get it going. How about 5GB? 5 GB * 5-8 processes <= 40 GB.
          Hide
          Philip Zeyliger added a comment -

          5GB seems like a reasonable number to me. I'm excited that our defaults may no longer fill up our hard disks when stuff is awry.

          Thanks!

          Show
          Philip Zeyliger added a comment - 5GB seems like a reasonable number to me. I'm excited that our defaults may no longer fill up our hard disks when stuff is awry. Thanks!
          Hide
          Patrick Hunt added a comment -

          Seems to be some consensus around 5gb. I'll update the patch to use max index of 20, with max file size of 256mb.

          @kihwal - You are specifically referring to this line, correct?

          export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,RFAAUDIT $HADOOP_NAMENODE_OPTS"

          where setting the security logger is overridden.

          I was hesitant to change hadoop-env.sh for a couple reasons, appreciate your help on this.

          1) my understanding is that hadoop-env.sh is customized by the end user. They would edit this file with specific requirements/customizations, but that the default should handle the common case.

          2) notice the following four lines, they are a hodgepodge of inconsistent threshold/appender settings. Are you suggesting that we override all of these based on the HADOOP_SECURITY_LOGGER setting?

          Esp wrt item 2) I was thinking along the lines of your other comment (re scope) - that addressing this would be good to break off into a separate jira focused exclusively on that issue. However perhaps I'm mistaken and there's a simple solution here that I'm missing.

          Show
          Patrick Hunt added a comment - Seems to be some consensus around 5gb. I'll update the patch to use max index of 20, with max file size of 256mb. @kihwal - You are specifically referring to this line, correct? export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,RFAAUDIT $HADOOP_NAMENODE_OPTS" where setting the security logger is overridden. I was hesitant to change hadoop-env.sh for a couple reasons, appreciate your help on this. 1) my understanding is that hadoop-env.sh is customized by the end user. They would edit this file with specific requirements/customizations, but that the default should handle the common case. 2) notice the following four lines, they are a hodgepodge of inconsistent threshold/appender settings. Are you suggesting that we override all of these based on the HADOOP_SECURITY_LOGGER setting? Esp wrt item 2) I was thinking along the lines of your other comment (re scope) - that addressing this would be good to break off into a separate jira focused exclusively on that issue. However perhaps I'm mistaken and there's a simple solution here that I'm missing.
          Hide
          Rajiv Chittajallu added a comment -

          5GB seems like a reasonable number to me. I'm excited that our defaults may no longer fill up our hard disks when stuff is awry.

          from my experience in running hadoop, having sufficient logs helped us in debugging issues when there are issues. With the current logging and error reporting of hadoop daemons, I wouldn't recommend log4j purging logs, especially by space, to any one.

          Show
          Rajiv Chittajallu added a comment - 5GB seems like a reasonable number to me. I'm excited that our defaults may no longer fill up our hard disks when stuff is awry. from my experience in running hadoop, having sufficient logs helped us in debugging issues when there are issues. With the current logging and error reporting of hadoop daemons, I wouldn't recommend log4j purging logs, especially by space, to any one.
          Hide
          Eli Collins added a comment -

          If you don't purge logs then eventually the partition fills up and your cluster fails. That's not an option. If you want to retain lots of logs then just bump the retention size in the config to whatever the partition your logs reside on can support and you're good.

          Show
          Eli Collins added a comment - If you don't purge logs then eventually the partition fills up and your cluster fails. That's not an option. If you want to retain lots of logs then just bump the retention size in the config to whatever the partition your logs reside on can support and you're good.
          Hide
          Eli Collins added a comment -

          Pat and I sync up wrt Kihwal's point about HADOOP_NAMENODE_OPTS and his bullet #2 above.

          • HADOOP_NAMENODE_OPTS, HADOOP_SECONDARYNAMENODE_OPTS, and HADOOP_DATANODE_OPTS all hard-code the logger values and instead use an environment variable. The NN and the 2NN are the same and can use HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER instead of needing new service specific variables (this is how yarn works).
          • HADOOP_DATANODE_OPTS uses ERROR,DRFAS instead of INFO,DRFAS like the NN and 2NN. It was originally created this way, anyone know why? Seems like it should use the same setting as the NN and 2NN and be exported like them as well. If that the case then we can use HADOOP_ROOT/SECURITY_LOGGER here as well.
          • Filed HADOOP-8200 with a patch to remove HADOOP_JOBTRACKER/TASKTRACKER_OPTS which are no longer relevant on trunk/23 so we can ignore these here
          Show
          Eli Collins added a comment - Pat and I sync up wrt Kihwal's point about HADOOP_NAMENODE_OPTS and his bullet #2 above. HADOOP_NAMENODE_OPTS, HADOOP_SECONDARYNAMENODE_OPTS, and HADOOP_DATANODE_OPTS all hard-code the logger values and instead use an environment variable. The NN and the 2NN are the same and can use HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER instead of needing new service specific variables (this is how yarn works). HADOOP_DATANODE_OPTS uses ERROR,DRFAS instead of INFO,DRFAS like the NN and 2NN. It was originally created this way, anyone know why? Seems like it should use the same setting as the NN and 2NN and be exported like them as well. If that the case then we can use HADOOP_ROOT/SECURITY_LOGGER here as well. Filed HADOOP-8200 with a patch to remove HADOOP_JOBTRACKER/TASKTRACKER_OPTS which are no longer relevant on trunk/23 so we can ignore these here
          Hide
          Eli Collins added a comment -

          @Kihwal, does the above sound good to you?

          @Pat, I committed HADOOP-8200, and think we can make HADOOP_DATANODE_OPTS use INFO,DRFAS, so you should just need to update the patch so HADOOP_ROOT/SECURITY_LOGGER are respected (ie HADOOP_*_OPTS should use the env instead of hard-coding them)

          Show
          Eli Collins added a comment - @Kihwal, does the above sound good to you? @Pat, I committed HADOOP-8200 , and think we can make HADOOP_DATANODE_OPTS use INFO,DRFAS, so you should just need to update the patch so HADOOP_ROOT/SECURITY_LOGGER are respected (ie HADOOP_*_OPTS should use the env instead of hard-coding them)
          Eli Collins made changes -
          Assignee Patrick Hunt [ phunt ] Eli Collins [ eli2 ]
          Eli Collins made changes -
          Assignee Eli Collins [ eli2 ] Patrick Hunt [ phunt ]
          Hide
          Patrick Hunt added a comment -

          I've created HADOOP-8216 in order to address the DRFAAUDIT missing from the main log4j.properties.

          Show
          Patrick Hunt added a comment - I've created HADOOP-8216 in order to address the DRFAAUDIT missing from the main log4j.properties.
          Hide
          Kihwal Lee added a comment -

          Eli and Pat,
          Thanks for fixing the script bugs. What you suggested sounds good. As for HADOOP_DATANODE_OPTS, it might have been set that way because INFO makes it too noisy and affects performance. The production clusters I've seen keep it at ERROR. But if you have seen INFO being used in production without causing insane log bloat or performance issues, then I agree we should use INFO. If unsure, we can leave it as ERROR and file a non-blocking jira to investigate. I tried running a DN at INFO and it seems the BlockPoolSliceScanner is the noisiest one, probably because the period is set to 0 by default.

          Show
          Kihwal Lee added a comment - Eli and Pat, Thanks for fixing the script bugs. What you suggested sounds good. As for HADOOP_DATANODE_OPTS, it might have been set that way because INFO makes it too noisy and affects performance. The production clusters I've seen keep it at ERROR. But if you have seen INFO being used in production without causing insane log bloat or performance issues, then I agree we should use INFO. If unsure, we can leave it as ERROR and file a non-blocking jira to investigate. I tried running a DN at INFO and it seems the BlockPoolSliceScanner is the noisiest one, probably because the period is set to 0 by default.
          Eli Collins made changes -
          Link This issue is related to HADOOP-8216 [ HADOOP-8216 ]
          Eli Collins made changes -
          Link This issue is related to HBASE-5655 [ HBASE-5655 ]
          Hide
          Patrick Hunt added a comment -

          I believe this updated patch addresses everyone's concerns:

          1) upped the cap to 5gb (256M * 20)
          2) addressed the security logger concerns for nn/2nn OPTS
          3) I left DN as is for the time being per Kihwal's comment (indeed the SecurityLogger seems noisy).
          4) HADOOP_ROOT_LOGGER is handled in hadoop-config.sh and not overridden by the nn/2nn OPTS, so I believe we're good with that as well.

          Show
          Patrick Hunt added a comment - I believe this updated patch addresses everyone's concerns: 1) upped the cap to 5gb (256M * 20) 2) addressed the security logger concerns for nn/2nn OPTS 3) I left DN as is for the time being per Kihwal's comment (indeed the SecurityLogger seems noisy). 4) HADOOP_ROOT_LOGGER is handled in hadoop-config.sh and not overridden by the nn/2nn OPTS, so I believe we're good with that as well.
          Patrick Hunt made changes -
          Attachment HADOOP-8149.patch [ 12520333 ]
          Patrick Hunt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER environment variables to use DRFA and DRFAS respectively. Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER environment variable to use DRFA.
          Patrick Hunt made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Patrick Hunt added a comment -

          5) I realized I missed keeping DRFAS in addition to keeping DRFA. It's in this latest patch.

          Show
          Patrick Hunt added a comment - 5) I realized I missed keeping DRFAS in addition to keeping DRFA. It's in this latest patch.
          Patrick Hunt made changes -
          Attachment HADOOP-8149.patch [ 12520335 ]
          Patrick Hunt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER environment variable to use DRFA. Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER environment variables to use DRFA and DRFAS respectively.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520335/HADOOP-8149.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/794//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/794//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520335/HADOOP-8149.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/794//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/794//console This message is automatically generated.
          Eli Collins made changes -
          Link This issue relates to HADOOP-8224 [ HADOOP-8224 ]
          Hide
          Eli Collins added a comment -

          +1 looks good. You should note in the release note that you can't override hdfs.audit.logger via HADOOP_*_LOGGER. I filed HADOOP-8224 in case people want to do that.

          Show
          Eli Collins added a comment - +1 looks good. You should note in the release note that you can't override hdfs.audit.logger via HADOOP_*_LOGGER. I filed HADOOP-8224 in case people want to do that.
          Hide
          Eli Collins added a comment -

          Also, mind chiming in wrt what testing you've done?

          Show
          Eli Collins added a comment - Also, mind chiming in wrt what testing you've done?
          Hide
          Patrick Hunt added a comment -

          In addition to running the unit tests I packaged everything up and ran that in the following scenarios:

          a) with the default hadoop-env.sh and log4j.properties, everything looked fine here, info level logs, etc...

          b) I added HADOOP_ROOT_LOGGER to the top of hadoop-env.sh with DEBUG logging turned on, I set the max file size to 10k and the max files to 3. started/stopped the daemons and verified that the files sizes were capped, and that rolling started once we reached 3 log files (backups)

          Show
          Patrick Hunt added a comment - In addition to running the unit tests I packaged everything up and ran that in the following scenarios: a) with the default hadoop-env.sh and log4j.properties, everything looked fine here, info level logs, etc... b) I added HADOOP_ROOT_LOGGER to the top of hadoop-env.sh with DEBUG logging turned on, I set the max file size to 10k and the max files to 3. started/stopped the daemons and verified that the files sizes were capped, and that rolling started once we reached 3 log files (backups)
          Eli Collins made changes -
          Link This issue relates to FLUME-1073 [ FLUME-1073 ]
          Eli Collins made changes -
          Link This issue relates to ZOOKEEPER-1435 [ ZOOKEEPER-1435 ]
          Hide
          Eli Collins added a comment -

          I've committed this and merged to branch-2.

          Thanks Pat! Thank you Kihwal and Rajiv for the feedback!

          Show
          Eli Collins added a comment - I've committed this and merged to branch-2. Thanks Pat! Thank you Kihwal and Rajiv for the feedback!
          Eli Collins made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Release Note Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER environment variables to use DRFA and DRFAS respectively. Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings of hadoop.root.logger and hadoop.security.logger can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER environment variables to use DRFA and DRFAS respectively.
          Target Version/s 0.23.3 [ 12320059 ]
          Fix Version/s 2.0.0 [ 12320352 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2024 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2024/)
          HADOOP-8149. Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2024 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2024/ ) HADOOP-8149 . Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1949 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1949/)
          HADOOP-8149. Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1949 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1949/ ) HADOOP-8149 . Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1962 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1962/)
          HADOOP-8149. Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1962 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1962/ ) HADOOP-8149 . Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1000 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1000/)
          HADOOP-8149. Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1000 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1000/ ) HADOOP-8149 . Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1035 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1035/)
          HADOOP-8149. Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1035 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1035/ ) HADOOP-8149 . Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Patrick Hunt
              Reporter:
              Patrick Hunt
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development