Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1874

Ship with log4j.properties file that has a reliable time based rolling policy

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0
    • Component/s: None
    • Labels:
      None

      Description

      i.e. http://logging.apache.org/log4j/companions/extras/apidocs/org/apache/log4j/rolling/TimeBasedRollingPolicy.html

      DailyRollingFileAppender exhibits data loss so that is a no-go

      Need to investigate what a reliable solution is

      1. FLUME-1874-1.patch
        3 kB
        Mike Percy
      2. FLUME-1874-0.patch
        2 kB
        Mike Percy

        Activity

        Hide
        mpercy Mike Percy added a comment -

        On 2nd thought, defaulting to something that can fill a disk is dangerous.

        But we should support something like this as an example that is disabled by default:

        # Warning! This will fill up your disk if you don't have a cleanup job!
        log4j.appender.ROLL=org.apache.log4j.rolling.RollingFileAppender
        log4j.appender.ROLL.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
        log4j.appender.ROLL.rollingPolicy.ActiveFileName=${flume.log.dir}/${flume.log.file}
        log4j.appender.ROLL.rollingPolicy.FileNamePattern=${flume.log.dir}/archive/${flume.log.file}.%d{yyyy-MM-dd_HH-MM}
        log4j.appender.ROLL.layout=org.apache.log4j.PatternLayout
        log4j.appender.ROLL.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n
        
        Show
        mpercy Mike Percy added a comment - On 2nd thought, defaulting to something that can fill a disk is dangerous. But we should support something like this as an example that is disabled by default: # Warning! This will fill up your disk if you don't have a cleanup job! log4j.appender.ROLL=org.apache.log4j.rolling.RollingFileAppender log4j.appender.ROLL.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy log4j.appender.ROLL.rollingPolicy.ActiveFileName=${flume.log.dir}/${flume.log.file} log4j.appender.ROLL.rollingPolicy.FileNamePattern=${flume.log.dir}/archive/${flume.log.file}.%d{yyyy-MM-dd_HH-MM} log4j.appender.ROLL.layout=org.apache.log4j.PatternLayout log4j.appender.ROLL.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n
        Hide
        mpercy Mike Percy added a comment -

        I was thinking something along these lines but I guess we should add the log4j-extras dep to all modules? Or maybe just the dist module? No reason I can think of to use it in unit tests, for example, it's just for deployments.

        Show
        mpercy Mike Percy added a comment - I was thinking something along these lines but I guess we should add the log4j-extras dep to all modules? Or maybe just the dist module? No reason I can think of to use it in unit tests, for example, it's just for deployments.
        Hide
        mpercy Mike Percy added a comment -

        I'm inclined to include this in v1.4.0, it's a minimal patch that mainly just adds log4j-extras as a dep with a log4j config example.

        Would be great if someone wants to review / commit it, if not I'll pull it in as part of my 1.4 RM duties.

        Show
        mpercy Mike Percy added a comment - I'm inclined to include this in v1.4.0, it's a minimal patch that mainly just adds log4j-extras as a dep with a log4j config example. Would be great if someone wants to review / commit it, if not I'll pull it in as part of my 1.4 RM duties.
        Hide
        mpercy Mike Percy added a comment -

        Updated patch

        Show
        mpercy Mike Percy added a comment - Updated patch
        Hide
        mpercy Mike Percy added a comment -

        I just pushed this to trunk and flume-1.4 branches.

        Show
        mpercy Mike Percy added a comment - I just pushed this to trunk and flume-1.4 branches.
        Hide
        hudson Hudson added a comment -

        Integrated in flume-trunk #454 (See https://builds.apache.org/job/flume-trunk/454/)
        FLUME-1874. Ship with log4j.properties file that has a reliable time based rolling policy. (Revision 2f90fd8b08bfda9c7b41d2f7a2ac03da10d6f057)

        Result = FAILURE
        mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=2f90fd8b08bfda9c7b41d2f7a2ac03da10d6f057
        Files :

        • pom.xml
        • conf/log4j.properties
        • flume-ng-node/pom.xml
        Show
        hudson Hudson added a comment - Integrated in flume-trunk #454 (See https://builds.apache.org/job/flume-trunk/454/ ) FLUME-1874 . Ship with log4j.properties file that has a reliable time based rolling policy. (Revision 2f90fd8b08bfda9c7b41d2f7a2ac03da10d6f057) Result = FAILURE mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=2f90fd8b08bfda9c7b41d2f7a2ac03da10d6f057 Files : pom.xml conf/log4j.properties flume-ng-node/pom.xml

          People

          • Assignee:
            mpercy Mike Percy
            Reporter:
            mpercy Mike Percy
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development