Log4j 2
  1. Log4j 2
  2. LOG4J2-44

log4j java date wise log every X MB

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0-beta1
    • Component/s: Appenders
    • Labels:
      None

      Description

      a log roll over that makes a new file every x MB/ KB but the current file is archived with a full date-time-random-int-to-make unique stamp

      so if the file is myProj_A.log after x MB is renamed to
      myProj_A_2010_09_10_02_48_02_XX34334.log (year-month-day- time ...) and so on

      Is there a pattern/ property to do this already?

      can see http://www.simonsite.org.uk/ and http://www.simonsite.org.uk/javadoc/org/apache/log4j/appender/TimeAndSizeRollingAppender.html here the code is a little mixed up (we want test and real code in seperate src roots) - and not sure if it works completely but it says it does. but it has our requirements.

      Code should write out buffer in bits instead of blocking for a long time emptying full buffer when buffer is full Or is that done by nio ?

      build and run script to test with params to control logging, number of threads, amount of data logged so we can see that the logs are rolling as planned we want the final roller to roll every day but can keep that as hour and minute params (so final will be 24 hours and 0 minutes, but for testing can be 5 minutes only, similarly size can be in KB) also need format of file name with prefix for folder so the root log folder can be c:\log then we specify the dynamic sub folder like yyyy\mm\dd meaning make sub folders for current year, inside that for month and inside that for day and finally the file name format with a prefix like myApp_{yyyy_mm_dd_HH_II_SS) and post fix so the final file will be in folder (example) C:\log\2010\10\29\myApp_2010_10_29_14_34_22_randXYZ.txt where randXYZ stand for some random text that is added to make the file name unique (as required or can use java inbuilt getTempFile function ...)

        Activity

        Hide
        Ralph Goers added a comment -

        The RollingFileAppender using the DefaultRolloverStrategy in my Log4J 2.0 branch can include both a timestamp and an integer as part of the pattern. The integer is not random but a counter within the time range. Will that suffice or is a random integer required for some reason?

        Show
        Ralph Goers added a comment - The RollingFileAppender using the DefaultRolloverStrategy in my Log4J 2.0 branch can include both a timestamp and an integer as part of the pattern. The integer is not random but a counter within the time range. Will that suffice or is a random integer required for some reason?
        Hide
        Tushar added a comment -

        no do not need a random int

        how do i test this code how do i get your Log4J 2.0 branch ?

        Show
        Tushar added a comment - no do not need a random int how do i test this code how do i get your Log4J 2.0 branch ?
        Hide
        Ralph Goers added a comment - - edited

        https://svn.apache.org/repos/asf/logging/log4j/branches/BRANCH_2_0_EXPERIMENTAL/rgoers/

        log4j2-api is a new API for Log4j 2.0. slf4j-impl contains a binding for the SLF4J API. log4j12-api contains much of the log4j 1.2 API, although it isn't 100% compliant as 2.0 does not have LoggerRepository.

        RollingAppenderTimeAndSizeTest should illustrate the concept you are asking about. The pattern indicates the file should roll every minute. The %i indicates that it should increment the counter and roll when the size is exceeded. Sometimes when the test executes it will all happen in the same minute and sometimes it will span two minutes.

        To see visually that it works you will have to remove the call the deleteDir() in the cleanupClass() method.

        Show
        Ralph Goers added a comment - - edited https://svn.apache.org/repos/asf/logging/log4j/branches/BRANCH_2_0_EXPERIMENTAL/rgoers/ log4j2-api is a new API for Log4j 2.0. slf4j-impl contains a binding for the SLF4J API. log4j12-api contains much of the log4j 1.2 API, although it isn't 100% compliant as 2.0 does not have LoggerRepository. RollingAppenderTimeAndSizeTest should illustrate the concept you are asking about. The pattern indicates the file should roll every minute. The %i indicates that it should increment the counter and roll when the size is exceeded. Sometimes when the test executes it will all happen in the same minute and sometimes it will span two minutes. To see visually that it works you will have to remove the call the deleteDir() in the cleanupClass() method.
        Hide
        Ralph Goers added a comment -

        I'm wondering if you have had a chance to test this. Significant enhancements have been made since June although I can't say the RollingFileAppender has changed much.

        Show
        Ralph Goers added a comment - I'm wondering if you have had a chance to test this. Significant enhancements have been made since June although I can't say the RollingFileAppender has changed much.
        Hide
        Tushar added a comment -

        i will test this week and have some results out to by saturday

        Show
        Tushar added a comment - i will test this week and have some results out to by saturday
        Hide
        Tushar added a comment -

        Ralph I need some instructions
        I guess i need maven- never used it but will get it

        after that what do I do? need to build top level pom or call all sub projects?

        D:\prog\j\log4j\RalphG\rgoers\pom.xml
        D:\prog\j\log4j\RalphG\rgoers\log4j2-api
        D:\prog\j\log4j\RalphG\rgoers\log4j2-core
        D:\prog\j\log4j\RalphG\rgoers\log4j2-jcl
        D:\prog\j\log4j\RalphG\rgoers\log4j12-api
        D:\prog\j\log4j\RalphG\rgoers\slf4j-impl
        D:\prog\j\log4j\RalphG\rgoers\src

        yes i'm on windows , have used ant but not maven

        Show
        Tushar added a comment - Ralph I need some instructions I guess i need maven- never used it but will get it after that what do I do? need to build top level pom or call all sub projects? D:\prog\j\log4j\RalphG\rgoers\pom.xml D:\prog\j\log4j\RalphG\rgoers\log4j2-api D:\prog\j\log4j\RalphG\rgoers\log4j2-core D:\prog\j\log4j\RalphG\rgoers\log4j2-jcl D:\prog\j\log4j\RalphG\rgoers\log4j12-api D:\prog\j\log4j\RalphG\rgoers\slf4j-impl D:\prog\j\log4j\RalphG\rgoers\src yes i'm on windows , have used ant but not maven
        Hide
        Ralph Goers added a comment -

        To build you run mvn clean install from the root directory of the project. If you want to skip unit tests add -Dmaven.test.skip. I would suggest you run them as I haven't done a build on Windows - at least, not recently - so I'd appreciate knowing if there are any test failures. After you run that you will have a maven repository installed at ~/.m2/repository with the jars in it. They will also be in each subproject's target directory.

        Show
        Ralph Goers added a comment - To build you run mvn clean install from the root directory of the project. If you want to skip unit tests add -Dmaven.test.skip. I would suggest you run them as I haven't done a build on Windows - at least, not recently - so I'd appreciate knowing if there are any test failures. After you run that you will have a maven repository installed at ~/.m2/repository with the jars in it. They will also be in each subproject's target directory.
        Hide
        Tushar added a comment -

        see http://sel2in.com/pages/log4j/2012march10.txt for mvn clean install output on your old 25 Oct 2011 build which I ran only today

        i see that the svn branch no longer exists

        do you have a new branch?

        sorry for the late reply but we temporarily fixed our prod problem by combing async with our rolling file appender

        Is there a way to add a listener to rolling file appender so we can do some custom task (like encrypt and zip) when a file is rolled?

        Or are you planning to add zip+encrypt of old logs (basically we do not want log4j to delete any of its logs, just want to archive them till they are manually deleted)

        Show
        Tushar added a comment - see http://sel2in.com/pages/log4j/2012march10.txt for mvn clean install output on your old 25 Oct 2011 build which I ran only today i see that the svn branch no longer exists do you have a new branch? sorry for the late reply but we temporarily fixed our prod problem by combing async with our rolling file appender Is there a way to add a listener to rolling file appender so we can do some custom task (like encrypt and zip) when a file is rolled? Or are you planning to add zip+encrypt of old logs (basically we do not want log4j to delete any of its logs, just want to archive them till they are manually deleted)
        Hide
        Tushar added a comment -

        Summary of improvement:
        A log roll over that makes a new file every x MB/ KB (and a new one every day & on app restart) but the current file is archived with a full date-time-random-int-to-make unique stamp
        Optionally zips and encrypts the old file into a sub folder (which is configurable on date like yyyy/mm/dd )

        Appender should write out buffer in parts instead of blocking for a long time (emptying full buffer when buffer is full)

        Show
        Tushar added a comment - Summary of improvement: A log roll over that makes a new file every x MB/ KB (and a new one every day & on app restart) but the current file is archived with a full date-time- random-int-to-make unique stamp Optionally zips and encrypts the old file into a sub folder (which is configurable on date like yyyy/mm/dd ) Appender should write out buffer in parts instead of blocking for a long time (emptying full buffer when buffer is full)
        Hide
        Ralph Goers added a comment -

        Log4j2 was moved from my experimental branch to a proper trunk at https://svn.apache.org/repos/asf/logging/log4j/log4j2/trunk. Since your post in October the Flume appender was also broken out into a subproject (actually two - one for Flume 0.9.4 and the other for Flume 1.0).

        I see from the output in your link above you have 9 unit test failures. I'd appreciate it if you could download the latest source and run the build again and then create a new Jira issue for those and attach the output as well as the output from each of the unit tests (you will find them in log4j2-core/target/surefire-reports/ with the file name matching the full class name and an extension of .txt).

        I'd like to see what you did for the Async appender. I've contemplated creating one.

        I'm sure it would be fairly easy to add a listener to the rolling file appender. I will look into that. The rolling file appender will already zip if the filePattern has a .zip extension.

        The current documentation is temporarily at http://people.apache.org/~rgoers/log4j2. That will move to the main Logging web site with the first release of Log4j2. (After looking at the documentation I notice that it doesn't mention that it will compress to zip or gzip if the file extension is specified in the filePattern. I will fix that). If you specify a very large number for "max" on the RolloverStrategy then none of the log files will be deleted.

        Log4j2 currently does everything you are mentioning except a) it doesn't encrypt and b) buffering is handled by whatever FileOutputStream does. Again, these can be tailored if required.

        Show
        Ralph Goers added a comment - Log4j2 was moved from my experimental branch to a proper trunk at https://svn.apache.org/repos/asf/logging/log4j/log4j2/trunk . Since your post in October the Flume appender was also broken out into a subproject (actually two - one for Flume 0.9.4 and the other for Flume 1.0). I see from the output in your link above you have 9 unit test failures. I'd appreciate it if you could download the latest source and run the build again and then create a new Jira issue for those and attach the output as well as the output from each of the unit tests (you will find them in log4j2-core/target/surefire-reports/ with the file name matching the full class name and an extension of .txt). I'd like to see what you did for the Async appender. I've contemplated creating one. I'm sure it would be fairly easy to add a listener to the rolling file appender. I will look into that. The rolling file appender will already zip if the filePattern has a .zip extension. The current documentation is temporarily at http://people.apache.org/~rgoers/log4j2 . That will move to the main Logging web site with the first release of Log4j2. (After looking at the documentation I notice that it doesn't mention that it will compress to zip or gzip if the file extension is specified in the filePattern. I will fix that). If you specify a very large number for "max" on the RolloverStrategy then none of the log files will be deleted. Log4j2 currently does everything you are mentioning except a) it doesn't encrypt and b) buffering is handled by whatever FileOutputStream does. Again, these can be tailored if required.
        Hide
        Tushar added a comment -

        hgas the pom file not got install any more? what is the command to give to build and install?

        This is form my prompt:

        G:\prog\j\log4j2>G:\prog\j\maven-1.1\bin\maven clean install

        ....banner

        build:start:

        clean:clean:
        xdoc:clean:

        -------------------------------------------------------------------------------
        >> Goal 'install' does not exist in this project.
        -------------------------------------------------------------------------------
        BUILD FAILED
        -------------------------------------------------------------------------------
        Total time : 1 seconds
        Finished at : Monday, March 19, 2012 2:23:12 AM GMT+05:30
        Final Memory : 3M/15M
        -------------------------------------------------------------------------------

        Show
        Tushar added a comment - hgas the pom file not got install any more? what is the command to give to build and install? This is form my prompt: G:\prog\j\log4j2>G:\prog\j\maven-1.1\bin\maven clean install ....banner build:start: clean:clean: xdoc:clean: ------------------------------------------------------------------------------- >> Goal 'install' does not exist in this project. ------------------------------------------------------------------------------- BUILD FAILED ------------------------------------------------------------------------------- Total time : 1 seconds Finished at : Monday, March 19, 2012 2:23:12 AM GMT+05:30 Final Memory : 3M/15M -------------------------------------------------------------------------------
        Hide
        Ralph Goers added a comment -

        If you are trying to build Log4j2 then you need Maven 2 or 3. The command is mvn clean install.

        Show
        Ralph Goers added a comment - If you are trying to build Log4j2 then you need Maven 2 or 3. The command is mvn clean install.
        Hide
        Tushar added a comment - - edited

        https://issues.apache.org/jira/browse/LOG4J2-59 issue for latest build

        i will show you the async appender sample tomorrow someone else wrote it but uses the standard async logger and attaches the rolling logger to that- what would be good to have is when it gets very busy to increse its buffer or drop items OR block when TOO big (optimal and MAX size) - all as properties

        right now are prod breaks as threads get blocked when too much logging happens while many txns are happening ... i would rather see log4j stop logging (as a preference) then block and causing are server to crash. the only long term fix we found was to log less (we were logging too many times at places and at one meta data reload - logging too long a string - a whole hash map made of a whole 'static' data table)

        but at other times too we have seen thread dead lock and the histo and thread dump via jstack says there were log4j objects .... i can copy paste the exact error msg if you want to see that too

        Show
        Tushar added a comment - - edited https://issues.apache.org/jira/browse/LOG4J2-59 issue for latest build i will show you the async appender sample tomorrow someone else wrote it but uses the standard async logger and attaches the rolling logger to that- what would be good to have is when it gets very busy to increse its buffer or drop items OR block when TOO big (optimal and MAX size) - all as properties right now are prod breaks as threads get blocked when too much logging happens while many txns are happening ... i would rather see log4j stop logging (as a preference) then block and causing are server to crash. the only long term fix we found was to log less (we were logging too many times at places and at one meta data reload - logging too long a string - a whole hash map made of a whole 'static' data table) but at other times too we have seen thread dead lock and the histo and thread dump via jstack says there were log4j objects .... i can copy paste the exact error msg if you want to see that too
        Hide
        Ralph Goers added a comment -

        Do you have an update to this record? The asynch appender and any other issues you are having should be opened as separate issues.

        Show
        Ralph Goers added a comment - Do you have an update to this record? The asynch appender and any other issues you are having should be opened as separate issues.
        Hide
        Tushar added a comment -

        i have not tested with log4j2

        saw an email where you answered a query about how to use log4j2 in a new project and how to use it in an existing project with the bridge

        Can you make the wiki pages for that? I will test after that.

        For the async appender sample look here http://sel2in.com/pages/log4j/TespAppLog4j1.zip or the attachment TespAppLog4j1.zip

        Show
        Tushar added a comment - i have not tested with log4j2 saw an email where you answered a query about how to use log4j2 in a new project and how to use it in an existing project with the bridge Can you make the wiki pages for that? I will test after that. For the async appender sample look here http://sel2in.com/pages/log4j/TespAppLog4j1.zip or the attachment TespAppLog4j1.zip
        Hide
        Tushar added a comment -

        Sample

        Show
        Tushar added a comment - Sample
        Hide
        Ralph Goers added a comment -

        I have added an AsynchAppender and I have updated the documentation to include information about migrating from Log4j 1.x. Please let me know if there is anything left to do on this issue.

        Show
        Ralph Goers added a comment - I have added an AsynchAppender and I have updated the documentation to include information about migrating from Log4j 1.x. Please let me know if there is anything left to do on this issue.
        Hide
        Tushar added a comment - - edited
        • If we dont already have a way to roll over on start up. i think its there.
        • and would be great if the out file can be dynamic - based on a java simple date format in some parts so we have a root path + pattern

        /archive/path/

        {YYYY}

        /

        {MM}

        /

        Need to make a simple date format (only the parts in side {} are to be parsed by simple date format), parse the date and then make the file object, call mkdirs() this way we don't need any external job to organise the archive except download and delete files from server !

        besides this i will give what is there a test and let close this when i get a chance to get from trunk ...

        Show
        Tushar added a comment - - edited If we dont already have a way to roll over on start up. i think its there. and would be great if the out file can be dynamic - based on a java simple date format in some parts so we have a root path + pattern /archive/path/ {YYYY} / {MM} / Need to make a simple date format (only the parts in side {} are to be parsed by simple date format), parse the date and then make the file object, call mkdirs() this way we don't need any external job to organise the archive except download and delete files from server ! besides this i will give what is there a test and let close this when i get a chance to get from trunk ...
        Hide
        Ralph Goers added a comment -

        http://people.apache.org/~rgoers/log4j2/manual/appenders.html#RollingFileAppender mentions the OnStartupTriggeringPolicy.

        The example in that same documentation shows how to use $

        {date:format}

        to add date elements to your path structure. For the above do:

        /archive/path/$$

        {date:yyyy/MM}

        or /archive/path/$$

        {date:yyyy}

        /$$

        {date:MM}
        Show
        Ralph Goers added a comment - http://people.apache.org/~rgoers/log4j2/manual/appenders.html#RollingFileAppender mentions the OnStartupTriggeringPolicy. The example in that same documentation shows how to use $ {date:format} to add date elements to your path structure. For the above do: /archive/path/$$ {date:yyyy/MM} or /archive/path/$$ {date:yyyy} /$$ {date:MM}

          People

          • Assignee:
            Unassigned
            Reporter:
            Tushar
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development