Uploaded image for project: 'Log4j 2'
  1. Log4j 2
  2. LOG4J2-1137

Allow events to be buffered until something triggers and causes them to be logged.

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4
    • Fix Version/s: None
    • Component/s: Core
    • Labels:
      None

      Description

      In many cases logging is filtered in production to reduce the overhead of logging. Unfortunately, this means that when an error occurs there may not be enough logging information available to diagnose the problem. On the other hand, if filtering is minimized then the amount of data being logged can be overwhelming.

      This enhancement would allow log events to be written to a buffer and not processed any further until a triggering event occurs. Once that trigger occurs all the events in the buffer would be logged in sequence and pass through the normal filtering process. Some log events that are deemed "special" would need to be allowed to bypass the buffer and immediately be processed.

      One way to accomplish this would be to add a new LogBuffer component to the configuration. If configured, the ReliabilityStrategies could be modified to work with the LogBuffer or a BufferingReliabilityWrapperStrategy could be created to wrap the existing strategies.

      Note - no code has been written for this yet. I'd want to figure out what the configuration might look like before starting.

        Issue Links

          Activity

          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          Can we use AsyncAppender for this purpose?

          Show
          mikaelstaldal Mikael Ståldal added a comment - Can we use AsyncAppender for this purpose?
          Hide
          remkop@yahoo.com Remko Popma added a comment - - edited

          So, the idea is to have a small ring buffer with the most recent LogEvent objects. Once the ring buffer is full, new log events replace the oldest log events in the ring buffer until a new log event contains some trigger that causes the full contents of the ring buffer to be drained to the target appender(s).

          What happens with new log events that arrive after a triggering event? Keep draining to the target appenders? (If so, keep draining only for some limited number of events, or until some "untrigger" event, or forever?)

          Since performance is a big part of the motivation for this idea, it would make sense to use a non-blocking data structure similar to the RingBuffer used in the LMAX Disruptor. In fact, I think the Disruptor library contains components to achieve this out of the box. Or does this need to be synchronous? (Synchronous here would mean that the application thread delivering the triggering event will block until the full content of the ring buffer has been processed by all appenders.)

          Show
          remkop@yahoo.com Remko Popma added a comment - - edited So, the idea is to have a small ring buffer with the most recent LogEvent objects. Once the ring buffer is full, new log events replace the oldest log events in the ring buffer until a new log event contains some trigger that causes the full contents of the ring buffer to be drained to the target appender(s). What happens with new log events that arrive after a triggering event? Keep draining to the target appenders? (If so, keep draining only for some limited number of events, or until some "untrigger" event, or forever?) Since performance is a big part of the motivation for this idea, it would make sense to use a non-blocking data structure similar to the RingBuffer used in the LMAX Disruptor. In fact, I think the Disruptor library contains components to achieve this out of the box. Or does this need to be synchronous? (Synchronous here would mean that the application thread delivering the triggering event will block until the full content of the ring buffer has been processed by all appenders.)
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          The AsyncAppender wouldn't really do what I am proposing. All the events would have already gone through filtering before landing on the AsyncAppender.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - The AsyncAppender wouldn't really do what I am proposing. All the events would have already gone through filtering before landing on the AsyncAppender.
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          I was thinking that when a triggering event occurs the RingBuffer would be swapped out and then all the events could be asynchronously logged. Ideally, the triggering LogEvent could be available for use in filtering events. However, this opens up some issues.
          1. What if a second triggering event occurs that overlaps with the first. If some events are filtered by the first log event they wouldn't be in the output for the second.
          2. What if triggering events happen too quickly. There should be some way to detect this and just bypass the buffer for some amount of time.

          Yes, I was thinking the Distruptor's RingBuffer might work for this.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - I was thinking that when a triggering event occurs the RingBuffer would be swapped out and then all the events could be asynchronously logged. Ideally, the triggering LogEvent could be available for use in filtering events. However, this opens up some issues. 1. What if a second triggering event occurs that overlaps with the first. If some events are filtered by the first log event they wouldn't be in the output for the second. 2. What if triggering events happen too quickly. There should be some way to detect this and just bypass the buffer for some amount of time. Yes, I was thinking the Distruptor's RingBuffer might work for this.

            People

            • Assignee:
              Unassigned
              Reporter:
              ralph.goers@dslextreme.com Ralph Goers
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development