Flume
  1. Flume
  2. FLUME-743

Log4J Avro Appender Ignores Logging Pattern

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: v0.9.4
    • Fix Version/s: v0.9.5
    • Component/s: Sinks+Sources
    • Labels:
    • Environment:

      Linux Ubuntu/Redhat

    • Release Note:
      Updated Log4J functionality to support layout formatting

      Description

      The log4J Avro appender does not use the pattern layout to format event messages. It only takes the message body, meaning that important information such as thread name, MDC/NDC values and stacktrace are lost.

      1. FLUME-743.diff
        8 kB
        Chris Jansen

        Activity

        Hide
        Chris Jansen added a comment -

        I've been working on a fix for this as per the flume-user mailing list here: http://mail-archives.apache.org/mod_mbox/incubator-flume-user/201108.mbox/%3CCANpnisohwf6UqEgPyMxcFBSMutxXTvzbUVXPKe4A4=xxk+5XnQ@mail.gmail.com%3E

        More than happy to be assigned this issue myself.

        Show
        Chris Jansen added a comment - I've been working on a fix for this as per the flume-user mailing list here: http://mail-archives.apache.org/mod_mbox/incubator-flume-user/201108.mbox/%3CCANpnisohwf6UqEgPyMxcFBSMutxXTvzbUVXPKe4A4=xxk+5XnQ@mail.gmail.com%3E More than happy to be assigned this issue myself.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1825/
        -----------------------------------------------------------

        Review request for Flume.

        Summary
        -------

        Fix to allow the flume log4j avro appender to include the log4j pattern layout.

        Updated the Log4JEventAdaptor constructor to require a layout object. As any class that uses the Log4JEventAdaptor extends the log4j AppenderSkeleton the layout object is always present. When a user wishes to configure a layout they use the setLayout method when setting up the log4j appender, if unset the layout will be defaulted to just displaying the event message body.

        Updated the getBody() method of the Log4JEventAdaptor class to apply the layout to the log4j event and output stacktrace.

        Updated FlumeLog4jAvroAppender to set up a default layout to display the message body if no user defined layout is specified.

        Added a new test to TestFlumeLog4jAvroAppender to test a logging pattern.

        Note that when using text sink only raw output will display stack traces on multiple lines, other formats display the newline escape characters.

        This addresses bug FLUME-743.
        https://issues.apache.org/jira/browse/FLUME-743

        Diffs


        /trunk/flume-core/src/main/java/com/cloudera/flume/handlers/log4j/FlumeSeqfileAppender.java 1170098
        /trunk/flume-core/src/main/java/com/cloudera/flume/handlers/log4j/Log4JAppenderEventSink.java 1170098
        /trunk/flume-core/src/main/java/com/cloudera/flume/handlers/log4j/Log4JEventAdaptor.java 1170098
        /trunk/flume-log4j-appender/src/main/java/com/cloudera/flume/log4j/appender/FlumeLog4jAvroAppender.java 1170098
        /trunk/flume-log4j-appender/src/test/java/com/cloudera/flume/log4j/appender/TestFlumeLog4jAvroAppender.java 1170098

        Diff: https://reviews.apache.org/r/1825/diff

        Testing
        -------

        Set up two physical nodes, one as the source and one as the collector. Set up the source with an avro logical node listening on port 12345 with the sink being the collector. Set up a text sink on the collector to log to a file with the output format set to raw: text("/tmp/test.log","raw").

        Use the test app code below to generate log events:

        import com.cloudera.flume.log4j.appender.FlumeLog4jAvroAppender;
        import org.apache.log4j.Logger;
        import org.apache.log4j.MDC;
        import org.apache.log4j.PatternLayout;

        import java.util.UUID;

        public class Main {
        static Logger log = Logger.getLogger(Main.class);

        public static void main(String[] args) {
        try {
        FlumeLog4jAvroAppender appender = new FlumeLog4jAvroAppender();
        appender.setHostname("localhost");
        appender.setPort(12345);
        appender.setLayout(new PatternLayout("%d [%c] (%t) <%X

        {user}

        %X

        {field}

        > %m"));
        appender.setReconnectAttempts(100);
        appender.activateOptions();

        log.addAppender(appender);

        MDC.put("user", "chris");
        while (true) {
        MDC.put("field", UUID.randomUUID().toString());
        log.info("Hello World");
        try

        { throw new Exception("Testing"); }

        catch (Exception e)

        { log.error("Gone wrong ", e); }

        }
        }
        catch (Exception e)

        { e.printStackTrace(); }

        }
        }

        Thanks,

        Chris

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1825/ ----------------------------------------------------------- Review request for Flume. Summary ------- Fix to allow the flume log4j avro appender to include the log4j pattern layout. Updated the Log4JEventAdaptor constructor to require a layout object. As any class that uses the Log4JEventAdaptor extends the log4j AppenderSkeleton the layout object is always present. When a user wishes to configure a layout they use the setLayout method when setting up the log4j appender, if unset the layout will be defaulted to just displaying the event message body. Updated the getBody() method of the Log4JEventAdaptor class to apply the layout to the log4j event and output stacktrace. Updated FlumeLog4jAvroAppender to set up a default layout to display the message body if no user defined layout is specified. Added a new test to TestFlumeLog4jAvroAppender to test a logging pattern. Note that when using text sink only raw output will display stack traces on multiple lines, other formats display the newline escape characters. This addresses bug FLUME-743 . https://issues.apache.org/jira/browse/FLUME-743 Diffs /trunk/flume-core/src/main/java/com/cloudera/flume/handlers/log4j/FlumeSeqfileAppender.java 1170098 /trunk/flume-core/src/main/java/com/cloudera/flume/handlers/log4j/Log4JAppenderEventSink.java 1170098 /trunk/flume-core/src/main/java/com/cloudera/flume/handlers/log4j/Log4JEventAdaptor.java 1170098 /trunk/flume-log4j-appender/src/main/java/com/cloudera/flume/log4j/appender/FlumeLog4jAvroAppender.java 1170098 /trunk/flume-log4j-appender/src/test/java/com/cloudera/flume/log4j/appender/TestFlumeLog4jAvroAppender.java 1170098 Diff: https://reviews.apache.org/r/1825/diff Testing ------- Set up two physical nodes, one as the source and one as the collector. Set up the source with an avro logical node listening on port 12345 with the sink being the collector. Set up a text sink on the collector to log to a file with the output format set to raw: text("/tmp/test.log","raw"). Use the test app code below to generate log events: import com.cloudera.flume.log4j.appender.FlumeLog4jAvroAppender; import org.apache.log4j.Logger; import org.apache.log4j.MDC; import org.apache.log4j.PatternLayout; import java.util.UUID; public class Main { static Logger log = Logger.getLogger(Main.class); public static void main(String[] args) { try { FlumeLog4jAvroAppender appender = new FlumeLog4jAvroAppender(); appender.setHostname("localhost"); appender.setPort(12345); appender.setLayout(new PatternLayout("%d [%c] (%t) <%X {user} %X {field} > %m")); appender.setReconnectAttempts(100); appender.activateOptions(); log.addAppender(appender); MDC.put("user", "chris"); while (true) { MDC.put("field", UUID.randomUUID().toString()); log.info("Hello World"); try { throw new Exception("Testing"); } catch (Exception e) { log.error("Gone wrong ", e); } } } catch (Exception e) { e.printStackTrace(); } } } Thanks, Chris
        Hide
        Chris Jansen added a comment -
        Show
        Chris Jansen added a comment - Code review: https://reviews.apache.org/r/1825/

          People

          • Assignee:
            Chris Jansen
            Reporter:
            Chris Jansen
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development