Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v0.9.4
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Release Note:
      A log4j Avro appender is now included with Flume.

      Description

      I'd love the ability to have log4j stream directly to a local Flume instance.

        Issue Links

          Activity

          Disabled imported user created issue -
          Hide
          Jonathan Hsieh added a comment -
          Show
          Jonathan Hsieh added a comment - An alternate approach: http://github.com/lenn0x/Scribe-log4j-Appender
          Jonathan Hsieh made changes -
          Field Original Value New Value
          Component/s Sinks+Sources [ 10041 ]
          Patrick Hunt made changes -
          Workflow jira [ 10059 ] flume-workflow [ 10165 ]
          Jonathan Hsieh made changes -
          Link This issue relates to FLUME-73 [ FLUME-73 ]
          Jonathan Hsieh made changes -
          Link This issue relates to FLUME-173 [ FLUME-173 ]
          Jonathan Hsieh made changes -
          Link This issue relates to FLUME-73 [ FLUME-73 ]
          Hide
          flume_psmith@apache.org added a comment -

          There's good reasons to have a log4j direct Flume source, because log4j has a fair bit of rich meta data (MDC, NDC and Properties) for a given logging event that is easily lost (or at least only munged into the Message if using the Tail source).

          As someone who uses log4j a lot (indeed, I am a committer for log4j, albeit out of touch with this project for a while) the MDC and Properties meta-data could easy map to the Event attributes.

          It seems a bit backwards to have log4j->scribe->flume->something. A more direct way from log4j->flume would be nice.

          Show
          flume_psmith@apache.org added a comment - There's good reasons to have a log4j direct Flume source, because log4j has a fair bit of rich meta data (MDC, NDC and Properties) for a given logging event that is easily lost (or at least only munged into the Message if using the Tail source). As someone who uses log4j a lot (indeed, I am a committer for log4j, albeit out of touch with this project for a while) the MDC and Properties meta-data could easy map to the Event attributes. It seems a bit backwards to have log4j->scribe->flume->something. A more direct way from log4j->flume would be nice.
          Hide
          Jonathan Hsieh added a comment -

          Hi Paul,

          I agree. Patches welcome obviously, but design help is welcome as well!

          Can you tell us what if any issues may occur if the log4j appender itself relies on code that uses log4j for logging? (Flume uses log4j fairly extensively) I seemed to have problems with early experiments – hints on the rules for this would be much appreciated!

          Jon.

          Show
          Jonathan Hsieh added a comment - Hi Paul, I agree. Patches welcome obviously, but design help is welcome as well! Can you tell us what if any issues may occur if the log4j appender itself relies on code that uses log4j for logging? (Flume uses log4j fairly extensively) I seemed to have problems with early experiments – hints on the rules for this would be much appreciated! Jon.
          Hide
          flume_psmith@apache.org added a comment -

          I'm actually interested in developing this myself (certainly happy to collaborate), as soon as I can get some more time to investigate the internals of the Flume source/sink APIs.

          Yes, the old chestnut/chicken-egg of how does a logging framework log itself, internally in the log4j 1.2 tree, we use the LogLog class, which outputs to System.out/err etc as appropriate. Output is controlled by the log4j.debug system property. That's really you're only hope with log4j 1.2 internally IIRC.

          Show
          flume_psmith@apache.org added a comment - I'm actually interested in developing this myself (certainly happy to collaborate), as soon as I can get some more time to investigate the internals of the Flume source/sink APIs. Yes, the old chestnut/chicken-egg of how does a logging framework log itself, internally in the log4j 1.2 tree, we use the LogLog class, which outputs to System.out/err etc as appropriate. Output is controlled by the log4j.debug system property. That's really you're only hope with log4j 1.2 internally IIRC.
          Jonathan Hsieh made changes -
          Link This issue relates to FLUME-283 [ FLUME-283 ]
          Jonathan Hsieh made changes -
          Link This issue blocks FLUME-179 [ FLUME-179 ]
          Hide
          Mathias Herberts added a comment -

          I have a somewhat working log4j appender. In order to retain all of the log4j details, the flume event body is really a serialized Thrift structure with placeholders for all of log4j's LoggingEvent fields (NDC, MDC, throwableInfo, locationInfo, ...).

          Don't know if that could be of help to any, but if so I'll gladly work towards opening it.

          Show
          Mathias Herberts added a comment - I have a somewhat working log4j appender. In order to retain all of the log4j details, the flume event body is really a serialized Thrift structure with placeholders for all of log4j's LoggingEvent fields (NDC, MDC, throwableInfo, locationInfo, ...). Don't know if that could be of help to any, but if so I'll gladly work towards opening it.
          Hide
          flume_psmith@apache.org added a comment -

          We should collaborate via a github project on this?

          Mathias, I'd love to see the patch for your code. What made you go down the Thrift way as opposed to packing this information as Flume Attributes? I had envisaged packing all the MDC/Properties with prefixed keys, so a "foo" MDC value would be keyed as a Flume attribute "MDC.foo" to disambiguate it. ThrowableInfo and LocationInfo are just strings, so could be in a well-named Attribute.

          Show
          flume_psmith@apache.org added a comment - We should collaborate via a github project on this? Mathias, I'd love to see the patch for your code. What made you go down the Thrift way as opposed to packing this information as Flume Attributes? I had envisaged packing all the MDC/Properties with prefixed keys, so a "foo" MDC value would be keyed as a Flume attribute "MDC.foo" to disambiguate it. ThrowableInfo and LocationInfo are just strings, so could be in a well-named Attribute.
          Hide
          Mathias Herberts added a comment -

          Lots of flume events we collect are actually stored in Hadoop SequenceFiles whose values are serialized thrift structs. This is mainly done so we can easily process all those data using Pig and a generic Thrift LoadFunc which nicely converts serialized Thrift to Pig tuples.

          Show
          Mathias Herberts added a comment - Lots of flume events we collect are actually stored in Hadoop SequenceFiles whose values are serialized thrift structs. This is mainly done so we can easily process all those data using Pig and a generic Thrift LoadFunc which nicely converts serialized Thrift to Pig tuples.
          Hide
          Jonathan Hsieh added a comment -

          FYI, there is some activity on FLUME-283. Not sure what the impact on this log4j work will be.

          Show
          Jonathan Hsieh added a comment - FYI, there is some activity on FLUME-283 . Not sure what the impact on this log4j work will be.
          Hide
          flume_psmith@apache.org added a comment -

          Don't think Flume itself switching to slf4j will affect things at all, this ticket is about an external system using log4j (perhaps even through slf4j) to communicate logging info into Flume as a Source right?

          Show
          flume_psmith@apache.org added a comment - Don't think Flume itself switching to slf4j will affect things at all, this ticket is about an external system using log4j (perhaps even through slf4j) to communicate logging info into Flume as a Source right?
          Hide
          Mathias Herberts added a comment -

          Exactly yes.

          On the log4j appender side, my main concern so far has been to select a buffering strategy which will ensure no log messages are lost in case Flume is unavailable. So far I did not come up with a satisfactory roadmap. Any thoughts on that?

          Show
          Mathias Herberts added a comment - Exactly yes. On the log4j appender side, my main concern so far has been to select a buffering strategy which will ensure no log messages are lost in case Flume is unavailable. So far I did not come up with a satisfactory roadmap. Any thoughts on that?
          Hide
          flume_psmith@apache.org added a comment -

          that's a difficult one, but is it needed? Do other external Sources use buffering? In many Flume scenarios (as I understand it), a source will be streaming logs, and if no-one is listening, such is life. Take syslog, an application writing to syslog which is connected to Flume, will not know if Flume is down.

          Is it just adding unnecessary complexity? Doesn't a local Flume agent buffer to defend against downstream problems? You may have a different use case, but I'd probably have a v1.0 log4j appender that just tries to reconnect periodically, something like what SocketAppender does which will drop logs if the remote end isn't there.

          Show
          flume_psmith@apache.org added a comment - that's a difficult one, but is it needed? Do other external Sources use buffering? In many Flume scenarios (as I understand it), a source will be streaming logs, and if no-one is listening, such is life. Take syslog, an application writing to syslog which is connected to Flume, will not know if Flume is down. Is it just adding unnecessary complexity? Doesn't a local Flume agent buffer to defend against downstream problems? You may have a different use case, but I'd probably have a v1.0 log4j appender that just tries to reconnect periodically, something like what SocketAppender does which will drop logs if the remote end isn't there.
          Hide
          Mathias Herberts added a comment -

          Ok, then I already have that kind of appender. I guess I could see how to contribute part of it (without the thrift LoggingEvent) to the flume community.

          Show
          Mathias Herberts added a comment - Ok, then I already have that kind of appender. I guess I could see how to contribute part of it (without the thrift LoggingEvent) to the flume community.
          Hide
          Jonathan Hsieh added a comment -

          Paul, Mathias,

          My recomendataion is to do the simple thing first (#1 below) – essentially what paul suggested, and the see if that is sufficient.

          I was thinking there are a few ways to potentially do this (each gets heavier and heavier weight).

          1) Implement a simple log4j appender that essentially has a flume sink that writes via some rpc mechanism to a node (possibly to a local agent or to a remote collector). This would be best effort and would have exponential backoff in the case of failures.
          1a) For an end-to-end reliability mode version, create a log4j appender that writes a flume agent's internal log files and moves them into the e2e's import dir. Periodically call import() on that wal manager to load the data so that it will deliver it.
          1b) For store on failure mode, use the same log4j appender that writes to flume agent's internal log files on network failures but otherwise relay to a local agent or remote collector.

          2) Embed a flume logical node into a log4j appender. This will have the logger inside a logger problem, and probably would heartbeat and thus would not not be reconfigurable (for failovers, etc).

          3) Embed a flume physical node into a log4j appender. This will have the logger inside a logger problem. It would heartbeat and could be reconfigurable but may be more heavy weight than desired.

          Jon.

          Show
          Jonathan Hsieh added a comment - Paul, Mathias, My recomendataion is to do the simple thing first (#1 below) – essentially what paul suggested, and the see if that is sufficient. I was thinking there are a few ways to potentially do this (each gets heavier and heavier weight). 1) Implement a simple log4j appender that essentially has a flume sink that writes via some rpc mechanism to a node (possibly to a local agent or to a remote collector). This would be best effort and would have exponential backoff in the case of failures. 1a) For an end-to-end reliability mode version, create a log4j appender that writes a flume agent's internal log files and moves them into the e2e's import dir. Periodically call import() on that wal manager to load the data so that it will deliver it. 1b) For store on failure mode, use the same log4j appender that writes to flume agent's internal log files on network failures but otherwise relay to a local agent or remote collector. 2) Embed a flume logical node into a log4j appender. This will have the logger inside a logger problem, and probably would heartbeat and thus would not not be reconfigurable (for failovers, etc). 3) Embed a flume physical node into a log4j appender. This will have the logger inside a logger problem. It would heartbeat and could be reconfigurable but may be more heavy weight than desired. Jon.
          E. Sammer made changes -
          Assignee Eric Sammer [ esammer ]
          E. Sammer made changes -
          Status Open [ 1 ] Patch Available [ 10000 ]
          Hide
          E. Sammer added a comment -
          Show
          E. Sammer added a comment - Review: https://review.cloudera.org/r/1665/
          Hide
          flume_petersontb added a comment - - edited

          NOTE: Passing messages using syslog protocol doesn't work well for longer messages. The syslog appender for Log4j is hardcoded to linewrap around 1024 characters in order to comply with the RFC. I got a sample program logging to syslog, picking it up with a syslogUdp source, with a JSON layout (to avoid new-lines in stack traces) only to find that anything but the smallest stack trace line-wrapped anyway. I can't see a way to reliably reconstruct the stack trace once it is wrapped and sent through the flume chain.

          Very interested in an Avro or Thrift appender for Log4j. See comment on FLUME-73

          Show
          flume_petersontb added a comment - - edited NOTE: Passing messages using syslog protocol doesn't work well for longer messages. The syslog appender for Log4j is hardcoded to linewrap around 1024 characters in order to comply with the RFC. I got a sample program logging to syslog, picking it up with a syslogUdp source, with a JSON layout (to avoid new-lines in stack traces) only to find that anything but the smallest stack trace line-wrapped anyway. I can't see a way to reliably reconstruct the stack trace once it is wrapped and sent through the flume chain. Very interested in an Avro or Thrift appender for Log4j. See comment on FLUME-73
          Hide
          flume_psmith@apache.org added a comment -

          Tyler is right, in that the latest log4j tip uses the 'spec compliant' syslog protocol field size. We've actually just patched it locally to allow a much large size. At the end of the day, doesn't Flume itself have a configured max size anyway? (I think 32k or something by default). Obviously that covers off the vast majority of logs, but still worth noting.

          Regarding the stack trace format, I created a JSON layout here: https://github.com/tallpsmith/json-log4j-layout that encodes the stack trace with literal character '\n' (not actual \n) so that any processing can replace that later on. This allows the event to stay on one line, plus allows the other meta-data like MDC that we use to be encoded too.

          Still, the Avro one Eric has done looks good, much nicer to avoid the extra layer in the stack.

          Show
          flume_psmith@apache.org added a comment - Tyler is right, in that the latest log4j tip uses the 'spec compliant' syslog protocol field size. We've actually just patched it locally to allow a much large size. At the end of the day, doesn't Flume itself have a configured max size anyway? (I think 32k or something by default). Obviously that covers off the vast majority of logs, but still worth noting. Regarding the stack trace format, I created a JSON layout here: https://github.com/tallpsmith/json-log4j-layout that encodes the stack trace with literal character '\n' (not actual \n) so that any processing can replace that later on. This allows the event to stay on one line, plus allows the other meta-data like MDC that we use to be encoded too. Still, the Avro one Eric has done looks good, much nicer to avoid the extra layer in the stack.
          Hide
          E. Sammer added a comment -

          Merged and pushed to master.

          Show
          E. Sammer added a comment - Merged and pushed to master.
          E. Sammer made changes -
          Status Patch Available [ 10000 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Release Note A log4j Avro appender is now included with Flume.
          Fix Version/s v0.9.4 [ 10050 ]
          Mark Thomas made changes -
          Project Import Tue Aug 02 16:57:12 UTC 2011 [ 1312304232406 ]
          Jonathan Hsieh made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              E. Sammer
              Reporter:
              Disabled imported user
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development