Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2041

Regex interceptor does not populate headers when using Log4j 2 FlumeAppender

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Not A Problem
    • Affects Version/s: NG alpha 2
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:

      Description

      When using a source configured as follows:

      alerts.sources.a1.type = avro
      alerts.sources.a1.channels = c1
      alerts.sources.a1.bind = 0.0.0.0
      alerts.sources.a1.port = 4141

      alerts.sources.a1.interceptors = i1
      alerts.sources.a1.interceptors.i1.type = regex_extractor
      alerts.sources.a1.interceptors.i1.regex = com.foo.(\\S*).*
      alerts.sources.a1.interceptors.i1.serializers = s1
      alerts.sources.a1.interceptors.i1.serializers.s1.name = app

      ... with the following Log4j2 appender:

      <Flume name="eventLogger" suppressExceptions="false" compress="false">
      <Agent host="localhost" port="4141"/>
      <RFC5424Layout mdcId="bar" appName="com.foo.bar"/>
      </Flume>

      ... the interceptor will NOT insert the "app=bar" header. Instead I get a timestamp header. I understand the timestamp header, but I don't understand why the app header is missing, since the output of the logs looks something like this:

      <134>1 2013-05-08T16:11:28.836-04:00 host1 com.foo.bar - - [mdc@18060] Did something interesting

      In fact, if I configure a netcat source and send this string directly through the SAME interceptor, the header is added as expected.

      As a further test, I configured an Avro sink and forward the event to another agent, where I then set up the same interceptor. In that case, both the app and timestamp headers were populated.

      So to summarize:
      FlumeAppender -> Avro source -> Regex interceptor = timestamp header
      Raw string input -> NetCat source -> Regex interceptor = app header
      FlumeAppender -> Avro source -> Avro sink -> Avro source -> Regex interceptor = both headers

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                rstrickland Robbie Strickland
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: