Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2217

Preserve priority, timestamp and hostname fields in MultiportSyslogTcp and Udp sources

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v1.5.0
    • Fix Version/s: v1.5.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Flume-1666 added the ability to preserve timestamp and hostname fields of a syslog message. We should also add this property to the MultiportSyslogTcp Source and the SyslogUdp sources.

      1. FLUME-2217.1.patch
        23 kB
        Jeff Lord
      2. FLUME-2217.2.patch
        23 kB
        Jeff Lord
      3. FLUME-2217.3.patch
        28 kB
        Jeff Lord
      4. FLUME-2217.6.patch
        30 kB
        Jeff Lord

        Issue Links

          Activity

          Hide
          jlord Jeff Lord added a comment -

          Here is a first pass at this.
          Please not that the functionality of all 3 sources was modified slightly such that with this patch we will now preserve the syslog priority as well as the timestamp and header. e.g.

          <10>2013-10-31T17:36:27.381-07:00 localhost.localdomain test UDP syslog data

          Show
          jlord Jeff Lord added a comment - Here is a first pass at this. Please not that the functionality of all 3 sources was modified slightly such that with this patch we will now preserve the syslog priority as well as the timestamp and header. e.g. <10>2013-10-31T17:36:27.381-07:00 localhost.localdomain test UDP syslog data
          Hide
          jlord Jeff Lord added a comment -

          Rev 2 based on feedbak from Mike Percy on reviewboard.

          Show
          jlord Jeff Lord added a comment - Rev 2 based on feedbak from Mike Percy on reviewboard.
          Hide
          jlord Jeff Lord added a comment -

          Removed baosRaw and extra byte array.
          Modified the initial parsing to include Priority e.g. <10>
          Modified the regex to account for this and we should be good now.
          Was able to build successfully.
          Please let me know if there is anything else I can do to get this committed.

          Show
          jlord Jeff Lord added a comment - Removed baosRaw and extra byte array. Modified the initial parsing to include Priority e.g. <10> Modified the regex to account for this and we should be good now. Was able to build successfully. Please let me know if there is anything else I can do to get this committed.
          Hide
          mpercy Mike Percy added a comment -

          +1 LGTM. I am going to commit this soon.

          Show
          mpercy Mike Percy added a comment - +1 LGTM. I am going to commit this soon.
          Hide
          mpercy Mike Percy added a comment -

          Pushed to trunk and flume-1.5 branches. Thanks for the patch Jeff!

          Show
          mpercy Mike Percy added a comment - Pushed to trunk and flume-1.5 branches. Thanks for the patch Jeff!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in flume-trunk #525 (See https://builds.apache.org/job/flume-trunk/525/)
          FLUME-2217. Add option to preserve all Syslog headers in syslog sources (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9790ca7587060285efa4ae64591cea17dd3f00cf)

          • flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java
          • flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java
          • flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogParser.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-core/src/test/java/org/apache/flume/source/TestMultiportSyslogTCPSource.java
          • flume-ng-core/src/main/java/org/apache/flume/source/SyslogParser.java
          • flume-ng-core/src/main/java/org/apache/flume/source/MultiportSyslogTCPSource.java
          • flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java
          • flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java
          • flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogTcpSource.java
          • flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in flume-trunk #525 (See https://builds.apache.org/job/flume-trunk/525/ ) FLUME-2217 . Add option to preserve all Syslog headers in syslog sources (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9790ca7587060285efa4ae64591cea17dd3f00cf ) flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogParser.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-core/src/test/java/org/apache/flume/source/TestMultiportSyslogTCPSource.java flume-ng-core/src/main/java/org/apache/flume/source/SyslogParser.java flume-ng-core/src/main/java/org/apache/flume/source/MultiportSyslogTCPSource.java flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogTcpSource.java flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java
          Hide
          xnag Xuri Nagarin added a comment -

          Including the PRI part in the message is unusual to say the least. None of the other major syslog processors (syslog-ng and rsyslog), by default, write out the PRI part. I think first you need to figure out what RFC are you looking to comply with? If it is 3164, which is still the most used, then you need to set separate flags for PRI and HEADER because as per 3164:
          "The full format of a syslog message seen on the wire has three discernable parts. The first part is called the PRI, the second part is the HEADER, and the third part is the MSG."

          "keepFields" clubs the PRI and HEADER into one field. Instead, you should use "keepPri" and "keepHeader".

          Show
          xnag Xuri Nagarin added a comment - Including the PRI part in the message is unusual to say the least. None of the other major syslog processors (syslog-ng and rsyslog), by default, write out the PRI part. I think first you need to figure out what RFC are you looking to comply with? If it is 3164, which is still the most used, then you need to set separate flags for PRI and HEADER because as per 3164: "The full format of a syslog message seen on the wire has three discernable parts. The first part is called the PRI, the second part is the HEADER, and the third part is the MSG." "keepFields" clubs the PRI and HEADER into one field. Instead, you should use "keepPri" and "keepHeader".
          Hide
          Lagarutte Jean lagarutte added a comment -

          hi
          i have the same problem.
          with flume 1.5 and keepFiels=false, the priority is added to the body.
          example :
          all my body message begins with <14> text ..... for example
          i don't have the problem with flume 1.4

          i use log4j 1.2.17.
          i don't have the problem with logback because the syslogudp appender results in : fields.flume.syslog.status = invalid
          so it's better with logback sending bad syslog message than log4j sending good log4j message

          what can i do ?
          is it possible to retrieve the same behavior as in flume 1.4 ?

          Show
          Lagarutte Jean lagarutte added a comment - hi i have the same problem. with flume 1.5 and keepFiels=false, the priority is added to the body. example : all my body message begins with <14> text ..... for example i don't have the problem with flume 1.4 i use log4j 1.2.17. i don't have the problem with logback because the syslogudp appender results in : fields.flume.syslog.status = invalid so it's better with logback sending bad syslog message than log4j sending good log4j message what can i do ? is it possible to retrieve the same behavior as in flume 1.4 ?
          Hide
          hshreedharan Hari Shreedharan added a comment -

          There is a fix to this that went into Flume, which will be part of the next release (hopefully soon).

          Show
          hshreedharan Hari Shreedharan added a comment - There is a fix to this that went into Flume, which will be part of the next release (hopefully soon).
          Hide
          Lagarutte Jean lagarutte added a comment -

          Good news !
          Do you know where i can find the fix to build a custom release ?

          Show
          Lagarutte Jean lagarutte added a comment - Good news ! Do you know where i can find the fix to build a custom release ?
          Hide
          hshreedharan Hari Shreedharan added a comment -

          Clone https://git-wip-us.apache.org/repos/asf?p=flume.git and build the trunk branch.

          Show
          hshreedharan Hari Shreedharan added a comment - Clone https://git-wip-us.apache.org/repos/asf?p=flume.git and build the trunk branch.

            People

            • Assignee:
              jlord Jeff Lord
              Reporter:
              jlord Jeff Lord
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development