Flume
  1. Flume
  2. FLUME-1938

Flume UDP Source for Windows Events being trimmed at \n characters by SyslogUtils.extractEvent() function

    Details

    • Type: New Feature New Feature
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.3.1
    • Fix Version/s: None
    • Labels:
    • Environment:

      Ubuntu/Linux

    • Release Note:
      Hide
      This feature can be disabled if causes any issues with headers or header extraction process in flume syslog sources (both TCP and UDP).

      To enable multi-line configuration set the 'body.multiline' source context configuration to true e.g.

      agent.sources.syslogudp.body.multiline=true
      Show
      This feature can be disabled if causes any issues with headers or header extraction process in flume syslog sources (both TCP and UDP). To enable multi-line configuration set the 'body.multiline' source context configuration to true e.g. agent.sources.syslogudp.body.multiline=true

      Description

      Flume trims UDP syslog events originating from Kiwi Windows Event Log Forwarder at the first new-line character.

      I would like to contribute with a patch to allow configurable inclusion of new-line characters for SyslogUDP source in Flume

      Background:
      I am using Solarwinds/Kiwi Syslog's Windows Event Log Forwarder. Ref: http://www.solarwinds.com/kiwi-syslog-server.aspx

      The Log Forwarder sends CR+LF & horizontal tab characters in the UDP/TCP syslog events that it sends. Other than Kiwi Syslog this problem exists for syslog-ng on certain linux syslog events where CR+LF characters are sent in syslog events.

      Syslog-ng handles this problem by replacing these characters with \011 and \015\012 but Flume has no such charset customization available for SyslogUDP. SyslogTCP does have charset customization along with character replace functionality for header.

      1. TestSyslogUtils.java
        16 kB
        Ambud Sharma
      2. SyslogUtils.java
        13 kB
        Ambud Sharma
      3. SyslogUDPSource.java
        5 kB
        Ambud Sharma
      4. SyslogSourceConfigurationConstants.java
        2 kB
        Ambud Sharma
      5. FLUME-1938.patch-README.txt
        0.2 kB
        Ambud Sharma
      6. FLUME-1938.patch
        8 kB
        Ambud Sharma

        Activity

        Hide
        Ambud Sharma added a comment -

        Here are the patched files.

        I am not sure about how to commit to source repository. What credentials do I use to commit to git?

        Show
        Ambud Sharma added a comment - Here are the patched files. I am not sure about how to commit to source repository. What credentials do I use to commit to git?
        Hide
        Roshan Naik added a comment -

        You need to submit patches and not patched files. Committers will do the commit. After making the changes.. run :

        git diff > FLUME-1938.patch

        Upload the FLUME-1938.patch file here as attachement. Then click on 'Sumbit Patch' to indicate that this is ready for a committer to take a look at.

        Show
        Roshan Naik added a comment - You need to submit patches and not patched files. Committers will do the commit. After making the changes.. run : git diff > FLUME-1938 .patch Upload the FLUME-1938 .patch file here as attachement. Then click on 'Sumbit Patch' to indicate that this is ready for a committer to take a look at.
        Hide
        Ambud Sharma added a comment -

        The patch diff should be in by the end of day today. There are still some last minute changes to make sure TCP Syslog compatibility as well.

        Show
        Ambud Sharma added a comment - The patch diff should be in by the end of day today. There are still some last minute changes to make sure TCP Syslog compatibility as well.
        Hide
        Ambud Sharma added a comment -

        Patch for both TCP and UDP configurations. README uploaded for configuration file settings to enable and disable multi-line feature.

        Show
        Ambud Sharma added a comment - Patch for both TCP and UDP configurations. README uploaded for configuration file settings to enable and disable multi-line feature.
        Hide
        Ambud Sharma added a comment -

        Readme for configuration

        Show
        Ambud Sharma added a comment - Readme for configuration
        Hide
        Ambud Sharma added a comment -

        SyslogTCP configure method still doesn't capture the configuration changes so will not work until the configuration values is loaded for multi-line configuration when using syslog-tcp mode. Let me know if you would like me to add that line as well.

        Show
        Ambud Sharma added a comment - SyslogTCP configure method still doesn't capture the configuration changes so will not work until the configuration values is loaded for multi-line configuration when using syslog-tcp mode. Let me know if you would like me to add that line as well.
        Hide
        Mike Percy added a comment -

        Sorry, this one won't make it for Flume 1.4.0. Clearing fixVersion.

        Show
        Mike Percy added a comment - Sorry, this one won't make it for Flume 1.4.0. Clearing fixVersion.

          People

          • Assignee:
            Unassigned
            Reporter:
            Ambud Sharma
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 120h
              120h
              Remaining:
              Remaining Estimate - 120h
              120h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development