Flume
  1. Flume
  2. FLUME-1116

Extra event created for max payload size of 2500 bytes in Flume syslogtcp source

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.2.0
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Environment:

      [RHEL 6.2 64-bit]

      Description

      When using the syslogtcp source, I use telnet to connect to the port and send an event with a leading <1> priority, followed by exactly 2500 bytes (the hardcoded max msg size). An extra event is created that contains only a single newline char.

      The config I used is:

      1. a = agent
      2. c = channel
      3. r = source
      4. k = sink

      a1.channels = c1 c2 c3
      a1.sources = r1
      a1.sinks = k1 k2 k3

      1. Channels
        a1.channels.c1.type = MEMORY
        a1.channels.c1.capacity = 100
        a1.channels.c1.transactionCapacity = 100
        a1.channels.c1.keep-alive = 3

      a1.channels.c2.type = MEMORY
      a1.channels.c2.capacity = 100
      a1.channels.c2.transactionCapacity = 100
      a1.channels.c2.keep-alive = 3

      a1.channels.c3.type = MEMORY
      a1.channels.c3.capacity = 100
      a1.channels.c3.transactionCapacity = 100
      a1.channels.c3.keep-alive = 3

      a1.sources.r1.type = SYSLOGTCP
      a1.sources.r1.channels = c1 c2 c3
      a1.sources.r1.selector.type = multiplexing
      a1.sources.r1.selector.header = flume.syslog.status
      a1.sources.r1.selector.default = c1
      a1.sources.r1.selector.mapping.Invalid = c2
      a1.sources.r1.selector.mapping.Incomplete = c3
      a1.sources.r1.host = 0.0.0.0
      a1.sources.r1.port = 1473

      a1.sinks.k1.type = FILE_ROLL
      a1.sinks.k1.channel = c1
      a1.sinks.k1.sink.directory = /home/will/files/good-events
      a1.sinks.k1.sink.rollInterval = 0

      a1.sinks.k2.type = FILE_ROLL
      a1.sinks.k2.channel = c2
      a1.sinks.k2.sink.directory = /home/will/files/invalid-events
      a1.sinks.k2.sink.rollInterval = 0

      a1.sinks.k3.type = FILE_ROLL
      a1.sinks.k3.channel = c3
      a1.sinks.k3.sink.directory = /home/will/files/incomplete-events
      a1.sinks.k3.sink.rollInterval = 0

      ...and the telnet command I used is:
      telnet <host> 1473

      ..and the syslog event I pasted is shown below (<1> followed by exactly 2500 'h' chars), after which I press <RET> to send it:

      <1>hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

      1. FLUME-1116-1.patch
        1.0 kB
        Hari Shreedharan

        Activity

        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4689/
        -----------------------------------------------------------

        Review request for Flume.

        Summary
        -------

        Ignore new line, if we are in start mode.

        This addresses bug FLUME-1116.
        https://issues.apache.org/jira/browse/FLUME-1116

        Diffs


        flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java c42b5f5

        Diff: https://reviews.apache.org/r/4689/diff

        Testing
        -------

        Thanks,

        Hari

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4689/ ----------------------------------------------------------- Review request for Flume. Summary ------- Ignore new line, if we are in start mode. This addresses bug FLUME-1116 . https://issues.apache.org/jira/browse/FLUME-1116 Diffs flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java c42b5f5 Diff: https://reviews.apache.org/r/4689/diff Testing ------- Thanks, Hari
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4689/#review6848
        -----------------------------------------------------------

        Ship it!

        lgtm

        • Prasad

        On 2012-04-10 05:21:46, Hari Shreedharan wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4689/

        -----------------------------------------------------------

        (Updated 2012-04-10 05:21:46)

        Review request for Flume.

        Summary

        -------

        Ignore new line, if we are in start mode.

        This addresses bug FLUME-1116.

        https://issues.apache.org/jira/browse/FLUME-1116

        Diffs

        -----

        flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java c42b5f5

        Diff: https://reviews.apache.org/r/4689/diff

        Testing

        -------

        Thanks,

        Hari

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4689/#review6848 ----------------------------------------------------------- Ship it! lgtm Prasad On 2012-04-10 05:21:46, Hari Shreedharan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4689/ ----------------------------------------------------------- (Updated 2012-04-10 05:21:46) Review request for Flume. Summary ------- Ignore new line, if we are in start mode. This addresses bug FLUME-1116 . https://issues.apache.org/jira/browse/FLUME-1116 Diffs ----- flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java c42b5f5 Diff: https://reviews.apache.org/r/4689/diff Testing ------- Thanks, Hari
        Hide
        Prasad Mujumdar added a comment -

        Patch committed to trunk
        Thanks Hari!

        Show
        Prasad Mujumdar added a comment - Patch committed to trunk Thanks Hari!
        Hide
        Hudson added a comment -

        Integrated in flume-trunk #169 (See https://builds.apache.org/job/flume-trunk/169/)
        FLUME-1116:Extra event created for max payload size of 2500 bytes in Flume syslogtcp source
        (Hari Shreedharan via Prasad Mujumdar) (Revision 1324815)

        Result = SUCCESS
        prasadm : http://svn.apache.org/viewvc/?view=rev&rev=1324815
        Files :

        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java
        Show
        Hudson added a comment - Integrated in flume-trunk #169 (See https://builds.apache.org/job/flume-trunk/169/ ) FLUME-1116 :Extra event created for max payload size of 2500 bytes in Flume syslogtcp source (Hari Shreedharan via Prasad Mujumdar) (Revision 1324815) Result = SUCCESS prasadm : http://svn.apache.org/viewvc/?view=rev&rev=1324815 Files : /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java

          People

          • Assignee:
            Hari Shreedharan
            Reporter:
            Will McQueen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development