Flume
  1. Flume
  2. FLUME-533

TailSource line length limitation

    Details

      Description

      It appears that having a line with length of 32767 bytes or more
      throws the TailSource into a, um, tail spin. I've set
      flume.event.max.size.bytes to 1000000, so that shouldn't be the
      limitation. Simple steps to reproduce:

      flume-0.9.3rc1> ruby -e 'puts "x"*32767' > /tmp/log
      flume-0.9.3rc1> bin/flume dump 'tail("/tmp/log")'

      It appears that the source sends an incomplete message to the sink,
      and then continuously retries to send it.

      I've attached a patch that creates a buffer of size flume.event.max.size.bytes,
      and if it detects a line longer than this, it gets thrown away. It's been working
      for our systems for a week, but I haven't had time to write a unit test and test
      the edge cases.

        Activity

        Disabled imported user created issue -
        Jonathan Hsieh made changes -
        Field Original Value New Value
        Fix Version/s v0.9.4 [ 10050 ]
        Jonathan Hsieh made changes -
        Status Open [ 1 ] Patch Available [ 10000 ]
        Jonathan Hsieh made changes -
        Assignee Brent Halsey [ flume_brenthalsey ]
        Anonymous made changes -
        Status Patch Available [ 10000 ] Open [ 1 ]
        Anonymous made changes -
        Status Open [ 1 ] Patch Available [ 10000 ]
        Jonathan Hsieh made changes -
        Fix Version/s v0.9.5 [ 10090 ]
        Fix Version/s v0.9.4 [ 10050 ]
        Mark Thomas made changes -
        Project Import Tue Aug 02 16:57:12 UTC 2011 [ 1312304232406 ]
        Jonathan Hsieh made changes -
        Assignee Disabled imported user [ flume_brenthalsey ] Jonathan Hsieh [ jmhsieh ]
        Brent Halsey made changes -
        Attachment 0001-Fix-for-FLUME-533-with-tests.patch [ 12502517 ]
        Brent Halsey made changes -
        Jonathan Hsieh made changes -
        Assignee Jonathan Hsieh [ jmhsieh ]
        Ashish Paliwal made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Disabled imported user
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development