Flume
  1. Flume
  2. FLUME-533

TailSource line length limitation

    Details

      Description

      It appears that having a line with length of 32767 bytes or more
      throws the TailSource into a, um, tail spin. I've set
      flume.event.max.size.bytes to 1000000, so that shouldn't be the
      limitation. Simple steps to reproduce:

      flume-0.9.3rc1> ruby -e 'puts "x"*32767' > /tmp/log
      flume-0.9.3rc1> bin/flume dump 'tail("/tmp/log")'

      It appears that the source sends an incomplete message to the sink,
      and then continuously retries to send it.

      I've attached a patch that creates a buffer of size flume.event.max.size.bytes,
      and if it detects a line longer than this, it gets thrown away. It's been working
      for our systems for a week, but I haven't had time to write a unit test and test
      the edge cases.

        Activity

          People

          • Assignee:
            Unassigned
            Reporter:
            Disabled imported user
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development