Flume
  1. Flume
  2. FLUME-533

TailSource line length limitation

    Details

      Description

      It appears that having a line with length of 32767 bytes or more
      throws the TailSource into a, um, tail spin. I've set
      flume.event.max.size.bytes to 1000000, so that shouldn't be the
      limitation. Simple steps to reproduce:

      flume-0.9.3rc1> ruby -e 'puts "x"*32767' > /tmp/log
      flume-0.9.3rc1> bin/flume dump 'tail("/tmp/log")'

      It appears that the source sends an incomplete message to the sink,
      and then continuously retries to send it.

      I've attached a patch that creates a buffer of size flume.event.max.size.bytes,
      and if it detects a line longer than this, it gets thrown away. It's been working
      for our systems for a week, but I haven't had time to write a unit test and test
      the edge cases.

        Activity

        Hide
        Brent Halsey added a comment -

        patch for trunk of https://github.com/apache/flume.git

        Brent Halsey and James Estes

        Show
        Brent Halsey added a comment - patch for trunk of https://github.com/apache/flume.git Brent Halsey and James Estes
        Hide
        Brent Halsey added a comment -

        patch for 0.9.4-branch in http://github.com/cloudera/flume.git

        -Brent Halsey and James Estes

        Show
        Brent Halsey added a comment - patch for 0.9.4-branch in http://github.com/cloudera/flume.git -Brent Halsey and James Estes
        Hide
        Jonathan Hsieh added a comment -

        I ported patch and created a test that is similar to the ruby script mentioned above. The entire event seems to have gotten dropped in this case. I'm editing to make it take the first maxeventsize bytes and then truncate the rest until a new event delimiter is reached.

        Show
        Jonathan Hsieh added a comment - I ported patch and created a test that is similar to the ruby script mentioned above. The entire event seems to have gotten dropped in this case. I'm editing to make it take the first maxeventsize bytes and then truncate the rest until a new event delimiter is reached.
        Hide
        Jonathan Hsieh added a comment -

        Hey Brent,

        I'm trying to catch up on the patch backlog and having problems trying to apply this patch. Do you have a github link or could you regenerate it?

        One comment from the patch is that to commit it we need some unit tests to be added. A simple test where a line is to long and it behaving in a particular way would be sufficient.

        Jon.

        Show
        Jonathan Hsieh added a comment - Hey Brent, I'm trying to catch up on the patch backlog and having problems trying to apply this patch. Do you have a github link or could you regenerate it? One comment from the patch is that to commit it we need some unit tests to be added. A simple test where a line is to long and it behaving in a particular way would be sufficient. Jon.
        Hide
        Disabled imported user added a comment -

        *I grant license to Cloudera for inclusion of my patches submitted to any
        Cloudera projects in Cloudera works (as per the Apache License §5).*

        Show
        Disabled imported user added a comment - *I grant license to Cloudera for inclusion of my patches submitted to any Cloudera projects in Cloudera works (as per the Apache License §5).*
        Hide
        Jonathan Hsieh added a comment -

        Brent,

        Take a look at this so we can proceed: https://github.com/cloudera/flume/wiki/HowToContribute

        At the least I need you to grant us rights to use the patch (oh legal people.. ).

        Thanks,
        Jon.

        Show
        Jonathan Hsieh added a comment - Brent, Take a look at this so we can proceed: https://github.com/cloudera/flume/wiki/HowToContribute At the least I need you to grant us rights to use the patch (oh legal people.. ). Thanks, Jon.
        Hide
        Alex Baranau added a comment -

        Could you please create a review request at http://review.cloudera.org for the patch?

        Thank you in advance.

        Show
        Alex Baranau added a comment - Could you please create a review request at http://review.cloudera.org for the patch? Thank you in advance.

          People

          • Assignee:
            Unassigned
            Reporter:
            Disabled imported user
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development