Flume
  1. Flume
  2. FLUME-533

TailSource line length limitation

    Details

      Description

      It appears that having a line with length of 32767 bytes or more
      throws the TailSource into a, um, tail spin. I've set
      flume.event.max.size.bytes to 1000000, so that shouldn't be the
      limitation. Simple steps to reproduce:

      flume-0.9.3rc1> ruby -e 'puts "x"*32767' > /tmp/log
      flume-0.9.3rc1> bin/flume dump 'tail("/tmp/log")'

      It appears that the source sends an incomplete message to the sink,
      and then continuously retries to send it.

      I've attached a patch that creates a buffer of size flume.event.max.size.bytes,
      and if it detects a line longer than this, it gets thrown away. It's been working
      for our systems for a week, but I haven't had time to write a unit test and test
      the edge cases.

        Activity

        Disabled imported user created issue -
        Hide
        Alex Baranau added a comment -

        Could you please create a review request at http://review.cloudera.org for the patch?

        Thank you in advance.

        Show
        Alex Baranau added a comment - Could you please create a review request at http://review.cloudera.org for the patch? Thank you in advance.
        Jonathan Hsieh made changes -
        Field Original Value New Value
        Fix Version/s v0.9.4 [ 10050 ]
        Jonathan Hsieh made changes -
        Status Open [ 1 ] Patch Available [ 10000 ]
        Hide
        Jonathan Hsieh added a comment -

        Brent,

        Take a look at this so we can proceed: https://github.com/cloudera/flume/wiki/HowToContribute

        At the least I need you to grant us rights to use the patch (oh legal people.. ).

        Thanks,
        Jon.

        Show
        Jonathan Hsieh added a comment - Brent, Take a look at this so we can proceed: https://github.com/cloudera/flume/wiki/HowToContribute At the least I need you to grant us rights to use the patch (oh legal people.. ). Thanks, Jon.
        Jonathan Hsieh made changes -
        Assignee Brent Halsey [ flume_brenthalsey ]
        Anonymous made changes -
        Status Patch Available [ 10000 ] Open [ 1 ]
        Anonymous made changes -
        Status Open [ 1 ] Patch Available [ 10000 ]
        Hide
        Disabled imported user added a comment -

        *I grant license to Cloudera for inclusion of my patches submitted to any
        Cloudera projects in Cloudera works (as per the Apache License §5).*

        Show
        Disabled imported user added a comment - *I grant license to Cloudera for inclusion of my patches submitted to any Cloudera projects in Cloudera works (as per the Apache License §5).*
        Hide
        Jonathan Hsieh added a comment -

        Hey Brent,

        I'm trying to catch up on the patch backlog and having problems trying to apply this patch. Do you have a github link or could you regenerate it?

        One comment from the patch is that to commit it we need some unit tests to be added. A simple test where a line is to long and it behaving in a particular way would be sufficient.

        Jon.

        Show
        Jonathan Hsieh added a comment - Hey Brent, I'm trying to catch up on the patch backlog and having problems trying to apply this patch. Do you have a github link or could you regenerate it? One comment from the patch is that to commit it we need some unit tests to be added. A simple test where a line is to long and it behaving in a particular way would be sufficient. Jon.
        Jonathan Hsieh made changes -
        Fix Version/s v0.9.5 [ 10090 ]
        Fix Version/s v0.9.4 [ 10050 ]
        Mark Thomas made changes -
        Project Import Tue Aug 02 16:57:12 UTC 2011 [ 1312304232406 ]
        Hide
        Jonathan Hsieh added a comment -

        I ported patch and created a test that is similar to the ruby script mentioned above. The entire event seems to have gotten dropped in this case. I'm editing to make it take the first maxeventsize bytes and then truncate the rest until a new event delimiter is reached.

        Show
        Jonathan Hsieh added a comment - I ported patch and created a test that is similar to the ruby script mentioned above. The entire event seems to have gotten dropped in this case. I'm editing to make it take the first maxeventsize bytes and then truncate the rest until a new event delimiter is reached.
        Jonathan Hsieh made changes -
        Assignee Disabled imported user [ flume_brenthalsey ] Jonathan Hsieh [ jmhsieh ]
        Hide
        Brent Halsey added a comment -

        patch for 0.9.4-branch in http://github.com/cloudera/flume.git

        -Brent Halsey and James Estes

        Show
        Brent Halsey added a comment - patch for 0.9.4-branch in http://github.com/cloudera/flume.git -Brent Halsey and James Estes
        Brent Halsey made changes -
        Attachment 0001-Fix-for-FLUME-533-with-tests.patch [ 12502517 ]
        Hide
        Brent Halsey added a comment -

        patch for trunk of https://github.com/apache/flume.git

        Brent Halsey and James Estes

        Show
        Brent Halsey added a comment - patch for trunk of https://github.com/apache/flume.git Brent Halsey and James Estes
        Brent Halsey made changes -
        Jonathan Hsieh made changes -
        Assignee Jonathan Hsieh [ jmhsieh ]
        Hide
        Ashish Paliwal added a comment -

        Won't fix. 0.X branch not maintained anymore

        Show
        Ashish Paliwal added a comment - Won't fix. 0.X branch not maintained anymore
        Ashish Paliwal made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Blocked Blocked Open Open
        22d 2h 14m 1 28/Apr/11 16:07
        Open Open Blocked Blocked
        42d 20h 17m 2 28/Apr/11 16:07
        Patch Available Patch Available Resolved Resolved
        1286d 18h 30m 1 Ashish Paliwal 05/Nov/14 09:37

          People

          • Assignee:
            Unassigned
            Reporter:
            Disabled imported user
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development