Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2405

Flume stops working on arbitrary Frame Size more than 16 MB on ScribeSource

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 1.4.0, 1.5.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources

      Description

      Flume stops receiving data on scribe source if there is an arbitrary size of data send from scribe, beyond 16MB.

      TFrameTransport throws exception, due to frame size is more than 16 MB, which is default size on TFrameTransport.

      This can be resolved by adding one more property "thriftFrameSize", which will allow custom frame size be configured, which will be used to create TFrameTransport.

      This happens while scribe lag is being cleared, scribe sends Frame size larger than 16MB.

      http://stackoverflow.com/questions/24150439/flume-scribesource-how-to-set-thrift-frame-size

      2014-06-10 19:40:40,405 WARN org.apache.thrift.server.THsHaServer: Exception while invoking!
      org.apache.thrift.transport.TTransportException: Frame size (23757404) larger than max length (16384000)!
      at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
      at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
      at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
      at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
      at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
      at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
      at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
      at org.apache.thrift.server.Invocation.run(Invocation.java:18)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:724)

        Attachments

        1. FLUME-2405.patch
          2 kB
          Mike Percy
        2. ScribeSource.java
          5 kB
          Marimuthu Ponnambalam

          Issue Links

            Activity

              People

              • Assignee:
                marimuthup Marimuthu Ponnambalam
                Reporter:
                marimuthup Marimuthu Ponnambalam
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: