[FLUME-2405] Flume stops working on arbitrary Frame Size more than 16 MB on ScribeSource

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 1.4.0, 1.5.0
• Component/s: Sinks+Sources
• Labels: FrameSize Scribe Source
• Assignee: Marimuthu Ponnambalam
• Reporter: Marimuthu Ponnambalam
• Created: 18/Jun/14 08:21
• Updated: 26/Dec/16 11:20
• Resolved: 04/Jul/14 23:06

Description

Flume stops receiving data on scribe source if there is an arbitrary size of data send from scribe, beyond 16MB.

TFrameTransport throws exception, due to frame size is more than 16 MB, which is default size on TFrameTransport.

This can be resolved by adding one more property "thriftFrameSize", which will allow custom frame size be configured, which will be used to create TFrameTransport.

This happens while scribe lag is being cleared, scribe sends Frame size larger than 16MB.

http://stackoverflow.com/questions/24150439/flume-scribesource-how-to-set-thrift-frame-size

2014-06-10 19:40:40,405 WARN org.apache.thrift.server.THsHaServer: Exception while invoking!
org.apache.thrift.transport.TTransportException: Frame size (23757404) larger than max length (16384000)!
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
at org.apache.thrift.server.Invocation.run(Invocation.java:18)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

Issue Links

• duplicates: FLUME-2404 Default maxReadBufferBytes might cause OOM and cause scribe source exit
• relates to: FLUME-2984 Default maxReadBufferBytes might cause OOM and cause thrift source exit
• links to: GitHub Pull Request #100