[FLUME-2984] Default maxReadBufferBytes might cause OOM and cause thrift source exit

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.6.0, 1.7.0
• Component/s: Sinks+Sources
• Assignee: Laxman
• Reporter: Laxman
• Created: 31/Aug/16 13:42
• Updated: 30/Jun/17 14:14

Description

Our usecase has multi agent flow and we are achieving this with thrift source and thrift sink combination. When there is a surge in incoming traffic downstream agent having thrift source is running out of memory. On further investigation of thrift transport/server code and scribe source found that we may need to fix the thrift transport's maxReadBufferBytes. Similar issue has been fixed in ScribeSource as part of FLUME-2404 by making this buffer configurable and capping it to a 16MB if not configured.

thrift code (org.apache.thrift.server.AbstractNonblockingServer)

  public abstract static class AbstractNonblockingServerArgs<T extends AbstractNonblockingServer.AbstractNonblockingServerArgs<T>> extends AbstractServerArgs<T> {
    public long maxReadBufferBytes = 9223372036854775807L;

    public AbstractNonblockingServerArgs(TNonblockingServerTransport transport) {
      super(transport);
      this.transportFactory(new Factory());
    }
  }

stacktrace

31 Aug 2016 12:52:24 ERROR [Thread-12] - run() exiting due to uncaught error
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:714)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
        at org.apache.thrift.server.TThreadedSelectorServer.requestInvoke(TThreadedSelectorServer.java:310)
        at org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:209)
        at org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.select(TThreadedSelectorServer.java:576)
        at org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.run(TThreadedSelectorServer.java:536)

Issue Links

• is related to: FLUME-2405 Flume stops working on arbitrary Frame Size more than 16 MB on ScribeSource
• is related to: FLUME-2404 Default maxReadBufferBytes might cause OOM and cause scribe source exit
• links to: GitHub Pull Request #100
• links to: GitHub Pull Request #102