Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2984

Default maxReadBufferBytes might cause OOM and cause thrift source exit

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.6.0, 1.7.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Our usecase has multi agent flow and we are achieving this with thrift source and thrift sink combination. When there is a surge in incoming traffic downstream agent having thrift source is running out of memory. On further investigation of thrift transport/server code and scribe source found that we may need to fix the thrift transport's maxReadBufferBytes. Similar issue has been fixed in ScribeSource as part of FLUME-2404 by making this buffer configurable and capping it to a 16MB if not configured.

      thrift code (org.apache.thrift.server.AbstractNonblockingServer)

        public abstract static class AbstractNonblockingServerArgs<T extends AbstractNonblockingServer.AbstractNonblockingServerArgs<T>> extends AbstractServerArgs<T> {
          public long maxReadBufferBytes = 9223372036854775807L;
      
          public AbstractNonblockingServerArgs(TNonblockingServerTransport transport) {
            super(transport);
            this.transportFactory(new Factory());
          }
        }
      

      stacktrace

      31 Aug 2016 12:52:24 ERROR [Thread-12] - run() exiting due to uncaught error
      java.lang.OutOfMemoryError: unable to create new native thread
              at java.lang.Thread.start0(Native Method)
              at java.lang.Thread.start(Thread.java:714)
              at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
              at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
              at org.apache.thrift.server.TThreadedSelectorServer.requestInvoke(TThreadedSelectorServer.java:310)
              at org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:209)
              at org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.select(TThreadedSelectorServer.java:576)
              at org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.run(TThreadedSelectorServer.java:536)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lakshman Laxman
                Reporter:
                lakshman Laxman
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: