Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-929

Exception "too many open files" in ThriftEventSource (with infinite loop as result)

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Won't Fix
    • Affects Version/s: 0.9.4
    • Fix Version/s: 0.9.5
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Environment:

      Flume 0.9.4-cdh3u2 on Linux Squeeze, System with one master and two nodes

      Description

      We use ThriftEventSource for logging huge amount of data. However, an exception is thrown randomly and not reproducable which results in an infinite loop in one of the worker threads. We had to stop all flume nodes because the log file was dramatically increasing due to logged exceptions.

      This is just a small part of the log file:

      2012-01-11 14:57:30,487 ERROR com.cloudera.flume.core.connector.DirectDriver: Exiting driver logicalNode newsletterImpressionLog01-21 in error state ThriftEventSource | CassandraSink because sleep interrupted
      2012-01-11 17:18:14,808 WARN org.apache.thrift.server.TSaneThreadPoolServer: Transport error occurred during acceptance of message.
      org.apache.thrift.transport.TTransportException: java.net.SocketException: Too many open files
      at org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:139)
      at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
      at org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
      Caused by: java.net.SocketException: Too many open files
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
      at java.net.ServerSocket.implAccept(ServerSocket.java:462)
      at java.net.ServerSocket.accept(ServerSocket.java:430)
      at org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:134)
      ... 2 more
      2012-01-11 17:18:14,809 WARN org.apache.thrift.server.TSaneThreadPoolServer: Transport error occurred during acceptance of message.
      org.apache.thrift.transport.TTransportException: java.net.SocketException: Too many open files
      at org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:139)
      at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
      at org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
      Caused by: java.net.SocketException: Too many open files
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
      at java.net.ServerSocket.implAccept(ServerSocket.java:462)
      at java.net.ServerSocket.accept(ServerSocket.java:430)
      at org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:134)
      ... 2 more
      2012-01-11 17:18:14,809 WARN org.apache.thrift.server.TSaneThreadPoolServer: Transport error occurred during acceptance of message.
      org.apache.thrift.transport.TTransportException: java.net.SocketException: Too many open files
      at org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:139)
      at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
      at org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
      Caused by: java.net.SocketException: Too many open files
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
      at java.net.ServerSocket.implAccept(ServerSocket.java:462)
      at java.net.ServerSocket.accept(ServerSocket.java:430)
      at org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:134)
      ... 2 more
      2012-01-11 17:18:14,809 WARN org.apache.thrift.server.TSaneThreadPoolServer: Transport error occurred during acceptance of message.
      org.apache.thrift.transport.TTransportException: java.net.SocketException: Too many open files
      at org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:139)
      at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
      at org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
      Caused by: java.net.SocketException: Too many open files
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
      at java.net.ServerSocket.implAccept(ServerSocket.java:462)
      at java.net.ServerSocket.accept(ServerSocket.java:430)
      at org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:134)
      ... 2 more
      2012-01-11 17:18:14,810 WARN org.apache.thrift.server.TSaneThreadPoolServer: Transport error occurred during acceptance of message.
      org.apache.thrift.transport.TTransportException: java.net.SocketException: Too many open files
      at org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:139)
      at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
      at org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
      Caused by: java.net.SocketException: Too many open files
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
      at java.net.ServerSocket.implAccept(ServerSocket.java:462)
      at java.net.ServerSocket.accept(ServerSocket.java:430)
      at org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:134)
      ... 2 more
      2012-01-11 17:18:14,810 WARN org.apache.thrift.server.TSaneThreadPoolServer: Transport error occurred during acceptance of message.
      org.apache.thrift.transport.TTransportException: java.net.SocketException: Too many open files
      at org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:139)
      at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
      at org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
      Caused by: java.net.SocketException: Too many open files
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
      at java.net.ServerSocket.implAccept(ServerSocket.java:462)
      at java.net.ServerSocket.accept(ServerSocket.java:430)
      at org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:134)
      ... 2 more
      2012-01-11 17:18:14,810 WARN org.apache.thrift.server.TSaneThreadPoolServer: Transport error occurred during acceptance of message.
      org.apache.thrift.transport.TTransportException: java.net.SocketException: Too many open files
      at org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:139)
      at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
      at org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
      Caused by: java.net.SocketException: Too many open files
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
      at java.net.ServerSocket.implAccept(ServerSocket.java:462)
      at java.net.ServerSocket.accept(ServerSocket.java:430)
      at org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:134)
      ... 2 more

      So there are actually 2 issues to get fixed:

      1. Why are there "too many open files"?
      2. The worker thread must shutdown correctly and not ending in an infinite loop.

      This is a blocker issue for us because with this kind of error we are not able to use flume furthermore.

      Best Regards

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              bertpassek bert Passek
            • Votes:
              4 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: