Flume
  1. Flume
  2. FLUME-1293

NIO channel selector open throws "Too many open files" exception

    Details

    • Type: Wish Wish
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: v1.1.0
    • Fix Version/s: None
    • Component/s: Docs
    • Labels:
    • Environment:

      Description

      2012-06-19 19:01:21,998 ERROR api.NettyAvroRpcClient: RPC connection error :
      java.io.IOException: Error connecting to /x.x.x.x:9090
      	at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249)
      	at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198)
      	at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147)
      	at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:115)
      	at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:104)
      	at org.apache.flume.api.NettyAvroRpcClient.<init>(NettyAvroRpcClient.java:89)
      	at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:131)
      	at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:148)
      	at org.apache.flume.sink.AvroSink.verifyConnection(AvroSink.java:176)
      	at org.apache.flume.sink.AvroSink.process(AvroSink.java:231)
      	at org.apache.flume.sink.FailoverSinkProcessor.process(FailoverSinkProcessor.java:162)
      	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
      	at java.lang.Thread.run(Thread.java:662)
      Caused by: org.jboss.netty.channel.ChannelException: Failed to create a selector.
      	at org.jboss.netty.channel.socket.nio.NioWorker.register(NioWorker.java:105)
      	at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:403)
      	at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370)
      	at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	... 1 more
      Caused by: java.io.IOException: Too many open files
      	at sun.nio.ch.IOUtil.initPipe(Native Method)
      	at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
      	at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
      	at java.nio.channels.Selector.open(Selector.java:209)
      	at org.jboss.netty.channel.socket.nio.NioWorker.register(NioWorker.java:103)
      

        Activity

        Hide
        Mubarak Seyed added a comment -

        The attached file (flume-1293-stacktrace.txt) contains full stacktrace. Thanks.

        Show
        Mubarak Seyed added a comment - The attached file (flume-1293-stacktrace.txt) contains full stacktrace. Thanks.
        Hide
        Mubarak Seyed added a comment -

        It turns out that i am running ulimit for flume user as 1024 and multiple client connections are talking to port 9090, let me bump up the ulimit for flume user to 32k and test it out.

        Show
        Mubarak Seyed added a comment - It turns out that i am running ulimit for flume user as 1024 and multiple client connections are talking to port 9090, let me bump up the ulimit for flume user to 32k and test it out.
        Hide
        Mubarak Seyed added a comment -

        Environment with high-volume may come across same issue, we need to add it in FAQ.

        Show
        Mubarak Seyed added a comment - Environment with high-volume may come across same issue, we need to add it in FAQ.
        Hide
        Mubarak Seyed added a comment -

        Proposed solution:

        • Create a file flume.conf under /etc/security/limits.d in linux
        • Add the following lines in flume.conf
          flume   soft   nofile   <size of ulimit. e.g 65536>
          flume   hard   nofile   <size of ulimit. e.g 65536>
          
        Show
        Mubarak Seyed added a comment - Proposed solution: Create a file flume.conf under /etc/security/limits.d in linux Add the following lines in flume.conf flume soft nofile <size of ulimit. e.g 65536> flume hard nofile <size of ulimit. e.g 65536>

          People

          • Assignee:
            Unassigned
            Reporter:
            Mubarak Seyed
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development