Avro
  1. Avro
  2. AVRO-1538

NettyTransceiver descriptor leaks

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.7.5, 1.7.6
    • Fix Version/s: None
    • Component/s: java
    • Labels:
      None
    • Environment:

      openSuse 13.1 64 bit, jdk1.7.0_51 64 bit

      Description

      NettyTransceiver created without channel factory results in file descriptors leak while connecting with IOException (java.io.IOException: Error connecting to, NettyTransceiver.java:280).

      It seems very similar to https://issues.apache.org/jira/browse/FLUME-2221. But I am not flume user.

      To reproduce - run the test below (IP:9999 was nobody listened.)
      In my enviroment (run from IDEA, openSuse 13.1 64bit, all system settings are by default) stable fails after 146 cycles (count = 146).

          @Test
          public void testAvroNettyTest() {
              for (int count = 0; count < 4500; count++) {
                  try {
                      NettyTransceiver apiClient = new NettyTransceiver(new InetSocketAddress("<IP>", 9999), 5000L);
                  } catch (Exception exc) {
                      System.out.println(String.format("=========================== %d ===========================", count));
                      exc.printStackTrace();
      
                      if ( exc.getMessage().contains("Too many open files")
                              || exc.getCause().getMessage().contains("Too many open files")  ) {
                          System.out.println("Found [Too many open files].");
                          break;
                      }
                  }
              }
          }
      

      And test that OK

          @Test
          public void testAvroCustomNettyTest() throws Exception {
              for (int count = 0; count < 4500; count++) {
                  NioClientSocketChannelFactory channelFactory = null;
                  ExecutorService bossExecutor = null, workerExecutor = null;
                  try {
                      bossExecutor = Executors.newCachedThreadPool(new NettyTransceiverThreadFactory(
                              "Avro " + NettyTransceiver.class.getSimpleName() + " Boss"));
                      workerExecutor = Executors.newCachedThreadPool(new NettyTransceiverThreadFactory(
                              "Avro " + NettyTransceiver.class.getSimpleName() + " I/O Worker"));
      
                      channelFactory = new NioClientSocketChannelFactory(
                              bossExecutor,
                              workerExecutor
                              );
      
                      NettyTransceiver apiClient = new NettyTransceiver(
                              new InetSocketAddress("<IP>", 9999),
                              channelFactory,
                              5000L);
                  } catch (Exception exc) {
                      System.out.println(String.format("=========================== %d ===========================", count));
                      exc.printStackTrace();
      
                      if ( channelFactory != null ) {
                          channelFactory.shutdown();
                      }
      
                      if ( exc.getMessage().contains("Too many open files")
                              || exc.getCause().getMessage().contains("Too many open files")  ) {
                          System.out.println("Found [Too many open files].");
                          break;
                      }
                  }
              }
          }
      

      So workaround exists it would be cool if NettyTransceiver in next release will be more "friendly" (release all internal resources if initialization failed).

        Activity

        Platon Vai created issue -
        Platon Vai made changes -
        Field Original Value New Value
        Description NettyTransceiver created without channel factory results in file descriptors leak while connecting with IOException (java.io.IOException: Error connecting to, NettyTransceiver.java:280).

        It seems very similar to https://issues.apache.org/jira/browse/FLUME-2221. But I am not flume user.

        To reproduce - run the test below (IP:9999 was nobody listened.)
        In my enviroment (run from IDEA, openSuse 13.1 64bit, all system settings are by default) stable fails after 146 cycles (count = 146).

            @Test
            public void testAvroNettyTest() {
                for (int count = 0; count < 4500; count++) {
                    try {
                        NettyTransceiver apiClient = new NettyTransceiver(new InetSocketAddress("<IP>", 9999), 5000L);
                    } catch (Exception exc) {
                        System.out.println(String.format("=========================== %d ===========================", count));
                        exc.printStackTrace();

                        if ( exc.getMessage().contains("Too many open files")
                                || exc.getCause().getMessage().contains("Too many open files") ) {
                            System.out.println("Found [Too many open files].");
                            break;
                        }
                    }
                }
            }

        And test that OK
            @Test
            public void testAvroCustomNettyTest() throws Exception {
                for (int count = 0; count < 4500; count++) {
                    NioClientSocketChannelFactory channelFactory = null;
                    ExecutorService bossExecutor = null, workerExecutor = null;
                    try {
                        bossExecutor = Executors.newCachedThreadPool(new NettyTransceiverThreadFactory(
                                "Avro " + NettyTransceiver.class.getSimpleName() + " Boss"));

                        workerExecutor = Executors.newCachedThreadPool(new NettyTransceiverThreadFactory(
                                "Avro " + NettyTransceiver.class.getSimpleName() + " I/O Worker"));

                        channelFactory = new NioClientSocketChannelFactory(
                                bossExecutor,
                                workerExecutor
                                );

                        NettyTransceiver apiClient = new NettyTransceiver(
                                new InetSocketAddress("<IP>", 9999),
                                channelFactory,
                                5000L);
                    } catch (Exception exc) {
                        System.out.println(String.format("=========================== %d ===========================", count));
                        exc.printStackTrace();

                        if ( channelFactory != null ) {
                            channelFactory.shutdown();
                        }

                        if ( exc.getMessage().contains("Too many open files")
                                || exc.getCause().getMessage().contains("Too many open files") ) {
                            System.out.println("Found [Too many open files].");
                            break;
                        }
                    }
                }
            }

        So workaround exists it would be cool if NettyTransceiver in next release will be more "friendly" (release all internal resources if initialization failed).
        NettyTransceiver created without channel factory results in file descriptors leak while connecting with IOException (java.io.IOException: Error connecting to, NettyTransceiver.java:280).

        It seems very similar to https://issues.apache.org/jira/browse/FLUME-2221. But I am not flume user.

        To reproduce - run the test below (IP:9999 was nobody listened.)
        In my enviroment (run from IDEA, openSuse 13.1 64bit, all system settings are by default) stable fails after 146 cycles (count = 146).
        <code>
            @Test
            public void testAvroNettyTest() {
                for (int count = 0; count < 4500; count++) {
                    try {
                        NettyTransceiver apiClient = new NettyTransceiver(new InetSocketAddress("<IP>", 9999), 5000L);
                    } catch (Exception exc) {
                        System.out.println(String.format("=========================== %d ===========================", count));
                        exc.printStackTrace();

                        if ( exc.getMessage().contains("Too many open files")
                                || exc.getCause().getMessage().contains("Too many open files") ) {
                            System.out.println("Found [Too many open files].");
                            break;
                        }
                    }
                }
            }
        </code>

        And test that OK
        <code>
            @Test
            public void testAvroCustomNettyTest() throws Exception {
                for (int count = 0; count < 4500; count++) {
                    NioClientSocketChannelFactory channelFactory = null;
                    ExecutorService bossExecutor = null, workerExecutor = null;
                    try {
                        bossExecutor = Executors.newCachedThreadPool(new NettyTransceiverThreadFactory(
                                "Avro " + NettyTransceiver.class.getSimpleName() + " Boss"));
        </code>

                        workerExecutor = Executors.newCachedThreadPool(new NettyTransceiverThreadFactory(
                                "Avro " + NettyTransceiver.class.getSimpleName() + " I/O Worker"));

                        channelFactory = new NioClientSocketChannelFactory(
                                bossExecutor,
                                workerExecutor
                                );

                        NettyTransceiver apiClient = new NettyTransceiver(
                                new InetSocketAddress("<IP>", 9999),
                                channelFactory,
                                5000L);
                    } catch (Exception exc) {
                        System.out.println(String.format("=========================== %d ===========================", count));
                        exc.printStackTrace();

                        if ( channelFactory != null ) {
                            channelFactory.shutdown();
                        }

                        if ( exc.getMessage().contains("Too many open files")
                                || exc.getCause().getMessage().contains("Too many open files") ) {
                            System.out.println("Found [Too many open files].");
                            break;
                        }
                    }
                }
            }

        So workaround exists it would be cool if NettyTransceiver in next release will be more "friendly" (release all internal resources if initialization failed).
        Platon Vai made changes -
        Description NettyTransceiver created without channel factory results in file descriptors leak while connecting with IOException (java.io.IOException: Error connecting to, NettyTransceiver.java:280).

        It seems very similar to https://issues.apache.org/jira/browse/FLUME-2221. But I am not flume user.

        To reproduce - run the test below (IP:9999 was nobody listened.)
        In my enviroment (run from IDEA, openSuse 13.1 64bit, all system settings are by default) stable fails after 146 cycles (count = 146).
        <code>
            @Test
            public void testAvroNettyTest() {
                for (int count = 0; count < 4500; count++) {
                    try {
                        NettyTransceiver apiClient = new NettyTransceiver(new InetSocketAddress("<IP>", 9999), 5000L);
                    } catch (Exception exc) {
                        System.out.println(String.format("=========================== %d ===========================", count));
                        exc.printStackTrace();

                        if ( exc.getMessage().contains("Too many open files")
                                || exc.getCause().getMessage().contains("Too many open files") ) {
                            System.out.println("Found [Too many open files].");
                            break;
                        }
                    }
                }
            }
        </code>

        And test that OK
        <code>
            @Test
            public void testAvroCustomNettyTest() throws Exception {
                for (int count = 0; count < 4500; count++) {
                    NioClientSocketChannelFactory channelFactory = null;
                    ExecutorService bossExecutor = null, workerExecutor = null;
                    try {
                        bossExecutor = Executors.newCachedThreadPool(new NettyTransceiverThreadFactory(
                                "Avro " + NettyTransceiver.class.getSimpleName() + " Boss"));
        </code>

                        workerExecutor = Executors.newCachedThreadPool(new NettyTransceiverThreadFactory(
                                "Avro " + NettyTransceiver.class.getSimpleName() + " I/O Worker"));

                        channelFactory = new NioClientSocketChannelFactory(
                                bossExecutor,
                                workerExecutor
                                );

                        NettyTransceiver apiClient = new NettyTransceiver(
                                new InetSocketAddress("<IP>", 9999),
                                channelFactory,
                                5000L);
                    } catch (Exception exc) {
                        System.out.println(String.format("=========================== %d ===========================", count));
                        exc.printStackTrace();

                        if ( channelFactory != null ) {
                            channelFactory.shutdown();
                        }

                        if ( exc.getMessage().contains("Too many open files")
                                || exc.getCause().getMessage().contains("Too many open files") ) {
                            System.out.println("Found [Too many open files].");
                            break;
                        }
                    }
                }
            }

        So workaround exists it would be cool if NettyTransceiver in next release will be more "friendly" (release all internal resources if initialization failed).
        NettyTransceiver created without channel factory results in file descriptors leak while connecting with IOException (java.io.IOException: Error connecting to, NettyTransceiver.java:280).

        It seems very similar to https://issues.apache.org/jira/browse/FLUME-2221. But I am not flume user.

        To reproduce - run the test below (IP:9999 was nobody listened.)
        In my enviroment (run from IDEA, openSuse 13.1 64bit, all system settings are by default) stable fails after 146 cycles (count = 146).
        {noformat}
            @Test
            public void testAvroNettyTest() {
                for (int count = 0; count < 4500; count++) {
                    try {
                        NettyTransceiver apiClient = new NettyTransceiver(new InetSocketAddress("<IP>", 9999), 5000L);
                    } catch (Exception exc) {
                        System.out.println(String.format("=========================== %d ===========================", count));
                        exc.printStackTrace();

                        if ( exc.getMessage().contains("Too many open files")
                                || exc.getCause().getMessage().contains("Too many open files") ) {
                            System.out.println("Found [Too many open files].");
                            break;
                        }
                    }
                }
            }
        {noformat}

        And test that OK
        {noformat}
            @Test
            public void testAvroCustomNettyTest() throws Exception {
                for (int count = 0; count < 4500; count++) {
                    NioClientSocketChannelFactory channelFactory = null;
                    ExecutorService bossExecutor = null, workerExecutor = null;
                    try {
                        bossExecutor = Executors.newCachedThreadPool(new NettyTransceiverThreadFactory(
                                "Avro " + NettyTransceiver.class.getSimpleName() + " Boss"));
                        workerExecutor = Executors.newCachedThreadPool(new NettyTransceiverThreadFactory(
                                "Avro " + NettyTransceiver.class.getSimpleName() + " I/O Worker"));

                        channelFactory = new NioClientSocketChannelFactory(
                                bossExecutor,
                                workerExecutor
                                );

                        NettyTransceiver apiClient = new NettyTransceiver(
                                new InetSocketAddress("<IP>", 9999),
                                channelFactory,
                                5000L);
                    } catch (Exception exc) {
                        System.out.println(String.format("=========================== %d ===========================", count));
                        exc.printStackTrace();

                        if ( channelFactory != null ) {
                            channelFactory.shutdown();
                        }

                        if ( exc.getMessage().contains("Too many open files")
                                || exc.getCause().getMessage().contains("Too many open files") ) {
                            System.out.println("Found [Too many open files].");
                            break;
                        }
                    }
                }
            }
        {noformat}

        So workaround exists it would be cool if NettyTransceiver in next release will be more "friendly" (release all internal resources if initialization failed).

          People

          • Assignee:
            Unassigned
            Reporter:
            Platon Vai
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development