Uploaded image for project: 'Avro'
  1. Avro
  2. AVRO-1538

NettyTransceiver descriptor leaks

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.7.5, 1.7.6
    • Fix Version/s: None
    • Component/s: java
    • Labels:
      None
    • Environment:

      openSuse 13.1 64 bit, jdk1.7.0_51 64 bit

      Description

      NettyTransceiver created without channel factory results in file descriptors leak while connecting with IOException (java.io.IOException: Error connecting to, NettyTransceiver.java:280).

      It seems very similar to https://issues.apache.org/jira/browse/FLUME-2221. But I am not flume user.

      To reproduce - run the test below (IP:9999 was nobody listened.)
      In my enviroment (run from IDEA, openSuse 13.1 64bit, all system settings are by default) stable fails after 146 cycles (count = 146).

          @Test
          public void testAvroNettyTest() {
              for (int count = 0; count < 4500; count++) {
                  try {
                      NettyTransceiver apiClient = new NettyTransceiver(new InetSocketAddress("<IP>", 9999), 5000L);
                  } catch (Exception exc) {
                      System.out.println(String.format("=========================== %d ===========================", count));
                      exc.printStackTrace();
      
                      if ( exc.getMessage().contains("Too many open files")
                              || exc.getCause().getMessage().contains("Too many open files")  ) {
                          System.out.println("Found [Too many open files].");
                          break;
                      }
                  }
              }
          }
      

      And test that OK

          @Test
          public void testAvroCustomNettyTest() throws Exception {
              for (int count = 0; count < 4500; count++) {
                  NioClientSocketChannelFactory channelFactory = null;
                  ExecutorService bossExecutor = null, workerExecutor = null;
                  try {
                      bossExecutor = Executors.newCachedThreadPool(new NettyTransceiverThreadFactory(
                              "Avro " + NettyTransceiver.class.getSimpleName() + " Boss"));
                      workerExecutor = Executors.newCachedThreadPool(new NettyTransceiverThreadFactory(
                              "Avro " + NettyTransceiver.class.getSimpleName() + " I/O Worker"));
      
                      channelFactory = new NioClientSocketChannelFactory(
                              bossExecutor,
                              workerExecutor
                              );
      
                      NettyTransceiver apiClient = new NettyTransceiver(
                              new InetSocketAddress("<IP>", 9999),
                              channelFactory,
                              5000L);
                  } catch (Exception exc) {
                      System.out.println(String.format("=========================== %d ===========================", count));
                      exc.printStackTrace();
      
                      if ( channelFactory != null ) {
                          channelFactory.shutdown();
                      }
      
                      if ( exc.getMessage().contains("Too many open files")
                              || exc.getCause().getMessage().contains("Too many open files")  ) {
                          System.out.println("Found [Too many open files].");
                          break;
                      }
                  }
              }
          }
      

      So workaround exists it would be cool if NettyTransceiver in next release will be more "friendly" (release all internal resources if initialization failed).

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              platon-vai Platon Vai
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: