Avro
  1. Avro
  2. AVRO-1538

NettyTransceiver descriptor leaks

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.7.5, 1.7.6
    • Fix Version/s: None
    • Component/s: java
    • Labels:
      None
    • Environment:

      openSuse 13.1 64 bit, jdk1.7.0_51 64 bit

      Description

      NettyTransceiver created without channel factory results in file descriptors leak while connecting with IOException (java.io.IOException: Error connecting to, NettyTransceiver.java:280).

      It seems very similar to https://issues.apache.org/jira/browse/FLUME-2221. But I am not flume user.

      To reproduce - run the test below (IP:9999 was nobody listened.)
      In my enviroment (run from IDEA, openSuse 13.1 64bit, all system settings are by default) stable fails after 146 cycles (count = 146).

          @Test
          public void testAvroNettyTest() {
              for (int count = 0; count < 4500; count++) {
                  try {
                      NettyTransceiver apiClient = new NettyTransceiver(new InetSocketAddress("<IP>", 9999), 5000L);
                  } catch (Exception exc) {
                      System.out.println(String.format("=========================== %d ===========================", count));
                      exc.printStackTrace();
      
                      if ( exc.getMessage().contains("Too many open files")
                              || exc.getCause().getMessage().contains("Too many open files")  ) {
                          System.out.println("Found [Too many open files].");
                          break;
                      }
                  }
              }
          }
      

      And test that OK

          @Test
          public void testAvroCustomNettyTest() throws Exception {
              for (int count = 0; count < 4500; count++) {
                  NioClientSocketChannelFactory channelFactory = null;
                  ExecutorService bossExecutor = null, workerExecutor = null;
                  try {
                      bossExecutor = Executors.newCachedThreadPool(new NettyTransceiverThreadFactory(
                              "Avro " + NettyTransceiver.class.getSimpleName() + " Boss"));
                      workerExecutor = Executors.newCachedThreadPool(new NettyTransceiverThreadFactory(
                              "Avro " + NettyTransceiver.class.getSimpleName() + " I/O Worker"));
      
                      channelFactory = new NioClientSocketChannelFactory(
                              bossExecutor,
                              workerExecutor
                              );
      
                      NettyTransceiver apiClient = new NettyTransceiver(
                              new InetSocketAddress("<IP>", 9999),
                              channelFactory,
                              5000L);
                  } catch (Exception exc) {
                      System.out.println(String.format("=========================== %d ===========================", count));
                      exc.printStackTrace();
      
                      if ( channelFactory != null ) {
                          channelFactory.shutdown();
                      }
      
                      if ( exc.getMessage().contains("Too many open files")
                              || exc.getCause().getMessage().contains("Too many open files")  ) {
                          System.out.println("Found [Too many open files].");
                          break;
                      }
                  }
              }
          }
      

      So workaround exists it would be cool if NettyTransceiver in next release will be more "friendly" (release all internal resources if initialization failed).

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Platon Vai
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development