Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2738

Async HBase sink FD leak on client shutdown

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.7.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Currently every time Async Hbase Sink calls HBaseSink.shutdown, there is FD leak due to HBaseSink using CustomChannelFactory where releaseExternalResources() is overridden to a No-op. Need to replace this with a standard NioClientSocketChannelFactory, that releases external resources properly

      1. FLUME-2738.patch
        2 kB
        Johny Rufus
      2. FLUME-2738-1.patch
        6 kB
        Johny Rufus
      3. FLUME-2738-2.patch
        6 kB
        Johny Rufus
      4. FLUME-2738-3.patch
        6 kB
        Johny Rufus

        Activity

        Hide
        hshreedharan Hari Shreedharan added a comment -

        I don't see how this fixes the issue. Tracing the releaseExternalResources calls only shows a bunch of buffers being cleared up. It is nice to clear up memory, but where are the sockets or FDs being closed/nulled out? I don't see that anywhere in the releaseExternalResources call chain.

        Show
        hshreedharan Hari Shreedharan added a comment - I don't see how this fixes the issue. Tracing the releaseExternalResources calls only shows a bunch of buffers being cleared up. It is nice to clear up memory, but where are the sockets or FDs being closed/nulled out? I don't see that anywhere in the releaseExternalResources call chain.
        Hide
        jrufus Johny Rufus added a comment -

        Attaching a patch that tests the FD leak. Consistently been able to reproduce the FD leak w/o the patch and with the patch the leaks do not happen. releaseExternalResources shuts down the executors which seems to impact the release of the sockets.
        http://netty.io/3.6/api/org/jboss/netty/channel/socket/nio/NioClientSocketChannelFactory.html mentions that
        "All the related resources such as Selector are also released when the boss and worker threads are released. "

        Show
        jrufus Johny Rufus added a comment - Attaching a patch that tests the FD leak. Consistently been able to reproduce the FD leak w/o the patch and with the patch the leaks do not happen. releaseExternalResources shuts down the executors which seems to impact the release of the sockets. http://netty.io/3.6/api/org/jboss/netty/channel/socket/nio/NioClientSocketChannelFactory.html mentions that "All the related resources such as Selector are also released when the boss and worker threads are released. "
        Hide
        jrufus Johny Rufus added a comment -

        Attached, latest patch with minor changes

        Show
        jrufus Johny Rufus added a comment - Attached, latest patch with minor changes
        Hide
        hshreedharan Hari Shreedharan added a comment -

        +1. Looks good to me.

        Show
        hshreedharan Hari Shreedharan added a comment - +1. Looks good to me.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 38f0b316ddcc834c6249c7179c4c1d23ab445a84 in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=38f0b31 ]

        FLUME-2738. Fix file descriptor leak in AsyncHBaseSink when HBase cluster goes down.

        (Johny Rufus via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 38f0b316ddcc834c6249c7179c4c1d23ab445a84 in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=38f0b31 ] FLUME-2738 . Fix file descriptor leak in AsyncHBaseSink when HBase cluster goes down. (Johny Rufus via Hari)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 90e13ab8becf763258883d13637e34fe34342561 in flume's branch refs/heads/flume-1.7 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=90e13ab ]

        FLUME-2738. Fix file descriptor leak in AsyncHBaseSink when HBase cluster goes down.

        (Johny Rufus via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 90e13ab8becf763258883d13637e34fe34342561 in flume's branch refs/heads/flume-1.7 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=90e13ab ] FLUME-2738 . Fix file descriptor leak in AsyncHBaseSink when HBase cluster goes down. (Johny Rufus via Hari)
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Committed! Thanks Johny Rufus!

        Show
        hshreedharan Hari Shreedharan added a comment - Committed! Thanks Johny Rufus !
        Hide
        hudson Hudson added a comment -

        UNSTABLE: Integrated in Flume-trunk-hbase-1 #110 (See https://builds.apache.org/job/Flume-trunk-hbase-1/110/)
        FLUME-2738. Fix file descriptor leak in AsyncHBaseSink when HBase cluster goes down. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=38f0b316ddcc834c6249c7179c4c1d23ab445a84)

        • flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java
        • flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java
        Show
        hudson Hudson added a comment - UNSTABLE: Integrated in Flume-trunk-hbase-1 #110 (See https://builds.apache.org/job/Flume-trunk-hbase-1/110/ ) FLUME-2738 . Fix file descriptor leak in AsyncHBaseSink when HBase cluster goes down. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=38f0b316ddcc834c6249c7179c4c1d23ab445a84 ) flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java

          People

          • Assignee:
            jrufus Johny Rufus
            Reporter:
            jrufus Johny Rufus
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development