Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2416

Use CodecPool in compressed stream to prevent leak of direct buffers

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
      None

      Description

      Even though they may no longer be references, Java only cleans up direct buffers on full gc. If there is enough heap available, a full GC is never hit and these buffers are leaked. Hadoop keeps creating new compressors instead of using the pools causing a leak - which is a bug in itself which is being addressed by HADOOP-10591

      1. FLUME-2416.patch
        2 kB
        Hari Shreedharan

        Issue Links

          Activity

          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          +1

          Show
          jarcec Jarek Jarcec Cecho added a comment - +1
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 9940dcbfefbe1248f65aa83f2f84e352ce022041 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9940dcb ]

          FLUME-2416: Use CodecPool in compressed stream to prevent leak of direct buffers

          (Hari Shreedharan via Jarek Jarcec Cecho)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 9940dcbfefbe1248f65aa83f2f84e352ce022041 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9940dcb ] FLUME-2416 : Use CodecPool in compressed stream to prevent leak of direct buffers (Hari Shreedharan via Jarek Jarcec Cecho)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d56feccfb922165ab35a856f3d2cc65649093571 in flume's branch refs/heads/flume-1.6 from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d56fecc ]

          FLUME-2416: Use CodecPool in compressed stream to prevent leak of direct buffers

          (Hari Shreedharan via Jarek Jarcec Cecho)

          Show
          jira-bot ASF subversion and git services added a comment - Commit d56feccfb922165ab35a856f3d2cc65649093571 in flume's branch refs/heads/flume-1.6 from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d56fecc ] FLUME-2416 : Use CodecPool in compressed stream to prevent leak of direct buffers (Hari Shreedharan via Jarek Jarcec Cecho)
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Thank you for your contribution Hari Shreedharan!

          Show
          jarcec Jarek Jarcec Cecho added a comment - Thank you for your contribution Hari Shreedharan !
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in flume-trunk #643 (See https://builds.apache.org/job/flume-trunk/643/)
          FLUME-2416: Use CodecPool in compressed stream to prevent leak of direct buffers (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9940dcbfefbe1248f65aa83f2f84e352ce022041)

          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in flume-trunk #643 (See https://builds.apache.org/job/flume-trunk/643/ ) FLUME-2416 : Use CodecPool in compressed stream to prevent leak of direct buffers (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9940dcbfefbe1248f65aa83f2f84e352ce022041 ) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Flume-trunk-hbase-98 #3 (See https://builds.apache.org/job/Flume-trunk-hbase-98/3/)
          FLUME-2416: Use CodecPool in compressed stream to prevent leak of direct buffers (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9940dcbfefbe1248f65aa83f2f84e352ce022041)

          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-98 #3 (See https://builds.apache.org/job/Flume-trunk-hbase-98/3/ ) FLUME-2416 : Use CodecPool in compressed stream to prevent leak of direct buffers (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9940dcbfefbe1248f65aa83f2f84e352ce022041 ) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java

            People

            • Assignee:
              hshreedharan Hari Shreedharan
              Reporter:
              hshreedharan Hari Shreedharan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development