HBase
  1. HBase
  2. HBASE-5881

BuiltIn Gzip compressor & decompressor not getting pooled, leading to native memory leak

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 0.92.1
    • Fix Version/s: 0.92.3
    • Component/s: io
    • Labels:
      None

      Description

      This issue will occur only in hadoop 23.x & above/

      In hadoop 0.20.x

      public static void returnDecompressor(Decompressor decompressor) {
          if (decompressor == null) {
            return;
          }
          decompressor.reset();
          payback(decompressorPool, decompressor);
        }
      

      In hadoop 0.23.x

        public static void returnDecompressor(Decompressor decompressor) {
          if (decompressor == null) {
            return;
          }
          // if the decompressor can't be reused, don't pool it.
          if (decompressor.getClass().isAnnotationPresent(DoNotPool.class)) {
            return;
          }
          decompressor.reset();
          payback(decompressorPool, decompressor);
        }
      

      Here annotation has been added. By default this library will be loaded if there are no native library.

      @DoNotPool
      public class BuiltInGzipDecompressor
      

      Due to this each time new compressor/decompressor will be loaded, this leads to native memory leak.

      2012-04-25 22:11:48,093 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.gz]
      2012-04-25 22:11:48,093 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.gz]
      2012-04-25 22:11:48,093 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.gz]
      

        Activity

        Hide
        Gopinathan A added a comment -

        OK, I feel.

        Show
        Gopinathan A added a comment - OK, I feel.
        Hide
        stack added a comment -

        Is the existance of this JIRA sufficient documentation until above is fixed?

        Show
        stack added a comment - Is the existance of this JIRA sufficient documentation until above is fixed?
        Hide
        ramkrishna.s.vasudevan added a comment -

        We need to document this?

        Show
        ramkrishna.s.vasudevan added a comment - We need to document this?

          People

          • Assignee:
            ramkrishna.s.vasudevan
            Reporter:
            Gopinathan A
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development