[HADOOP-12619] Native memory leaks in CompressorStream

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.0
• Assignee: Unassigned
• Reporter: wangchao
• Created: 07/Dec/15 15:13
• Updated: 21/Dec/16 19:14

Description

The constructor of org.apache.hadoop.io.compress.CompressorStream requires an org.apache.hadoop.io.compress.Compressor object to compress bytes but it does not invoke the compressor's finish method when close method are called. This may causes the native memory leaks if the compressor is only used by this CompressorStream object.

I found this when set up a flume agent with gzip compression, the native memory grows slowly and cannot fall back.

  @Override
  public CompressionOutputStream createOutputStream(OutputStream out) 
    throws IOException {
    return (ZlibFactory.isNativeZlibLoaded(conf)) ?
               new CompressorStream(out, createCompressor(),
                                    conf.getInt("io.file.buffer.size", 4*1024)) :
               new GzipOutputStream(out);
  }

  @Override
  public Compressor createCompressor() {
    return (ZlibFactory.isNativeZlibLoaded(conf))
      ? new GzipZlibCompressor(conf)
      : null;
  }

The method of CompressorStream is

  @Override
  public void close() throws IOException {
    if (!closed) {
      finish();
      out.close();
      closed = true;
    }
  }

  @Override
  public void finish() throws IOException {
    if (!compressor.finished()) {
      compressor.finish();
      while (!compressor.finished()) {
        compress();
      }
    }
  }

No one will end the compressor.