[PARQUET-1320] Fast clean unused direct memory when decompress

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: parquet-mr
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: zhoukang
• Created: 08/Jun/18 11:40
• Updated: 23/Jun/24 03:30
• Resolved: 25/Feb/19 12:26

Description

When use NonBlockedDecompressorStream which call:

SnappyDecompressor.setInput

public synchronized void setInput(byte[] buffer, int off, int len) {
 SnappyUtil

public synchronized void setInput(byte[] buffer, int off, int len) {
 SnappyUtil.validateBuffer(buffer, off, len);

 if (inputBuffer.capacity() - inputBuffer.position() < len) {
 ByteBuffer newBuffer = ByteBuffer.allocateDirect(inputBuffer.position() + len);
 inputBuffer.rewind();
 newBuffer.put(inputBuffer);
 inputBuffer = newBuffer; 
 } else {
 inputBuffer.limit(inputBuffer.position() + len);
 }
 inputBuffer.put(buffer, off, len);
}

.validateBuffer(buffer, off, len);

 if (inputBuffer.capacity() - inputBuffer.position() < len) {
 ByteBuffer newBuffer = ByteBuffer.allocateDirect(inputBuffer.position() + len);
 inputBuffer.rewind();
 newBuffer.put(inputBuffer);
 inputBuffer = newBuffer; 
 } else {
 inputBuffer.limit(inputBuffer.position() + len);
 }
 inputBuffer.put(buffer, off, len);
}

If we do not get any full gc for old gen.we may failed by off-heap memory leak
 

Issue Links

• duplicates: PARQUET-1485 Snappy Decompressor/Compressor may cause direct memory leak
• links to: GitHub Pull Request #492