[HADOOP-19339] OutofBounds Exception due to assumption about buffer size in BlockCompressorStream

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.1
• Fix Version/s: 3.5.0
• Component/s: common
• Labels: pull-request-available
• Target Version/s: 3.4.1
• Hadoop Flags: Reviewed
• Assignee: ConfX
• Reporter: ConfX
• Created: 18/Nov/24 01:53
• Updated: 08/Dec/24 10:44
• Resolved: 08/Dec/24 10:44

Description

What Happened: 

Got an OutofBounds exception when io.compression.codec.snappy.buffersize is set to 7. BlockCompressorStream assumes that the buffer size will always be greater than the compression overhead, and consequently MAX_INPUT_SIZE will always be greater than or equal to 0. 

Buggy Code: 

When io.compression.codec.snappy.buffersize is set to 7, compressionOverhead is 33 and MAX_INPUT_SIZE is -26. 

public BlockCompressorStream(OutputStream out, Compressor compressor, 
                             int bufferSize, int compressionOverhead) {
  super(out, compressor, bufferSize);
  MAX_INPUT_SIZE = bufferSize - compressionOverhead; // -> Assumes bufferSize is always greater than compressionOverhead and MAX_INPUT_SIZE is non-negative. 
} 

Stack Trace: 

java.lang.ArrayIndexOutOfBoundsException
        at org.apache.hadoop.io.compress.snappy.SnappyCompressor.setInput(SnappyCompressor.java:86)
        at org.apache.hadoop.io.compress.BlockCompressorStream.write(BlockCompressorStream.java:112) 

How to Reproduce: 

(1) Set io.compression.codec.snappy.buffersize to 7

(2) Run test: org.apache.hadoop.io.compress.TestCodec#testSnappyMapFile

 

Issue Links

• links to: GitHub Pull Request #7167