[HBASE-28655] TestHFileCompressionZstd fails with IllegalArgumentException: Illegal bufferSize

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0, 3.0.0-beta-1, 2.5.8
• Fix Version/s: 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.11
• Component/s: HFile, Operability
• Labels: pull-request-available
• Assignee: Pankaj Kumar
• Reporter: Pankaj Kumar
• Created: 11/Jun/24 13:47
• Updated: 27/Jul/24 04:52
• Resolved: 25/Jul/24 18:29

Description

HADOOP-18810 added io.compression.codec.zstd.buffersize in core-default.xml with default value as 0.

So ZSTD buffer size will be returned as 0 based on core-default.xml,

  static int getBufferSize(Configuration conf) {
    return conf.getInt(ZSTD_BUFFER_SIZE_KEY,
      conf.getInt(CommonConfigurationKeys.IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_KEY,
        // IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_DEFAULT is 0! We can't allow that.
        ZSTD_BUFFER_SIZE_DEFAULT));
  }

HBASE-26259 added a value check, but got reverted in HBASE-26959.

 

This issue will also occur during region flush and abort the RegionServer.

 

TestHFileCompressionZstd and other zstd related test cases are are also failing,

java.lang.IllegalArgumentException: Illegal bufferSize
	at org.apache.hadoop.io.compress.CompressorStream.<init>(CompressorStream.java:42)
	at org.apache.hadoop.io.compress.BlockCompressorStream.<init>(BlockCompressorStream.java:56)
	at org.apache.hadoop.hbase.io.compress.aircompressor.ZstdCodec.createOutputStream(ZstdCodec.java:106)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.createPlainCompressionStream(Compression.java:454)
	at org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:99)
	at org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder.newDataBlockEncodingContext(NoOpDataBlockEncoder.java:85)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:846)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.finishInit(HFileWriterImpl.java:304)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.<init>(HFileWriterImpl.java:185)
	at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:312)
	at org.apache.hadoop.hbase.io.compress.HFileTestBase.doTest(HFileTestBase.java:73)
	at org.apache.hadoop.hbase.io.compress.aircompressor.TestHFileCompressionZstd.test(TestHFileCompressionZstd.java:54)

Issue Links

• links to: GitHub Pull Request #6114
• links to: GitHub Pull Request #6121