[HADOOP-17219] ZStandardCodec compression mail fail(generic error) when encounter specific file

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: 2.6.5

Environment

zstd 1.3.3

hadoop 2.6.5 

 

— a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/zstd/TestZStandardCompressorDecompressor.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/zstd/TestZStandardCompressorDecompressor.java
@@ -62,10 +62,8 @@
@BeforeClass
public static void beforeClass() throws Exception {
CONFIGURATION.setInt(IO_FILE_BUFFER_SIZE_KEY, 1024 * 64);

• uncompressedFile = new File(TestZStandardCompressorDecompressor.class
• .getResource("/zstd/test_file.txt").toURI());
• compressedFile = new File(TestZStandardCompressorDecompressor.class
• .getResource("/zstd/test_file.txt.zst").toURI());
  + uncompressedFile = new File("/tmp/badcase.data");
  + compressedFile = new File("/tmp/badcase.data.zst");

• Assignee: Unassigned
• Reporter: Igloo
• Created: 30/Jun/20 03:41
• Updated: 04/Oct/21 02:45
• Resolved: 04/Oct/21 02:45

Description

Problem: 

In our production environment,  we put file in hdfs with zstd compressor, recently, we find that a specific file may leads to zstandard compressor failures. 

And we can reproduce the issue with specific file(attached file: badcase.data)

 

Analysis: 

ZStandarCompressor use buffersize( From zstd recommended compress out buffer size)  for both inBufferSize and outBufferSize 

but zstd indeed provides two separately recommending inputBufferSize and outputBufferSize  

 

Workaround

One workaround,  using recommended in/out buffer size provided by zstd lib can avoid the problem, but we don't know why. 

zstd recommended input buffer size:  1301072 (128 * 1024)

zstd recommended ouput buffer size: 131591 

 

 

 

 

 

 

 

Issue Links

• duplicates: HADOOP-17096 Fix ZStandardCompressor input buffer offset
• relates to: HADOOP-13578 Add Codec for ZStandard Compression