Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17219

ZStandardCodec compression mail fail(generic error) when encounter specific file

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Duplicate
    • 2.6.5
    • None
    • None
    • None

    Description

      Problem: 

      In our production environment,  we put file in hdfs with zstd compressor, recently, we find that a specific file may leads to zstandard compressor failures. 

      And we can reproduce the issue with specific file(attached file: badcase.data)

       

      Analysis

      ZStandarCompressor use buffersize( From zstd recommended compress out buffer size)  for both inBufferSize and outBufferSize 

      but zstd indeed provides two separately recommending inputBufferSize and outputBufferSize  

       

      Workaround

      One workaround,  using recommended in/out buffer size provided by zstd lib can avoid the problem, but we don't know why. 

      zstd recommended input buffer size:  1301072 (128 * 1024)

      zstd recommended ouput buffer size: 131591 

       

       

       

       

       

       

       

      Attachments

        1. HDFS-15445.patch
          24.56 MB
          Igloo
        2. image-2020-06-30-11-51-18-026.png
          56 kB
          Igloo
        3. image-2020-06-30-11-42-44-585.png
          11 kB
          Igloo
        4. image-2020-06-30-11-39-17-861.png
          30 kB
          Igloo
        5. image-2020-06-30-11-35-46-859.png
          47 kB
          Igloo
        6. badcase.data
          19.05 MB
          Igloo

        Issue Links

          Activity

            People

              Unassigned Unassigned
              igloo1986 Igloo
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: