Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-9919

ZSTD Compressor/Decompressor support in Lucene

    XMLWordPrintableJSON

    Details

    • Lucene Fields:
      New

      Description

      Lucene currently supports LZ4 and Zlib compression/decompression for StoredFieldsFormat, DocValuesFormat, TermVectorsFormat and PostingsFormat codecs. We propose Zstandard (https://facebook.github.io/zstd/) compression/decompression for all codecs mentioned earlier for following reasons:

      • ZStandard is being used in some of the most popular open source projects like Apache Cassandra, Hadoop and Kafka.
      • Zstandard, at the default setting of 3, is expected to show substantial improvements in both compression and decompression speed, while compressing at the same ratio as zlib as per study mentioned by Yann Collet at Facebook.
      • Zstandard currently offers 22 different Compression levels, which enable flexible, granular trade-offs between compression speed and ratios for future data. For example, we can use level 1 if speed is most important and level 22 if size is most important.
      • Zstandard designed to scale with modern hardware.
      • Small data

                - It has APIs for dictionary compression as well. Small data compression can range anywhere from 2x to 5x better than compression without dictionaries.

      • Zstandard is being continuously improved by Facebook/Community.

       

      Kindly go through below link for more details:       

      https://engineering.fb.com/2016/08/31/core-data/smaller-and-faster-data-compression-with-zstandard/

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                PraveenNishchal Praveen Nishchal
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: