Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34651

Improve ZSTD support

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • Spark Core, SQL

    Attachments

      Issue Links

        relates to

        Improvement - An improvement or enhancement to an existing feature or task. AVRO-3060 Support ZSTD level and BufferPool options

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. AVRO-3072 Use ZSTD NoFinalizer classes and bump to 1.4.9-1

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. KAFKA-12442 Upgrade ZSTD JNI from 1.4.8-4 to 1.4.9-1

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. KAFKA-12921 Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. PARQUET-1973 Support ZSTD JNI BufferPool

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. PARQUET-1994 Upgrade ZSTD JNI to 1.4.9-1

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. PARQUET-2057 Upgrade ZSTD-JNI to 1.5.0-1

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. ORC-813 Upgrade ZSTD to 1.5.0

        • Major - Major loss of function.
        • Closed
        1.
        		 Upgrade zstd-jni to 1.4.8-3 Sub-task Resolved William Hyun
        2.
        		 Upgrade ORC to 1.6.6 Sub-task Resolved Dongjoon Hyun
        3.
        		 Update ORC data source documentation Sub-task Resolved Dongjoon Hyun
        4.
        		 Add ZStandardBenchmark Sub-task Resolved Dongjoon Hyun
        5.
        		 Support ZSTD JNI BufferPool Sub-task Resolved Dongjoon Hyun
        6.
        		 Support ZSTD compression in ORC data source Sub-task Resolved Dongjoon Hyun
        7.
        		 Upgrade ZSTD-JNI to 1.4.8-5 to API compatibility Sub-task Resolved Dongjoon Hyun
        8.
        		 Add zstandard codec to spark.sql.avro.compression.codec Sub-task Resolved Yuming Wang
        9.
        		 Enable Zstandard buffer pool by default Sub-task Resolved Dongjoon Hyun
        10.
        		 Upgrade to ZSTD JNI 1.4.8-6 Sub-task Resolved Dongjoon Hyun
        11.
        		 Use zstd for spark.eventLog.compression.codec by default Sub-task Resolved Dongjoon Hyun
        12.
        		 Exclude Avro's transitive zstd-jni dependency Sub-task Resolved Dongjoon Hyun
        13.
        		 Exclude zstd-jni transitive dependency from Kafka Client Sub-task Resolved Dongjoon Hyun
        14.
        		 Upgrade ZSTD-JNI to 1.4.8-7 and use NoFinalizer classes Sub-task Resolved Dongjoon Hyun
        15.
        		 Upgrade ZSTD-JNI to 1.4.9-1 Sub-task Resolved Dongjoon Hyun
        16.
        		 Support ZSTD compression in Parquet data sources Sub-task Resolved Unassigned
        17.
        		 Add Parquet ZSTD compression test coverage Sub-task Resolved Dongjoon Hyun
        18.
        		 Upgrade Kafka to 2.8.0 Sub-task Resolved Dongjoon Hyun
        19.
        		 Upgrade ZSTD-JNI to 1.5.0-2 Sub-task Resolved David Christle
        20.
        		 Upgrade aircompressor to 1.21 Sub-task Resolved Dongjoon Hyun
        21.
        		 Use zstd codec name in ORC file names Sub-task Resolved Dongjoon Hyun
        22.
        		 Change default spark.io.compression.codec to zstd Sub-task In Progress Unassigned

        Activity

          People

            dongjoon Dongjoon Hyun
            dongjoon Dongjoon Hyun
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: