[SPARK-34651] Improve ZSTD support

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Spark Core, SQL
• Labels: releasenotes
• Target Version/s: 3.2.0
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 06/Mar/21 22:54
• Updated: 01/Sep/21 13:54
• Resolved: 01/Sep/21 13:53

Issue Links

• relates to: AVRO-3060 Support ZSTD level and BufferPool options
• relates to: AVRO-3072 Use ZSTD NoFinalizer classes and bump to 1.4.9-1
• relates to: KAFKA-12442 Upgrade ZSTD JNI from 1.4.8-4 to 1.4.9-1
• relates to: KAFKA-12921 Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1
• relates to: PARQUET-1973 Support ZSTD JNI BufferPool
• relates to: PARQUET-1994 Upgrade ZSTD JNI to 1.4.9-1
• relates to: PARQUET-2057 Upgrade ZSTD-JNI to 1.5.0-1
• relates to: ORC-813 Upgrade ZSTD to 1.5.0

Sub-Tasks

• Upgrade zstd-jni to 1.4.8-3: Resolved
• Upgrade ORC to 1.6.6: Resolved
• Update ORC data source documentation: Resolved
• Add ZStandardBenchmark: Resolved
• Support ZSTD JNI BufferPool: Resolved
• Support ZSTD compression in ORC data source: Resolved
• Upgrade ZSTD-JNI to 1.4.8-5 to API compatibility: Resolved
• Add zstandard codec to spark.sql.avro.compression.codec: Resolved
• Enable Zstandard buffer pool by default: Resolved
• Upgrade to ZSTD JNI 1.4.8-6: Resolved
• Use zstd for spark.eventLog.compression.codec by default: Resolved
• Exclude Avro's transitive zstd-jni dependency: Resolved
• Exclude zstd-jni transitive dependency from Kafka Client: Resolved
• Upgrade ZSTD-JNI to 1.4.8-7 and use NoFinalizer classes: Resolved
• Upgrade ZSTD-JNI to 1.4.9-1: Resolved
• Support ZSTD compression in Parquet data sources: Resolved
• Add Parquet ZSTD compression test coverage: Resolved
• Upgrade Kafka to 2.8.0: Resolved
• Upgrade ZSTD-JNI to 1.5.0-2: Resolved
• Upgrade aircompressor to 1.21: Resolved
• Use zstd codec name in ORC file names: Resolved
• Change default spark.io.compression.codec to zstd: In Progress