[SPARK-34651] Improve ZSTD support

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Spark Core, SQL
• Labels: releasenotes
• Target Version/s: 3.2.0
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 06/Mar/21 22:54
• Updated: 01/Sep/21 13:54
• Resolved: 01/Sep/21 13:53

Issue Links

• relates to: AVRO-3060 Support ZSTD level and BufferPool options
• relates to: AVRO-3072 Use ZSTD NoFinalizer classes and bump to 1.4.9-1
• relates to: KAFKA-12442 Upgrade ZSTD JNI from 1.4.8-4 to 1.4.9-1
• relates to: KAFKA-12921 Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1
• relates to: PARQUET-1973 Support ZSTD JNI BufferPool
• relates to: PARQUET-1994 Upgrade ZSTD JNI to 1.4.9-1
• relates to: PARQUET-2057 Upgrade ZSTD-JNI to 1.5.0-1
• relates to: ORC-813 Upgrade ZSTD to 1.5.0

Sub-Tasks

• Change default spark.io.compression.codec to zstd: In Progress