[SPARK-19112] add codec for ZStandard

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: Spark Core
• Assignee: Sital Kedia
• Reporter: Thomas Graves
• Created: 06/Jan/17 22:37
• Updated: 12/Dec/22 18:10
• Resolved: 01/Nov/17 13:54

Description

ZStandard: https://github.com/facebook/zstd and http://facebook.github.io/zstd/ has been in use for a while now. v1.0 was recently released. Hadoop (https://issues.apache.org/jira/browse/HADOOP-13578) and others (https://issues.apache.org/jira/browse/KAFKA-4514) are adopting it.

Zstd seems to give great results => Gzip level Compression with Lz4 level CPU.

Issue Links

• is duplicated by: SPARK-19958 Support ZStandard Compression
• links to: [Github] Pull Request #17303 (dongjinleekr)
• links to: [Github] Pull Request #18805 (sitalkedia)
• links to: [Github] Pull Request #21431 (wangyum)
• mentioned in: Page Loading...