[SPARK-35612] Support LZ4 compression in ORC data source

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Han
• Reporter: Han
• Created: 02/Jun/21 13:24
• Updated: 03/Jun/21 21:08
• Resolved: 03/Jun/21 21:08

Description

Apache ORC supports LZ4 compression, but we cannot set LZ4 compression in the ORC data source

scala> spark.range(10).write.option("compression", "lz4").orc("/tmp/lz4")
java.lang.IllegalArgumentException: Codec [lz4] is not available. Available codecs are uncompressed, lzo, snappy, zlib, none, zstd.

Issue Links

• links to: [Github] Pull Request #32751 (fornaix)