[SPARK-43273] Support lz4raw compression codec for Parquet

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: SQL
• Assignee: Yuming Wang
• Reporter: Andrew Grigorev
• Created: 25/Apr/23 06:59
• Updated: 08/Jun/23 13:24
• Resolved: 08/Jun/23 08:48

Description

hadoop-parquet version should be updated to 1.3.0 (together with other parquet-mr libs)

java.util.concurrent.ExecutionException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 1.0 failed 1 times, most recent failure: Lost task 2.0 in stage 1.0 (TID 3) (f2b63fdfa0a6 executor driver): java.lang.IllegalArgumentException: No enum constant org.apache.parquet.hadoop.metadata.CompressionCodecName.LZ4_RAW
    at java.base/java.lang.Enum.valueOf(Enum.java:273)
    at org.apache.parquet.hadoop.metadata.CompressionCodecName.valueOf(CompressionCodecName.java:26)
    at org.apache.parquet.format.converter.ParquetMetadataConverter.fromFormatCodec(ParquetMetadataConverter.java:636)
... 

Issue Links

• relates to: PARQUET-2196 Support LZ4_RAW codec
• links to: [Github] Pull Request #41507 (wangyum)