[SPARK-36679] Remove lz4 hadoop wrapper classes after Hadoop 3.3.2

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: L. C. Hsieh
• Created: 06/Sep/21 21:12
• Updated: 09/Mar/22 07:33
• Resolved: 09/Mar/22 07:33

Description

Lz4-java as provided dependency is not correctly excluded from relocation in Hadoop shaded client libraries in Hadoop 3.3.1. (HADOOP-17891)

 

In order to deal the issue without reverting back to non-shade client libraries, we add a few Lz4 Hadoop wrapper classes `LZ4Factory`, `LZ4Compressor`, and `LZ4SafeDecompressor`, under the package `org.apache.hadoop.shaded.net.jpountz.lz4`.

We should remove these wrapper classes after Hadoop 3.3.2 release which should include the fix.

 

Issue Links

• depends upon: SPARK-37600 Upgrade to Hadoop 3.3.2
• is duplicated by: SPARK-37600 Upgrade to Hadoop 3.3.2
• is related to: SPARK-36669 Fail to load Lz4 codec