[SPARK-44719] NoClassDefFoundError when using Hive UDF

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0, 4.0.0
• Component/s: Build, SQL
• Labels: pull-request-available
• Assignee: Yuming Wang
• Reporter: Yuming Wang
• Created: 08/Aug/23 12:18
• Updated: 21/Feb/24 20:38
• Resolved: 12/Aug/23 04:47

Description

How to reproduce:

spark-sql (default)> add jar /Users/yumwang/Downloads/HiveUDFs-1.0-SNAPSHOT.jar;
Time taken: 0.413 seconds
spark-sql (default)> CREATE TEMPORARY FUNCTION long_to_ip as 'net.petrabarus.hiveudfs.LongToIP';
Time taken: 0.038 seconds
spark-sql (default)> SELECT long_to_ip(2130706433L) FROM range(10);
23/08/08 20:17:58 ERROR SparkSQLDriver: Failed in [SELECT long_to_ip(2130706433L) FROM range(10)]
java.lang.NoClassDefFoundError: org/codehaus/jackson/map/type/TypeFactory
	at org.apache.hadoop.hive.ql.udf.UDFJson.<clinit>(UDFJson.java:64)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
...

Issue Links

• is caused by: SPARK-43225 Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
• is related to: SPARK-47119 Add `hive-jackson-provided` profile
• links to: [Github] Pull Request #42446 (wangyum)
• links to: [Github] Pull Request #42447 (wangyum)
• links to: GitHub Pull Request #42446