[SPARK-43099] `Class.getCanonicalName` return null for anonymous class on JDK15+, impacting function registry

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: Spark Core
• Assignee: Alex Jing
• Reporter: Alex Jing
• Shepherd: Gengliang Wang
• Created: 11/Apr/23 22:19
• Updated: 17/Apr/23 03:22
• Resolved: 17/Apr/23 03:15

Description

On JDK15+, lambda and method references are implemented using hidden classes (https://openjdk.org/jeps/371) According to the JEP, 

Class::getCanonicalName returns null, indicating the hidden class has no canonical name. (Note that the Class object for an anonymous class in the Java language has the same behavior.)

This means https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L53 will always be null.

 

This can be fixed by replacing `getCanonicalName` with `getName`

 

Issue Links

• links to: [Github] Pull Request #40747 (alexjinghn)