[SPARK-42886] ClassNotFoundException: scala.math.Ordering$Reverse

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.2
• Component/s: Build

Environment

Development environment

MacBook Pro

Java JDK ibm-1.8-362

 

• Assignee: Unassigned
• Reporter: Steve Chong
• Created: 21/Mar/23 17:40
• Updated: 25/Sep/23 17:27

Description

Hi,

We are using the spark-mllib_2.12 dependency in a Java project.

We are attempting to upgrade from version 3.3.1 to 3.3.2. This results in unit tests breaking with exception: ClassNotFoundException: scala.math.Ordering$Reverse

A change was made to add the class to the KyroSerializer  https://issues.apache.org/jira/browse/SPARK-42071

scala.math.Ordering$Reverse was introduced int Scala 2.12.12. The maven dependency tree (mvn dependency:tree) shows that spark-mllib_2.12 brings in scala-library version 2.12.8. Therefore, it doesn't contain scala.math.Ordering$Reverse. 

If the scala-library transitive dependency is excluded from the POM and an explicit dependency declared on with version >=2.12.12, the tests will pass.

Should the scala-library version contained in 3.3.2 be upgraded to >=2.12.12?