[FLINK-12462] Classloading fails in 1.8.0 when Scalatest is loaded first

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Workaround
• Affects Version/s: 1.8.0
• Component/s: API / Scala
• Assignee: Unassigned
• Reporter: Gaël Renoux
• Created: 09/May/19 08:47
• Updated: 06/Jan/20 08:56
• Resolved: 13/May/19 10:14

Description

When using Flink 1.8.0 and Scala 2.11.12, if Scalatest is loaded first and I use some class from the Scala library, I get a: java.lang.ClassNotFoundException: scala.reflect.io.JavaToolsPlatformArchive

To reproduce:

• start with the Scala Maven archetype (org.apache.flink:flink-quickstart-scala:1.8.0)
• in the POM, add a Scalatest dependency with test scope, before the Flink dependencies (org.scalatest:scalatest_${scala.binary.version}:3.0.4:test)
• in StreamingJob, add: env.fromElements[List[String]]()

I got a similar issue using Scala 2.12.7: java.lang.ClassNotFoundException: scala.reflect.internal.SymbolTable$ReflectStats

It works when changing the Flink version to 1.7.2. It also works if Flink is loaded first, but this can be a bit harder to ensure sometimes (our own POM has a few profile definitions).

I also managed to make it work by forcing the following dependency: org.scala-lang:scala-reflect:${scala.version}. Looking at the dependency tree, it looks like Flink needs the most recent version of scala-reflect but somehow doesn't enforce it. When Scalatest is loaded first, the version it declares (wich is slightly older: 2.11.11 or 2.12.3) gets used instead.