[SPARK-26037] ./dev/make-distribution.sh fails for Scala 2.12

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.4.0
• Component/s: Build
• Assignee: Unassigned
• Reporter: Damian Momot
• Created: 13/Nov/18 10:15
• Updated: 13/Nov/18 15:59
• Resolved: 13/Nov/18 10:41

Description

Trying to create spark distribution for scala 2.12 and Spark 2.4.0

First add cloudera repository:

<repository>
  <id>cloudera</id>
  <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>

Try to make distribution:

./dev/change-scala-version.sh 2.12
./dev/make-distribution.sh --name scala_2.12-cdh5.15.1 --tgz -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.15.1 -DskipTests -Phive -Phive-thriftserver -Pyarn -Dscala-2.12

I'm getting multiple failures in Spark Project Sketch - all look the same, here's first for reference

~/spark-2.4.0/common/sketch/src/test/scala/org/apache/spark/util/sketch/BitArraySuite.scala:32: exception during macro expansion: 
java.lang.ClassNotFoundException: scala.runtime.LazyRef
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.scalactic.MacroOwnerRepair$Utils.repairOwners(MacroOwnerRepair.scala:66)
at org.scalactic.MacroOwnerRepair.repairOwners(MacroOwnerRepair.scala:46)
at org.scalactic.BooleanMacro.genMacro(BooleanMacro.scala:837)
at org.scalatest.AssertionsMacro$.assert(AssertionsMacro.scala:34)
assert(new BitArray(64).bitSize() == 64)

Same commands for Scala 2.11 work fine