[SPARK-11832] Spark shell does not work from sbt with scala 2.11

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Spark Shell
• Assignee: Jakob Odersky
• Reporter: Jakob Odersky
• Created: 18/Nov/15 23:26
• Updated: 10/Dec/15 16:36
• Resolved: 10/Dec/15 16:36

Description

Using Scala 2.11, running the spark shell task from within sbt fails, however running it from a distribution works.

Steps to reproduce

1. change scala version dev/change-scala-version.sh 2.11
2. start sbt build/sbt -Dscala-2.11
3. run shell task sparkShell

Stacktrace

Failed to initialize compiler: object scala in compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programmatically, settings.usejavacp.value = true.
Exception in thread "main" java.lang.NullPointerException
	at scala.reflect.internal.SymbolTable.exitingPhase(SymbolTable.scala:256)
	at scala.tools.nsc.interpreter.IMain$Request.x$20$lzycompute(IMain.scala:894)
	at scala.tools.nsc.interpreter.IMain$Request.x$20(IMain.scala:893)
	at scala.tools.nsc.interpreter.IMain$Request.importsPreamble$lzycompute(IMain.scala:893)
	at scala.tools.nsc.interpreter.IMain$Request.importsPreamble(IMain.scala:893)
	at scala.tools.nsc.interpreter.IMain$Request$Wrapper.preamble(IMain.scala:915)
	at scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$apply$23.apply(IMain.scala:1325)
	at scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$apply$23.apply(IMain.scala:1324)
	at scala.tools.nsc.util.package$.stringFromWriter(package.scala:64)
	at scala.tools.nsc.interpreter.IMain$CodeAssembler$class.apply(IMain.scala:1324)
	at scala.tools.nsc.interpreter.IMain$Request$Wrapper.apply(IMain.scala:906)
	at scala.tools.nsc.interpreter.IMain$Request.compile$lzycompute(IMain.scala:995)
	at scala.tools.nsc.interpreter.IMain$Request.compile(IMain.scala:990)
	at scala.tools.nsc.interpreter.IMain.compile(IMain.scala:577)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:563)
	at scala.tools.nsc.interpreter.ILoop.reallyInterpret$1(ILoop.scala:802)
	at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:836)
	at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:694)
	at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:404)
	at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcZ$sp(SparkILoop.scala:39)
	at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:38)
	at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:38)
	at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:213)
	at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:38)
	at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:94)
	at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:922)
	at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:911)
	at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:911)
	at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
	at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:911)
	at org.apache.spark.repl.Main$.main(Main.scala:49)
	at org.apache.spark.repl.Main.main(Main.scala)

Workaround

In repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala, append to the repl settings s.usejavacp.value = true

I haven't looked into the details of scala.tools.nsc.Settings, maybe someone has an idea of what's going on.
Also, to be clear, this bug only affects scala 2.11 from within sbt; calling spark-shell from a distribution or from anywhere using scala 2.10 works.

Issue Links

• is duplicated by: SPARK-12202 Pass additional Scala REPL options to the underlying REPL (2.11)
• links to: [Github] Pull Request #9824 (jodersky)