[SPARK-25906] spark-shell cannot handle `-i` option correctly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.1
• Component/s: Spark Shell, SQL
• Target Version/s: 2.4.1, 3.0.0
• Assignee: Hyukjin Kwon
• Reporter: Dongjoon Hyun
• Created: 01/Nov/18 06:25
• Updated: 12/Dec/22 18:11
• Resolved: 06/Nov/18 02:41

Description

This is a regression on Spark 2.4.0.

Spark 2.3.2

$ cat test.scala
spark.version
case class Record(key: Int, value: String)
spark.sparkContext.parallelize((1 to 2).map(i => Record(i, s"val_$i"))).toDF.show

$ bin/spark-shell -i test.scala
18/10/31 23:22:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1541053368478).
Spark session available as 'spark'.
Loading test.scala...
res0: String = 2.3.2
defined class Record
18/10/31 23:22:56 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
+---+-----+
|key|value|
+---+-----+
|  1|val_1|
|  2|val_2|
+---+-----+

Spark 2.4.0 RC5

$ bin/spark-shell -i test.scala
2018-10-31 23:23:14 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1541053400312).
Spark session available as 'spark'.
test.scala:17: error: value toDF is not a member of org.apache.spark.rdd.RDD[Record]
Error occurred in an application involving default arguments.
       spark.sparkContext.parallelize((1 to 2).map(i => Record(i, s"val_$i"))).toDF.show

WORKAROUND
Add the following line at the first of the script.

import spark.implicits._

Issue Links

• is duplicated by: SPARK-25587 NPE in Dataset when reading from Parquet as Product
• links to: [Github] Pull Request #22919 (HyukjinKwon)