Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25906

spark-shell cannot handle `-i` option correctly

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.4.1
    • Spark Shell, SQL
    • None

    Description

      This is a regression on Spark 2.4.0.

      Spark 2.3.2

      $ cat test.scala
      spark.version
      case class Record(key: Int, value: String)
      spark.sparkContext.parallelize((1 to 2).map(i => Record(i, s"val_$i"))).toDF.show
      
      $ bin/spark-shell -i test.scala
      18/10/31 23:22:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      Setting default log level to "WARN".
      To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
      Spark context Web UI available at http://localhost:4040
      Spark context available as 'sc' (master = local[*], app id = local-1541053368478).
      Spark session available as 'spark'.
      Loading test.scala...
      res0: String = 2.3.2
      defined class Record
      18/10/31 23:22:56 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
      +---+-----+
      |key|value|
      +---+-----+
      |  1|val_1|
      |  2|val_2|
      +---+-----+
      

      Spark 2.4.0 RC5

      $ bin/spark-shell -i test.scala
      2018-10-31 23:23:14 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      Setting default log level to "WARN".
      To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
      Spark context Web UI available at http://localhost:4040
      Spark context available as 'sc' (master = local[*], app id = local-1541053400312).
      Spark session available as 'spark'.
      test.scala:17: error: value toDF is not a member of org.apache.spark.rdd.RDD[Record]
      Error occurred in an application involving default arguments.
             spark.sparkContext.parallelize((1 to 2).map(i => Record(i, s"val_$i"))).toDF.show
      

      WORKAROUND
      Add the following line at the first of the script.

      import spark.implicits._
      

      Attachments

        Issue Links

          Activity

            People

              hyukjin.kwon Hyukjin Kwon
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: