Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25906

spark-shell cannot handle `-i` option correctly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.1
    • Component/s: Spark Shell, SQL
    • Labels:
      None

      Description

      This is a regression on Spark 2.4.0.

      Spark 2.3.2

      $ cat test.scala
      spark.version
      case class Record(key: Int, value: String)
      spark.sparkContext.parallelize((1 to 2).map(i => Record(i, s"val_$i"))).toDF.show
      
      $ bin/spark-shell -i test.scala
      18/10/31 23:22:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      Setting default log level to "WARN".
      To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
      Spark context Web UI available at http://localhost:4040
      Spark context available as 'sc' (master = local[*], app id = local-1541053368478).
      Spark session available as 'spark'.
      Loading test.scala...
      res0: String = 2.3.2
      defined class Record
      18/10/31 23:22:56 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
      +---+-----+
      |key|value|
      +---+-----+
      |  1|val_1|
      |  2|val_2|
      +---+-----+
      

      Spark 2.4.0 RC5

      $ bin/spark-shell -i test.scala
      2018-10-31 23:23:14 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      Setting default log level to "WARN".
      To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
      Spark context Web UI available at http://localhost:4040
      Spark context available as 'sc' (master = local[*], app id = local-1541053400312).
      Spark session available as 'spark'.
      test.scala:17: error: value toDF is not a member of org.apache.spark.rdd.RDD[Record]
      Error occurred in an application involving default arguments.
             spark.sparkContext.parallelize((1 to 2).map(i => Record(i, s"val_$i"))).toDF.show
      

      WORKAROUND
      Add the following line at the first of the script.

      import spark.implicits._
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hyukjin.kwon Hyukjin Kwon
                Reporter:
                dongjoon Dongjoon Hyun
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: