Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25138

Spark Shell should show the Scala prompt after initialization is complete

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: Spark Shell
    • Labels:
      None
    • Target Version/s:

      Description

      In previous Spark versions, the Spark Shell used to only show the Scala prompt after Spark has initialized. i.e. when the user is able to enter code, the Spark context, Spark session etc have all completed initialization, so sc, spark are all ready to use.

      In the current Spark master branch (to become Spark 2.4.0), the Scala prompt shows up immediately, while Spark itself is still in initialization in the background. It's very easy for the user to feel as if the shell is ready and start typing, only to find that Spark isn't ready yet, and Spark's initialization logs get in the way of typing. This new behavior is rather annoying from a usability's perspective.

      A typical startup of the Spark Shell in current master:

      $ bin/spark-shell
      18/08/16 23:18:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
      Setting default log level to "WARN".
      To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
      Welcome to
            ____              __
           / __/__  ___ _____/ /__
          _\ \/ _ \/ _ `/ __/  '_/
         /___/ .__/\_,_/_/ /_/\_\   version 2.4.0-SNAPSHOT
            /_/
               
      Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131)
      Type in expressions to have them evaluated.
      Type :help for more information.
      
      scala> spark.range(1)Spark context Web UI available at http://localhost:4040
      Spark context available as 'sc' (master = local[*], app id = local-1534486692744).
      Spark session available as 'spark'.
      .show
      +---+
      | id|
      +---+
      |  0|
      +---+
      
      
      scala> 
      

      Could you see that it was running spark.range(1).show ?

      In contrast, previous versions of Spark Shell would wait for Spark to fully initialization:

      $ bin/spark-shell
      18/08/16 23:20:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
      Setting default log level to "WARN".
      To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
      Spark context Web UI available at http://10.0.0.76:4040
      Spark context available as 'sc' (master = local[*], app id = local-1534486813159).
      Spark session available as 'spark'.
      Welcome to
            ____              __
           / __/__  ___ _____/ /__
          _\ \/ _ \/ _ `/ __/  '_/
         /___/ .__/\_,_/_/ /_/\_\   version 2.3.3-SNAPSHOT
            /_/
               
      Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131)
      Type in expressions to have them evaluated.
      Type :help for more information.
      
      scala> spark.range(1).show
      +---+
      | id|
      +---+
      |  0|
      +---+
      
      
      scala> 
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                rednaxelafx Kris Mok
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: