[SPARK-23502] Support async init of spark context during spark-shell startup

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.0.0
• Component/s: Spark Shell
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Sital Kedia
• Created: 23/Feb/18 19:13
• Updated: 21/May/19 04:11
• Resolved: 21/May/19 04:11

Description

Currently, whenever a user starts the spark shell, we initialize the spark context before returning the prompt to the user. In environments, where spark context initialization takes several seconds, it is not a very good user experience for the user to wait for the prompt. Instead of waiting for the initialization of spark context, we can initialize it in the background while we return the prompt to the user as soon as possible. Please note that even if we return the prompt to the user soon, we still need to make sure to wait for the spark context initialization to complete before any query is executed. 

Please note that the scala interpreter already does very similar async initialization in order to return the prompt to the user faster - https://github.com/scala/scala/blob/v2.12.2/src/repl/scala/tools/nsc/interpreter/ILoop.scala#L414. We will be emulating the behavior for Spark.