Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
None
Description
Currently, the kernel creates and stores the SparkSession in a field to share between interpreters. If the user closes a SparkSession and creates a new one, then the Kernel still returns the original. Users may need to restart Spark sessions for long-running notebooks or to deal with Spark errors without losing datasets that have been pulled back to the notebook.
I think that Toree should always return the current Spark session by calling SparkSession.builder.getOrCreate.
Attachments
Issue Links
- links to