Major Description
When the underlying SQLContext is instantiated for a SparkSession, the instance is saved as a class attribute and returned from subsequent calls to SQLContext.getOrCreate(). If the SparkContext is stopped and a new one started, the SQLContext class attribute is never cleared so any code which calls SQLContext.getOrCreate() will get a SQLContext with a reference to the old, unusable SparkContext.
A similar issue was identified and fixed for SparkSession in SPARK-19055, but the fix did not change SQLContext as well. I ran into this because mllib still uses SQLContext.getOrCreate() under the hood.
I've already written a fix for this, which I'll be sharing in a PR, that clears the class attribute on SQLContext when the SparkSession is stopped. Another option would be to deprecate SQLContext.getOrCreate() entirely since the corresponding Scala method is itself deprecated. That seems like a larger change for a relatively minor issue, however.
Attachments
Issue Links
- links to