[SPARK-17261] Using HiveContext after re-creating SparkContext in Spark 2.0 throws "Java.lang.illegalStateException: Cannot call methods on a stopped sparkContext"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: PySpark

Environment

Amazon AWS EMR 5.0

• Assignee: Jeff Zhang
• Reporter: Rahul Jain
• Created: 26/Aug/16 11:48
• Updated: 11/Mar/17 20:45
• Resolved: 02/Sep/16 17:08

Description

After stopping SparkSession if we recreate it and use HiveContext in it. it will throw error.

Steps to reproduce:
spark = SparkSession.builder.enableHiveSupport().getOrCreate()
spark.sql("show databases")
spark.stop()
spark = SparkSession.builder.enableHiveSupport().getOrCreate()
spark.sql("show databases")

"Java.lang.illegalStateException: Cannot call methods on a stopped sparkContext"

Above error occurs only in case of Pyspark not in SparkShell

Issue Links

• links to: [Github] Pull Request #14857 (zjffdu)
• links to: [Github] Pull Request #17261 (elviento)
• links to: [Github] Pull Request #17262 (elviento)