[SPARK-32991] RESET can clear StaticSQLConfs

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.1.0
• Component/s: SQL
• Target Version/s: 3.1.0
• Assignee: Kent Yao 2
• Reporter: Herman van Hövell
• Created: 24/Sep/20 22:13
• Updated: 16/Dec/20 14:37
• Resolved: 16/Dec/20 14:37

Description

The RESET command can clear a sessions' static SQL configurations, when that static SQL configuration was set on a SparkSession that uses a pre-existing SparkContext. Here is repro:

// Blow away any pre-existing session thread locals
org.apache.spark.sql.SparkSession.clearDefaultSession()
org.apache.spark.sql.SparkSession.clearActiveSession()

// Create new session and explicitly set a spark context
val newSession = org.apache.spark.sql.SparkSession.builder
 .sparkContext(sc)
 .config("spark.sql.globalTempDatabase", "bob")
 .getOrCreate()
assert(newSession.conf.get("spark.sql.globalTempDatabase") == "bob")
newSession.sql("reset")
assert(newSession.conf.get("spark.sql.globalTempDatabase") == "bob") // Boom!

The problem is that RESET assumes it can use the SparkContext's configurations as its default.

 

 

Issue Links

• is blocked by: SPARK-33718 SharedState shall be an system-wide singleton
• links to: [Github] Pull Request #30045 (yaooqinn)
• links to: [Github] Pull Request #30642 (yaooqinn)