[SPARK-27253] SparkSession clone discards SQLConf overrides in favor of SparkConf defaults

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Chakravarthi
• Reporter: Jose Torres
• Created: 22/Mar/19 20:55
• Updated: 12/Dec/22 18:11
• Resolved: 01/Apr/19 00:33

Description

SparkSession.cloneSession() is normally supposed to create a child session which inherits all the SQLConf values of its parent session. But when a SQL conf is given a global default through the SparkConf, this does not happen; the child session will receive the SparkConf default rather than its parent's SQLConf override.

 

This is particularly impactful in structured streaming, as the microbatches run in a cloned child session.

Issue Links

• links to: GitHub Pull Request #24189
• links to: GitHub Pull Request #24279
• links to: GitHub Pull Request #24326
• links to: GitHub Pull Request #24540