[SPARK-42663] Fix `default_session` to work properly

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Component/s: Connect, Pandas API on Spark
• Assignee: Unassigned
• Reporter: Haejoon Lee
• Created: 03/Mar/23 16:13
• Updated: 18/Apr/23 05:20
• Resolved: 18/Apr/23 05:20

Description

Currently, default_session is not working properly in Spark Connect as below:

>>> spark = default_session()
>>> spark.conf.set("default_index_type", "sequence")
>>> spark.conf.get("default_index_type")
'sequence'
>>>
>>> spark = default_session()
>>> spark.conf.get("default_index_type")
Traceback (most recent call last):
...
pyspark.errors.exceptions.connect.SparkConnectGrpcException: (java.util.NoSuchElementException) default_index_type

It should work as expected in regular PySpark as below:

>>> spark = default_session()
>>> spark.conf.set("default_index_type", "sequence")
>>> spark.conf.get("default_index_type")
'sequence'
>>>
>>> spark = default_session()
>>> spark.conf.get("default_index_type")
'sequence'

Issue Links

• is duplicated by: SPARK-42859 Basic support for pandas API on Spark