[SPARK-47986] [CONNECT][PYTHON] Unable to create a new session when the default session is closed by the server

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0, 3.5.1
• Fix Version/s: 4.0.0
• Component/s: Connect, PySpark
• Labels: pull-request-available
• Assignee: Niranjan Jayakar
• Reporter: Niranjan Jayakar
• Created: 25/Apr/24 08:04
• Updated: 21/Jun/24 14:26
• Resolved: 09/May/24 08:23

Description

When the server closes a session, usually after a cluster restart, the client is unaware of this until it receives an error.

Once it does so, there is no way for the client to create a new session since the stale sessions are still recorded as default and active sessions.

The only solution currently is to restart the Python interpreter on the client, or to reach into the session builder and change the active or default session.

Issue Links

• links to: GitHub Pull Request #46221
• links to: GitHub Pull Request #46435
• links to: GitHub Pull Request #47008