[SPARK-48970] Avoid using SparkSession.getActiveSession in spark ML reader/writer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: ML, PySpark
• Labels: pull-request-available
• Assignee: Weichen Xu
• Reporter: Weichen Xu
• Created: 23/Jul/24 00:37
• Updated: 24/Jul/24 15:06
• Resolved: 23/Jul/24 11:19

Description

`SparkSession.getActiveSession` is thread-local session, but spark ML reader / writer might be executed in different threads which causes `SparkSession.getActiveSession` returning None.

Issue Links

• links to: GitHub Pull Request #47453