[SPARK-35303] Enable pinned thread mode by default

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 04/May/21 04:09
• Updated: 09/Feb/24 06:53
• Resolved: 18/Jun/21 03:17

Description

Pinned thread mode was added at SPARK-22340. We should enable it back to map Python thread to JVM thread in order to prevent potential issues such as thread local inheritance.

Issue Links

• causes: SPARK-37004 Job cancellation causes py4j errors on Jupyter due to pinned thread mode
• causes: SPARK-37012 Disable pinned thread mode by default
• links to: [Github] Pull Request #32429 (HyukjinKwon)
• links to: [Github] Pull Request #32429 (HyukjinKwon)
• links to: [Github] Pull Request #32962 (HyukjinKwon)
• links to: [Github] Pull Request #32962 (HyukjinKwon)
• links to: GitHub Pull Request #32429