[SPARK-37004] Job cancellation causes py4j errors on Jupyter due to pinned thread mode

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.1, 3.3.0
• Component/s: PySpark
• Target Version/s: 3.2.1
• Assignee: Hyukjin Kwon
• Reporter: Xiangrui Meng
• Created: 14/Oct/21 05:15
• Updated: 12/Dec/22 18:11
• Resolved: 06/Dec/21 08:36

Description

Spark 3.2.0 turned on py4j pinned thread mode by default (SPARK-35303). However, in a jupyter notebook, after I cancel (interrupt) a long-running Spark job, the next Spark command will fail with some py4j errors. See attached notebook for repro.

Cannot reproduce the issue after I turn off pinned thread mode .

Issue Links

• is caused by: SPARK-35303 Enable pinned thread mode by default
• links to: [Github] Pull Request #34814 (HyukjinKwon)
• links to: [Github] Pull Request #34814 (HyukjinKwon)