Critical Description
PySpark cancellation calls SparkEnv#destroyPythonWorker. Since there is one python worker per process, this would seem like a sensible thing to do. Unfortunately, this method actually destroys a python daemon, and all associated workers, which generally means that we can cause failures in unrelated Pyspark jobs.
The severity of this bug is limited by the fact that the Pyspark daemon is easily recreated, so the tasks will succeed after being restarted.
Attachments
Issue Links
- links to
