[SPARK-43952] Cancel Spark jobs not only by a single "jobgroup", but allow multiple "job tags"

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Spark Core
• Assignee: Juliusz Sompolski
• Reporter: Juliusz Sompolski
• Created: 02/Jun/23 15:23
• Updated: 13/Jul/23 09:04
• Resolved: 21/Jun/23 00:22

Description

Currently, the only way to cancel running Spark Jobs is by using SparkContext.cancelJobGroup, using a job group name that was previously set using SparkContext.setJobGroup. This is problematic if multiple different parts of the system want to do cancellation, and set their own ids.

For example, https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala#L133 sets it's own job group, which may override job group set by user. This way, if user cancels the job group they set, it will not cancel these broadcast jobs launches from within their jobs...

As a solution, consider adding SparkContext.addJobTag / SparkContext.removeJobTag, which would allow to have multiple "tags" on the jobs, and introduce SparkContext.cancelJobsByTag to allow more flexible cancelling of jobs.

Issue Links

• links to: [Github] Pull Request #41440 (juliuszsompolski)
• links to: [Github] Pull Request #41979 (ulysses-you)
• links to: [Github] Pull Request #41979 (ulysses-you)