[SPARK-40021] `Cancel workflow` can not cancel some jobs

• Type: Test
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: Project Infra
• Assignee: Unassigned
• Reporter: Ruifeng Zheng
• Created: 09/Aug/22 14:43
• Updated: 12/Dec/22 18:11

Description

I have been observing this behavior for a long time:

sometime I want to cancel the workflow to release resource for other workflows, so I click the Cancel workflow button, most jobs in current workflow will be canceled in several minutes, but some jobs can never be killed, including:

• Run / Build modules: pyspark-sql, pyspark-mllib, pyspark-resource
• Run / Build modules: pyspark-core, pyspark-streaming, pyspark-ml
• Run / Build modules: pyspark-pandas
• Run / Build modules: pyspark-pandas-slow
• Run / Linters, licenses, dependencies and documentation generation