[SPARK-20217] Executor should not fail stage if killed task throws non-interrupted exception

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.0
• Component/s: Spark Core
• Assignee: Eric Liang
• Reporter: Eric Liang
• Created: 04/Apr/17 23:59
• Updated: 11/May/17 14:03
• Resolved: 06/Apr/17 02:37

Description

This is reproducible as follows. Run the following, and then use SparkContext.killTaskAttempt to kill one of the tasks. The entire stage will fail since we threw a RuntimeException instead of InterruptedException.

We should probably unconditionally return TaskKilled instead of TaskFailed if the task was killed by the driver, regardless of the actual exception thrown.

spark.range(100).repartition(100).foreach { i =>
  try {
    Thread.sleep(10000000)
  } catch {
    case t: InterruptedException =>
      throw new RuntimeException(t)
  }
}

Based on the code in TaskSetManager, I think this also affects kills of speculative tasks. However, since the number of speculated tasks is few, and usually you need to fail a task a few times before the stage is cancelled, probably no-one noticed this in production.

Issue Links

• breaks: SPARK-20358 Executors failing stage on interrupted exception thrown by cancelled tasks
• is duplicated by: SPARK-19354 Killed tasks are getting marked as FAILED
• links to: [Github] Pull Request #17531 (ericl)