[SPARK-19807] Add reason for cancellation when a stage is killed using web UI

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: Web UI
• Assignee: shaolinliu
• Reporter: Jacek Laskowski
• Created: 03/Mar/17 08:39
• Updated: 05/Apr/17 12:49
• Resolved: 05/Apr/17 12:48

Description

When a user kills a stage using web UI (in Stages page), StagesTab.handleKillRequest requests SparkContext to cancel the stage without giving a reason. SparkContext has cancelStage(stageId: Int, reason: String) that Spark could use to pass the information for monitoring/debugging purposes.

scala> sc.range(0, 5, 1, 1).mapPartitions { nums => { Thread.sleep(60 * 1000); nums } }.count

Use http://localhost:4040/stages/ and click Kill.

org.apache.spark.SparkException: Job 0 cancelled because Stage 0 was cancelled
  at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1486)
  at org.apache.spark.scheduler.DAGScheduler.handleJobCancellation(DAGScheduler.scala:1426)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleStageCancellation$1.apply$mcVI$sp(DAGScheduler.scala:1415)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleStageCancellation$1.apply(DAGScheduler.scala:1408)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleStageCancellation$1.apply(DAGScheduler.scala:1408)
  at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofInt.foreach(ArrayOps.scala:234)
  at org.apache.spark.scheduler.DAGScheduler.handleStageCancellation(DAGScheduler.scala:1408)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1670)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1656)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1645)
  at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
  at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2019)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2040)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2059)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2084)
  at org.apache.spark.rdd.RDD.count(RDD.scala:1158)
  ... 48 elided

Issue Links

• links to: [Github] Pull Request #17258 (shaolinliu)