[SPARK-33699] Spark web ui kill application but the thread still exists background

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.3
• Component/s: Spark Core, Web UI

Environment

spark-2.4.3

CentOS 7.0

JDK 1.8.0_201

• Assignee: Unassigned
• Reporter: Xiaoming W
• Created: 08/Dec/20 02:49
• Updated: 22/Dec/20 06:50

Description

When I kill an application on the Web UI (which I submit with standalone-client mode), it seems to be killed already; But when I use 'jps' command I can still see the application running background. This is my demo code to reappear this problem.

// code placeholder
val rdd = sparkSession.sparkContext.parallelize(List(1, 2, 3, 4)).repartition(4)

//section 1: 
//do something for a long time
val rdd2 = rdd.map(x => { 
  for (i <- 1 to 300) {
    for (j <- 1 to 999999999) {
    }

    if (i % 10 == 0) {
      println(i + " rdd map process running!")
    }

  }

  x * 2
})
rdd2.take(10).foreach(println(_))

//section 2: 
//do something for a long time in driver
for (i <- 1 to 500) {
  for (j <- 1 to 999999999) {
    }

  if (i % 10 == 0) {
    println(i + " main process running!")
  }
}

And, 

1. If I kill the application on web ui when section 1 rdd.map process, it can be stopped clearly;
2. If I kill the application on web ui when section 2 do something in the driver, and then it stll running background.

So, is it a bug of spark and how to solve it ?