Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33699

Spark web ui kill application but the thread still exists background

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.3
    • Fix Version/s: None
    • Component/s: Spark Core, Web UI
    • Labels:
      None
    • Environment:

      spark-2.4.3

      CentOS 7.0

      JDK 1.8.0_201

      Description

      When I kill an application on the Web UI (which I submit with standalone-client mode), it seems to be killed already; But when I use 'jps' command I can still see the application running background. This is my demo code to reappear this problem.

      // code placeholder
      val rdd = sparkSession.sparkContext.parallelize(List(1, 2, 3, 4)).repartition(4)
      
      //section 1: 
      //do something for a long time
      val rdd2 = rdd.map(x => { 
        for (i <- 1 to 300) {
          for (j <- 1 to 999999999) {
          }
      
          if (i % 10 == 0) {
            println(i + " rdd map process running!")
          }
      
        }
      
        x * 2
      })
      rdd2.take(10).foreach(println(_))
      
      //section 2: 
      //do something for a long time in driver
      for (i <- 1 to 500) {
        for (j <- 1 to 999999999) {
          }
      
        if (i % 10 == 0) {
          println(i + " main process running!")
        }
      }
      
      

      And, 

      1. If I kill the application on web ui when section 1 rdd.map process, it can be stopped clearly;
      2. If I kill the application on web ui when section 2 do something in the driver, and then it stll running background.

      So, is it a bug of spark and how to solve it ?

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              xiaoming666 Xiaoming W
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: