[LIVY-1002] LivyThriftSessionManager killSparkApp may have NPE

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 0.8.0
• Component/s: Thriftserver
• Flags: Important

Language

scala

• Assignee: Unassigned
• Reporter: wangdengshan
• Created: 30/May/24 08:42
• Updated: 21/Jun/24 08:07
• Resolved: 21/Jun/24 08:07

Description

There may be a null pointer exception in killSparkApp. sparkApps will only put after the session is created. When the creation process fails, the kill application will not be successful.

Below is my runtime stack：

2024-05-30 03:39:12 [ERROR] [LivyServer2-Background-Pool: Thread-139977] org.apache.livy.thriftserver.LivyThriftSessionManager:error:58 - Kill application failed
java.lang.NullPointerException
        at org.apache.livy.thriftserver.LivyThriftSessionManager.killSparkApp$1(LivyThriftSessionManager.scala:148)
        at org.apache.livy.thriftserver.LivyThriftSessionManager.getLivySession(LivyThriftSessionManager.scala:180)
        at org.apache.livy.thriftserver.LivyExecuteStatementOperation.execute(LivyExecuteStatementOperation.scala:157)
        at org.apache.livy.thriftserver.LivyExecuteStatementOperation$$anon$1$$anon$2.run(LivyExecuteStatementOperation.scala:102)
        at org.apache.livy.thriftserver.LivyExecuteStatementOperation$$anon$1$$anon$2.run(LivyExecuteStatementOperation.scala:99)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2248)
        at org.apache.livy.thriftserver.LivyExecuteStatementOperation$$anon$1.run(LivyExecuteStatementOperation.scala:112)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

 

The key code is here

    def killSparkApp = {
      try {
        sparkApps.get(sessionHandle.toString).foreach(sparkApp => {
          info("Start kill spark job")
          sparkApp.kill()
          info("End kill spark job")
        })
      } catch {
        case ex: Throwable => error("Kill application failed", ex)
      } finally {
        sparkApps.remove(sessionHandle.toString)
      }
    }