[SPARK-38930] Spark Executors status always is KILLED

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2, 3.1.3
• Component/s: Spark Core
• Flags: Important
• Assignee: Unassigned
• Reporter: sichun zhai
• Created: 18/Apr/22 08:50
• Updated: 18/Apr/22 09:17

Description

In  standalone deploy mode, try run spark org.apache.spark.examples.SparkPi or other spark program ,the ui always show Executors  status is killed

spark worker error log :

22/04/18 17:08:27 INFO Worker: Asked to kill executor app-20220418170822-0039/0
22/04/18 17:08:27 INFO ExecutorRunner: Runner thread for executor app-20220418170822-0039/0 interrupted
22/04/18 17:08:27 INFO ExecutorRunner: Killing process!
22/04/18 17:08:27 DEBUG SizeBasedRollingPolicy: 55 + 18896 > 1073741824
22/04/18 17:08:27 DEBUG RollingFileAppender: Closed file /opt/spark/work/app-20220418170822-0039/0/stderr
22/04/18 17:08:27 DEBUG RollingFileAppender: Closed file /opt/spark/work/app-20220418170822-0039/0/stdout
22/04/18 17:08:27 INFO ExecutorRunner: exitCode:Some(143)
22/04/18 17:08:27 INFO Worker: Executor app-20220418170822-0039/0 finished with state KILLED exitStatus 143

 

haved patch  https://github.com/apache/spark/pull/12012

run SparkPi commad:

/opt/app/applications/bd-spark/bin/run-example  {}class org.apache.spark.examples.SparkPi{-}  -master spark://10.205.90.120:7077,10.205.90.131:7077 --deploy-mode cluster --driver-java-options "-Dlog4j.configuration=file:/opt/app/applications/bd-spark/conf/log4j.properties" --conf spark.executor.extraJavaOptions="-Dlog4j.configuration=file:/opt/app/applications/bd-spark/conf/log4j.properties"