[SPARK-29152] Spark Executor Plugin API shutdown is not proper when dynamic allocation enabled

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Rakesh Raushan
• Reporter: jobit mathew
• Created: 18/Sep/19 13:07
• Updated: 02/Jun/21 16:29
• Resolved: 10/Dec/19 22:24

Description

Issue Description

Spark Executor Plugin API shutdown handling is not proper, when dynamic allocation enabled .Plugin's shutdown method is not processed when dynamic allocation is enabled and executors become dead after inactive time.

Test Precondition
1. Create a plugin and make a jar named SparkExecutorplugin.jar

import org.apache.spark.ExecutorPlugin;
public class ExecutoTest1 implements ExecutorPlugin{
public void init()

{ System.out.println("Executor Plugin Initialised."); }

public void shutdown()

{ System.out.println("Executor plugin closed successfully."); }

}

2. Create the jars with the same and put it in folder /spark/examples/jars

Test Steps

1. launch bin/spark-sql with dynamic allocation enabled

./spark-sql --master yarn --conf spark.executor.plugins=ExecutoTest1 --jars /opt/HA/C10/install/spark/spark/examples/jars/SparkExecutorPlugin.jar --conf spark.dynamicAllocation.enabled=true --conf spark.dynamicAllocation.initialExecutors=2 --conf spark.dynamicAllocation.minExecutors=1

2 create a table , insert the data and select * from tablename
3.Check the spark UI Jobs tab/SQL tab
4. Check all Executors(executor tab will give all executors details) application log file for Executor plugin Initialization and Shutdown messages or operations.
Example /yarn/logdir/application_1567156749079_0025/container_e02_1567156749079_0025_01_000005/ stdout

5. Wait for the executor to be dead after the inactive time and check the same container log
6. Kill the spark sql and check the container log for executor plugin shutdown.

Expect Output

1. Job should be success. Create table ,insert and select query should be success.

2.While running query All Executors log should contain the executor plugin Init messages or operations.
"Executor Plugin Initialised.

3.Once the executors are dead ,shutdown message should be there in log file.
“ Executor plugin closed successfully.

4.Once the sql application closed ,shutdown message should be there in log.
“ Executor plugin closed successfully".

Actual Output

Shutdown message is not called when executor is dead after inactive time.

Observation
Without dynamic allocation Executor plugin is working fine. But after enabling dynamic allocation,Executor shutdown is not processed.

Issue Links

• causes: SPARK-35610 Memory leak in Spark interpreter
• relates to: SPARK-24918 Executor Plugin API
• links to: GitHub Pull Request #26315
• links to: GitHub Pull Request #26506
• links to: GitHub Pull Request #26810
• links to: GitHub Pull Request #26841
• links to: GitHub Pull Request #26901