[SPARK-31246] GracefulShutdown does not work when application is terminated from RestSubmissionClient or YarnClient

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.4.3
• Component/s: Java API
• Labels: bulk-closed

Environment

spark-2.4.3

• Assignee: Unassigned
• Reporter: Gajanan Hebbar
• Created: 25/Mar/20 08:57
• Updated: 25/May/21 01:53
• Resolved: 25/May/21 01:41

Description

While starting the Spark Application "spark.streaming.stopGracefullyOnShutdown" is set to true

try to terminate the application programatically using JAVA API

1 using RestSubmissionClient client = new RestSubmissionClient(masterUrl);
             SubmitRestProtocolResponse statusResponse = client.killSubmission(submissionId);  

 

2. using getYarnClient().killApplication(appId);

 

In both the cases Application dose not stop gracefully

 

But killing the Application using  

 

Kill -SIGTERM <pid>  will shutdown the application gracefully.

Expected : Application should have terminated gracefully in all cases when spark.streaming.stopGracefullyOnShutdown is set