Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31246

GracefulShutdown does not work when application is terminated from RestSubmissionClient or YarnClient

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.3
    • Fix Version/s: None
    • Component/s: Java API
    • Labels:
      None
    • Environment:

      spark-2.4.3

      Description

      While starting the Spark Application "spark.streaming.stopGracefullyOnShutdown" is set to true

      try to terminate the application programatically using JAVA API

      1 using RestSubmissionClient client = new RestSubmissionClient(masterUrl);
                   SubmitRestProtocolResponse statusResponse = client.killSubmission(submissionId);  

       

      2. using getYarnClient().killApplication(appId);

       

      In both the cases Application dose not stop gracefully

       

      But killing the Application using  

       

      Kill -SIGTERM <pid>  will shutdown the application gracefully.

      Expected : Application should have terminated gracefully in all cases when spark.streaming.stopGracefullyOnShutdown is set

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Gaja Gajanan Hebbar
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: