Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20843

Cannot gracefully kill drivers which take longer than 10 seconds to die

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.1
    • Fix Version/s: 2.1.2, 2.2.0
    • Component/s: Spark Core
    • Labels:

      Description

      Commit https://github.com/apache/spark/commit/1c9a386c6b6812a3931f3fb0004249894a01f657 changed the behavior of driver process termination. Whereas before `Process.destroyForcibly` was never called, now it is called (on Java VM's supporting that API) if the driver process does not die within 10 seconds.

      This prevents apps which take longer than 10 seconds to shutdown gracefully from shutting down gracefully. For example, streaming apps with a large batch duration (say, 30 seconds+) can take minutes to shutdown.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zsxwing Shixiong Zhu
                Reporter:
                michael Michael Allman
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: