Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20624 SPIP: Add better handling for node shutdown
  3. SPARK-20628

Keep track of nodes which are going to be shut down & avoid scheduling new tasks

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.2.0, 2.3.0
    • 3.1.0
    • Spark Core
    • None

    Description

      Keep track of nodes which are going to be shut down. We considered adding this for YARN but took a different approach, for instances where we can't control instance termination though (EC2, GCE, etc.) this may make more sense.

      Attachments

        Issue Links

          causes

          Sub-task - The sub-task of the issue SPARK-30981 Fix flaky "Test basic decommissioning" test

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue SPARK-46817 Fix `spark-daemon.sh` usage by adding `decommission` command

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-31011 Failed to register signal handler for PWR

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-19941 Spark should not schedule tasks on executors on decommissioning YARN nodes

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-19941 Spark should not schedule tasks on executors on decommissioning YARN nodes

          • Major - Major loss of function.
          • Resolved
          is related to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-28022 k8s pod affinity to achieve cloud native friendly autoscaling

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-30610 spark worker graceful shutdown

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          links to

          Pull request #19045 [Github] Pull Request #19045 (holdenk)

          Pull request #19267 [Github] Pull Request #19267 (juanrh)

          Web Link GitHub Pull Request #19045

          Web Link GitHub Pull Request #19267

          Web Link GitHub Pull Request #26440

          Web Link GitHub Pull Request #27588

          Activity

            People

              holden Holden Karau
              holden Holden Karau
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: