Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1860

Standalone Worker cleanup should not clean up running executors

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 1.0.0
    • 1.2.0
    • Deploy
    • None

    Description

      The default values of the standalone worker cleanup code cleanup all application data every 7 days. This includes jars that were added to any executors that happen to be running for longer than 7 days, hitting streaming jobs especially hard.

      Executor's log/data folders should not be cleaned up if they're still running. Until then, this behavior should not be enabled by default.

      Attachments

        Issue Links

          blocks

          Task - A task that needs to be done. SPARK-3805 Enable Standalone worker cleanup by default

          • Major - Major loss of function.
          • Closed
          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-1154 Spark fills up disk with app-* folders

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is duplicated by

          New Feature - A new feature of the product, which has yet to be developed. SPARK-786 Clean up old work directories in standalone worker

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #2609 [Github] Pull Request #2609 (mccheah)

          Activity

            People

              Unassigned Unassigned
              ilikerps Aaron Davidson
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: