Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20624 SPIP: Add better handling for node shutdown
  3. SPARK-32745

Add an option to reject block migrations when under disk pressure

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Spark Core
    • None

    Description

      If we have a very low minimum number of nodes, we may fill up the disks of executors by migrating shuffle files to them. We should add a configuration option to only accept remote shuffle blocks when there is sufficient free disk space.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            holden Holden Karau

            Dates

              Created:
              Updated:

              Slack

                Issue deployment