[SPARK-32745] Add an option to reject block migrations when under disk pressure

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Holden Karau
• Created: 31/Aug/20 02:57
• Updated: 31/Aug/20 02:57

Description

If we have a very low minimum number of nodes, we may fill up the disks of executors by migrating shuffle files to them. We should add a configuration option to only accept remote shuffle blocks when there is sufficient free disk space.