[SPARK-34337] Reject disk blocks when out of disk space

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.1, 3.1.2, 3.2.0
• Component/s: Spark Core
• Target Version/s: 4.0.0
• Assignee: Unassigned
• Reporter: Holden Karau
• Created: 02/Feb/21 22:25
• Updated: 09/Aug/23 18:34

Description

Now that we have the ability to store shuffle blocks on dis-aggregated storage (when configured) we should add the option to reject storing blocks locally on an executor at a certain disk pressure threshold.

Sub-Tasks

• Allow users to configure a maximum amount of remote shuffle block storage: Resolved
• Monitor disk usage and use to reject blocks when under disk pressure: Open