[MAPREDUCE-7457] Limit number of spill files getting created

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: mrv2
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Mudit Sharma
• Reporter: Mudit Sharma
• Created: 06/Oct/23 17:08
• Updated: 27/Jan/24 00:39
• Resolved: 30/Oct/23 22:59

Description

Hi,

 

We have been facing some issues where many of our cluster node disks go full because of some rogue applications creating a lot of spill data

We wanted to fail the app if more than a threshold amount of spill files are written

Please let us know if any such capability is supported

 

If the capability is not there, we are proposing it to support it via a config, we have added a PR for the same: https://github.com/apache/hadoop/pull/6155  please let us know your thoughts on it

Issue Links

• links to: GitHub Pull Request #6155