[HUDI-3425] Clean up spill path created by Hudi during uneventful shutdown

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Cannot Reproduce
• Fix Version/s: 0.12.1
• Component/s: compaction
• Story Points: 2
• Assignee: sivabalan narayanan
• Reporter: sivabalan narayanan
• Created: 14/Feb/22 20:25
• Updated: 09/Aug/23 08:57
• Resolved: 15/Sep/22 23:40

Description

Hudi spill path not getting cleared when containers getting killed abruptly. 

 

When yarn kills the containers abruptly for any reason while hudi stage is in progress then the spill path created by hudi on the disk is not cleaned and as a result of which the nodes on the cluster start running out of space. We need to clear the spill path manually to free out disk space.

 

Ref issue: https://github.com/apache/hudi/issues/4771

Issue Links

• links to: issue-4771