[HUDI-945] Cleanup spillable map files eagerly as part of close

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Component/s: writer-core
• Labels: core-flow-ds pull-request-available sev:high
• Assignee: sivabalan narayanan
• Reporter: Balaji Varadarajan
• Created: 25/May/20 18:32
• Updated: 29/Jul/22 06:28
• Resolved: 29/Jul/22 06:28

Description

Currently, files used by external spillable map are deleted on exits. For spark-streaming/deltastreamer continuous-mode cases which runs several iterations, it is better to eagerly delete files on closing the handles using it. 

We need to eagerly delete the files on following cases:

1. HoodieMergeHandle
2. HoodieMergedLogRecordScanner
3. SpillableMapBasedFileSystemView

Issue Links

• Blocked: HUDI-1652 DiskBasedMap:As time goes by, the number of /temp/***** file handles held by the executor process is increasing
• links to: GitHub Pull Request #2069