[FLINK-25436] Allow BlobServer/BlobCache to clean up unused blobs after recovering from working directory

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.15.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Till Rohrmann
• Reporter: Till Rohrmann
• Created: 23/Dec/21 16:12
• Updated: 19/Jan/22 15:47
• Resolved: 19/Jan/22 15:47

Description

In order to let the BlobServer and the BlobCache properly clean up unused blobs that are recovered from the working directory, we have to register them for clean up and offer hooks to delete irrelevant job artifacts.

I propose to scan the blobStorage directory at startup and to register for transient blobs the expiry timeouts. Moreover, for the BlobServer we need to add a retainJobs method that deletes all jobs that are not in the given list of JobIDs. Last but not least we also need to register the permanent blobs in the PermanentBlobCacheService so that they get timed out if not used anymore.

Issue Links

• links to: GitHub Pull Request #18194