[FLINK-25403] Make BlobServer/BlobCache compatible with local working directories

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.15.0
• Component/s: Runtime / Coordination
• Assignee: Till Rohrmann
• Reporter: Till Rohrmann
• Created: 21/Dec/21 15:32
• Updated: 19/Jan/22 15:47
• Resolved: 19/Jan/22 15:47

Description

With FLIP-198 Flink processes can now have a working directory that they use to store "semi-persistent" information. The stored blobs by the BlobServer and the BlobCache can be candidates for such a storage because in case of a recovery we might not have to re-download them.

In order to make the BlobServer and BlobCache work together with the local working directories, they have be able to detect corrupted blobs (e.g. due to process crashes when downloading blobs) and delete them. Therefore, I propose to let the BlobServer and BlobCache remember which blobs they have verified (by recomputing their checksums) and to delete those that did not pass.

Issue Links

• requires: FLINK-25402 FLIP-198: Working directory for Flink processes

Sub-Tasks

• Store blobs in <WORKING_DIR>/blobs: Closed
• Let BlobServer/BlobCache detect & delete corrupted blobs: Closed
• Make sure that FileSystemBlobStore flushes all writes to disk to persist stored blobs: Closed
• Allow BlobServer/BlobCache to clean up unused blobs after recovering from working directory: Closed