[FLINK-23354] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.14.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Zhilong Hong
• Reporter: Zhilong Hong
• Created: 12/Jul/21 07:39
• Updated: 23/Sep/21 18:09
• Resolved: 29/Jul/21 09:54

Description

This is the part 3 of the optimization related to task deployments. For more details about the overall description and the part 1, please see FLINK-23005. For more details about the part 2 please see FLINK-23218.

Currently a TaskExecutor uses BlobCache to cache the blobs transported from JobManager. The caches are the local file stored on the TaskExecutor. The blob cache will not be cleaned up until one hour after the related job is finished. In FLINK-23218, we are going to distribute the cached ShuffleDescriptors via blob. When large amount of failovers happen, there will be a lot of cache stored on local disk. The blob cache will occupy large amount of disk space. In extreme cases, the blob would blow up the disk space.

So we need to add a limit size for the ShuffleDescriptors stored in PermanentBlobCache on TaskExecutor, as described in the comments of FLINK-23218. The main idea is to add a size limit and and delete the blobs in LRU order if the size limit is exceeded. Before a blob item is cached, TaskExecutor will firstly check the overall size of cache. If the overall size exceeds the limit, the blob will be deleted in LRU order until the limit is not exceeded anymore. For the blob cache that is deleted, if it is used afterwards, it will be downloaded from the HA or the blob server again.

The default value of the size limit for the ShuffleDescriptors in PermanentBlobCache on TaskExecutor will be 100 MiB.

Issue Links

• links to: GitHub Pull Request #16498