[MAPREDUCE-2572] Throttle the deletion of data from the distributed cache

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.23.0
• Fix Version/s: 0.23.0
• Component/s: distributed-cache
• Assignee: Robert Joseph Evans
• Reporter: Robert Joseph Evans
• Created: 07/Jun/11 14:01
• Updated: 15/Nov/11 00:48
• Resolved: 09/Sep/11 14:03

Description

When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling.

On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.