[MAPREDUCE-1568] TrackerDistributedCacheManager should clean up cache in a background thread

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.22.0
• Fix Version/s: 0.21.0
• Hadoop Flags: Reviewed

Release Note

MAPREDUCE-1568. TrackerDistributedCacheManager should clean up cache in a background thread. (Scott Chen via zshao)

• Assignee: Scott Chen
• Reporter: Scott Chen
• Created: 06/Mar/10 01:43
• Updated: 24/Aug/10 21:21
• Resolved: 29/Apr/10 22:43

Description

Right now the TrackerDistributedCacheManager do the clean up with the following code path:

TaskRunner.run() -> 
TrackerDistributedCacheManager.setup() ->
TrackerDistributedCacheManager.getLocalCache() -> 
TrackerDistributedCacheManager.deleteCache()

The deletion of the cache files can take a long time and it should not be done by a task. We suggest that there should be a separate thread checking and clean up the cache files.

Issue Links

• is blocked by: MAPREDUCE-1538 TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit