Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1568

TrackerDistributedCacheManager should clean up cache in a background thread

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      MAPREDUCE-1568. TrackerDistributedCacheManager should clean up cache in a background thread. (Scott Chen via zshao)
      Show
      MAPREDUCE-1568 . TrackerDistributedCacheManager should clean up cache in a background thread. (Scott Chen via zshao)

      Description

      Right now the TrackerDistributedCacheManager do the clean up with the following code path:

      TaskRunner.run() -> 
      TrackerDistributedCacheManager.setup() ->
      TrackerDistributedCacheManager.getLocalCache() -> 
      TrackerDistributedCacheManager.deleteCache()
      

      The deletion of the cache files can take a long time and it should not be done by a task. We suggest that there should be a separate thread checking and clean up the cache files.

      1. MAPREDUCE-1568.txt
        6 kB
        Scott Chen
      2. MAPREDUCE-1568-v2.1.txt
        16 kB
        Scott Chen
      3. MAPREDUCE-1568-v2.txt
        15 kB
        Scott Chen
      4. MAPREDUCE-1568-v3.1.txt
        17 kB
        Scott Chen
      5. MAPREDUCE-1568-v3.txt
        17 kB
        Scott Chen

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Scott Chen
              Reporter:
              Scott Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development