Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1568

TrackerDistributedCacheManager should clean up cache in a background thread

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      MAPREDUCE-1568. TrackerDistributedCacheManager should clean up cache in a background thread. (Scott Chen via zshao)
      Show
      MAPREDUCE-1568 . TrackerDistributedCacheManager should clean up cache in a background thread. (Scott Chen via zshao)

      Description

      Right now the TrackerDistributedCacheManager do the clean up with the following code path:

      TaskRunner.run() -> 
      TrackerDistributedCacheManager.setup() ->
      TrackerDistributedCacheManager.getLocalCache() -> 
      TrackerDistributedCacheManager.deleteCache()
      

      The deletion of the cache files can take a long time and it should not be done by a task. We suggest that there should be a separate thread checking and clean up the cache files.

        Attachments

        1. MAPREDUCE-1568-v3.1.txt
          17 kB
          Scott Chen
        2. MAPREDUCE-1568-v3.txt
          17 kB
          Scott Chen
        3. MAPREDUCE-1568-v2.1.txt
          16 kB
          Scott Chen
        4. MAPREDUCE-1568-v2.txt
          15 kB
          Scott Chen
        5. MAPREDUCE-1568.txt
          6 kB
          Scott Chen

          Issue Links

            Activity

              People

              • Assignee:
                schen Scott Chen
                Reporter:
                schen Scott Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: