Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1213

TaskTrackers restart is very slow because it deletes distributed cache directory synchronously

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.20.1
    • 0.21.0
    • None
    • None
    • Incompatible change, Reviewed
    • Directories specified in mapred.local.dir that can not be created now cause the TaskTracker to fail to start.

    Description

      We are seeing that when we restart a tasktracker, it tries to recursively delete all the file in the distributed cache. It invoked FileUtil.fullyDelete() which is very very slow. This means that the TaskTracker cannot join the cluster for an extended period of time (upto 2 hours for us). The problem is acute if the number of files in a distributed cache is a few-thousands.

      Attachments

        1. MAPREDUCE-1213.1.patch
          14 kB
          Zheng Shao
        2. MAPREDUCE-1213.2.patch
          14 kB
          Zheng Shao
        3. MAPREDUCE-1213.3.patch
          14 kB
          Zheng Shao
        4. MAPREDUCE-1213.4.patch
          14 kB
          Zheng Shao
        5. MAPREDUCE-1213.branch-0.20.patch
          16 kB
          Zheng Shao
        6. MAPREDUCE-1213.branch-0.20.2.patch
          14 kB
          Zheng Shao

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-1303 Merge org.apache.hadoop.mapred.CleanupQueue with MRAsyncDiskService

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-1302 TrackerDistributedCacheManager can delete file asynchronously

          • Major - Major loss of function.
          • Closed
          breaks

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-4481 User Log Retention across TT restarts

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Reopened
          is blocked by

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-6433 Add AsyncDiskService that is used in both hdfs and mapreduce

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-611 Heartbeats times from Datanodes increase when there are plenty of blocks to delete

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-2049 JT and TT should prune invalid local dirs on startup

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-1382 MRAsyncDiscService should tolerate missing local.dir

          • Major - Major loss of function.
          • Closed

          Activity

            People

              zshao Zheng Shao
              dhruba Dhruba Borthakur
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: