Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1098

Incorrect synchronization in DistributedCache causes TaskTrackers to freeze up during localization of Cache for tasks.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.21.0
    • tasktracker
    • None
    • Fixed the distributed cache's localizeCache to lock only the uri it is localizing.

    Description

      Currently org.apache.hadoop.filecache.DistributedCache.getLocalCache(URI, Configuration, Path, FileStatus, boolean, long, Path, boolean) allows only one TaskRunner thread in TT to localize DistributedCache across jobs. Current way of synchronization is across baseDir this has to be changed to lock on the same baseDir.

      Attachments

        1. patch-1098.txt
          1.0 kB
          Amareshwari Sriramadasu
        2. patch-1098-1.txt
          4 kB
          Amareshwari Sriramadasu
        3. patch-1098-0.20.txt
          3 kB
          Amareshwari Sriramadasu
        4. patch-1098-2.txt
          4 kB
          Amareshwari Sriramadasu
        5. patch-1098-ydist.txt
          4 kB
          Amareshwari Sriramadasu
        6. MAPREDUCE-1098.patch
          5 kB
          Arun Murthy
        7. MAPREDUCE-1098.patch
          5 kB
          Arun Murthy
        8. MAPREDUCE-1098.patch
          16 kB
          Arun Murthy
        9. patch-1098-3.txt
          29 kB
          Amareshwari Sriramadasu
        10. patch-1098-4.txt
          31 kB
          Amareshwari Sriramadasu
        11. patch-1098-5.txt
          31 kB
          Amareshwari Sriramadasu
        12. patch-1098-6.txt
          31 kB
          Amareshwari Sriramadasu
        13. patch-1098-ydist.txt
          25 kB
          Amareshwari Sriramadasu
        14. patch-1098-7.txt
          31 kB
          Amareshwari Sriramadasu
        15. patch-1098-7.txt
          32 kB
          Arun Murthy
        16. patch-1098-ydist.txt
          26 kB
          Arun Murthy
        17. patch-1098-7-y20.txt
          32 kB
          Hemanth Yamijala

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-1186 While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir

          • Major - Major loss of function.
          • Closed

          Activity

            People

              amareshwari Amareshwari Sriramadasu
              sreekanth Sreekanth Ramakrishnan
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: