Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1098

Incorrect synchronization in DistributedCache causes TaskTrackers to freeze up during localization of Cache for tasks.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: tasktracker
    • Labels:
      None
    • Release Note:
      Fixed the distributed cache's localizeCache to lock only the uri it is localizing.

      Description

      Currently org.apache.hadoop.filecache.DistributedCache.getLocalCache(URI, Configuration, Path, FileStatus, boolean, long, Path, boolean) allows only one TaskRunner thread in TT to localize DistributedCache across jobs. Current way of synchronization is across baseDir this has to be changed to lock on the same baseDir.

      1. MAPREDUCE-1098.patch
        16 kB
        Arun C Murthy
      2. MAPREDUCE-1098.patch
        5 kB
        Arun C Murthy
      3. MAPREDUCE-1098.patch
        5 kB
        Arun C Murthy
      4. patch-1098.txt
        1.0 kB
        Amareshwari Sriramadasu
      5. patch-1098-0.20.txt
        3 kB
        Amareshwari Sriramadasu
      6. patch-1098-1.txt
        4 kB
        Amareshwari Sriramadasu
      7. patch-1098-2.txt
        4 kB
        Amareshwari Sriramadasu
      8. patch-1098-3.txt
        29 kB
        Amareshwari Sriramadasu
      9. patch-1098-4.txt
        31 kB
        Amareshwari Sriramadasu
      10. patch-1098-5.txt
        31 kB
        Amareshwari Sriramadasu
      11. patch-1098-6.txt
        31 kB
        Amareshwari Sriramadasu
      12. patch-1098-7.txt
        32 kB
        Arun C Murthy
      13. patch-1098-7.txt
        31 kB
        Amareshwari Sriramadasu
      14. patch-1098-7-y20.txt
        32 kB
        Hemanth Yamijala
      15. patch-1098-ydist.txt
        26 kB
        Arun C Murthy
      16. patch-1098-ydist.txt
        25 kB
        Amareshwari Sriramadasu
      17. patch-1098-ydist.txt
        4 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Sreekanth Ramakrishnan created issue -
          Amareshwari Sriramadasu made changes -
          Field Original Value New Value
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-6249 MAPREDUCE-1098
          Affects Version/s 0.20.0 [ 12313438 ]
          Component/s tasktracker [ 12312906 ]
          Component/s filecache [ 12312386 ]
          Amareshwari Sriramadasu made changes -
          Assignee Amareshwari Sriramadasu [ amareshwari ]
          Fix Version/s 0.21.0 [ 12314045 ]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1098.txt [ 12421946 ]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1098-1.txt [ 12422215 ]
          Amareshwari Sriramadasu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1098-0.20.txt [ 12422323 ]
          Amareshwari Sriramadasu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1098-2.txt [ 12422535 ]
          Amareshwari Sriramadasu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1098-ydist.txt [ 12422874 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Arun C Murthy made changes -
          Attachment MAPREDUCE-1098.patch [ 12422888 ]
          Arun C Murthy made changes -
          Attachment MAPREDUCE-1098.patch [ 12422988 ]
          Arun C Murthy made changes -
          Attachment MAPREDUCE-1098.patch [ 12422990 ]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1098-3.txt [ 12423017 ]
          Amareshwari Sriramadasu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1098-4.txt [ 12423098 ]
          Amareshwari Sriramadasu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amareshwari Sriramadasu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1098-5.txt [ 12423109 ]
          Amareshwari Sriramadasu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amareshwari Sriramadasu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1098-6.txt [ 12423115 ]
          Amareshwari Sriramadasu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amareshwari Sriramadasu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1098-ydist.txt [ 12423125 ]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1098-7.txt [ 12423126 ]
          Amareshwari Sriramadasu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Arun C Murthy made changes -
          Attachment patch-1098-7.txt [ 12423135 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Attachment patch-1098-ydist.txt [ 12423137 ]
          Amareshwari Sriramadasu made changes -
          Release Note Fixed the distributed cache's localizeCache to lock only the uri it is localizing.
          Amareshwari Sriramadasu made changes -
          Fix Version/s 0.22.0 [ 12314184 ]
          Fix Version/s 0.21.0 [ 12314045 ]
          Zheng Shao made changes -
          Link This issue relates to MAPREDUCE-1186 [ MAPREDUCE-1186 ]
          Hemanth Yamijala made changes -
          Attachment patch-1098-7-y20.txt [ 12431207 ]
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314045 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Sreekanth Ramakrishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development