Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1354

Incremental enhancements to the JobTracker for better scalability

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Incremental enhancements to the JobTracker include a no-lock version of JT.getTaskCompletion events, no lock on the JT while doing i/o during job-submission and several fixes to cut down configuration parsing during heartbeat-handling.

      Description

      It'd be nice to have the JobTracker object not be locked while accessing the HDFS for reading the jobconf file and while writing the jobinfo file in the submitJob method. We should see if we can avoid taking the lock altogether.

      1. MAPREDUCE-1354_yhadoop20.patch
        5 kB
        Arun C Murthy
      2. MAPREDUCE-1354_yhadoop20.patch
        15 kB
        Arun C Murthy
      3. MAPREDUCE-1354_yhadoop20.patch
        16 kB
        Arun C Murthy
      4. MAPREDUCE-1354_yhadoop20.patch
        19 kB
        Arun C Murthy
      5. MAPREDUCE-1354_yhadoop20.patch
        21 kB
        Arun C Murthy
      6. MAPREDUCE-1354_yhadoop20.patch
        23 kB
        Arun C Murthy
      7. MAPREDUCE-1354_yhadoop20.patch
        24 kB
        Arun C Murthy
      8. mr-1354-y20.patch
        23 kB
        Hemanth Yamijala
      9. mapreduce-1354--2010-03-10.patch
        26 kB
        Dick King
      10. mapreduce-1354--2010-05-13.patch
        26 kB
        Dick King

        Activity

        Devaraj Das created issue -
        Arun C Murthy made changes -
        Field Original Value New Value
        Issue Type Improvement [ 4 ] Bug [ 1 ]
        Assignee Arun C Murthy [ acmurthy ]
        Priority Major [ 3 ] Critical [ 2 ]
        Component/s jobtracker [ 12312907 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-1354_yhadoop20.patch [ 12435469 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-1354_yhadoop20.patch [ 12435683 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-1354_yhadoop20.patch [ 12435761 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-1354_yhadoop20.patch [ 12435925 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-1354_yhadoop20.patch [ 12435932 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-1354_yhadoop20.patch [ 12435947 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-1354_yhadoop20.patch [ 12436004 ]
        Arun C Murthy made changes -
        Summary Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS accesses Incremental enhancements to the JobTracker for better scalability
        Arun C Murthy made changes -
        Environment Incremental enhancements to the JobTracker include a no-lock version of JT.getTaskCompletion events, no lock on the JT while doing i/o during job-submission and several fixes to cut down configuration parsing during heartbeat-handling.
        Arun C Murthy made changes -
        Release Note Incremental enhancements to the JobTracker include a no-lock version of JT.getTaskCompletion events, no lock on the JT while doing i/o during job-submission and several fixes to cut down configuration parsing during heartbeat-handling.
        Environment Incremental enhancements to the JobTracker include a no-lock version of JT.getTaskCompletion events, no lock on the JT while doing i/o during job-submission and several fixes to cut down configuration parsing during heartbeat-handling.
        Hemanth Yamijala made changes -
        Attachment mr-1354-y20.patch [ 12437010 ]
        Dick King made changes -
        Attachment mapreduce-1354--2010-03-10.patch [ 12439506 ]
        Dick King made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Dick King made changes -
        Assignee Arun C Murthy [ acmurthy ] Dick King [ dking ]
        Dick King made changes -
        Attachment mapreduce-1354--2010-05-13.patch [ 12444432 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Hadoop Flags [Reviewed]
        Issue Type Bug [ 1 ] Improvement [ 4 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Vinod Kumar Vavilapalli made changes -
        Fix Version/s 0.22.0 [ 12314184 ]
        Konstantin Shvachko made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Dick King
            Reporter:
            Devaraj Das
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development