Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2026

JobTracker.getJobCounters() should not hold JobTracker lock while calling JobInProgress.getCounters()

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      JobTracker.getJobCounter() will lock JobTracker and call JobInProgress.getCounters().
      JobInProgress.getCounters() can be very expensive because it aggregates all the task counters.
      We found that from the JobTracker jstacks that this method is one of the bottleneck of the JobTracker performance.

      JobInProgress.getCounters() should be able to be called out side the JobTracker lock because it already has JobInProgress lock.
      For example, it is used by jobdetails.jsp without a JobTracker lock.

        Attachments

        1. MAPREDUCE-2026.txt
          1 kB
          Scott Chen
        2. 2026.1.patch
          3 kB
          Joydeep Sen Sarma

          Issue Links

            Activity

              People

              • Assignee:
                jsensarma Joydeep Sen Sarma
                Reporter:
                schen Scott Chen
              • Votes:
                1 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: