Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2026

JobTracker.getJobCounters() should not hold JobTracker lock while calling JobInProgress.getCounters()

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      JobTracker.getJobCounter() will lock JobTracker and call JobInProgress.getCounters().
      JobInProgress.getCounters() can be very expensive because it aggregates all the task counters.
      We found that from the JobTracker jstacks that this method is one of the bottleneck of the JobTracker performance.

      JobInProgress.getCounters() should be able to be called out side the JobTracker lock because it already has JobInProgress lock.
      For example, it is used by jobdetails.jsp without a JobTracker lock.

      1. MAPREDUCE-2026.txt
        1 kB
        Scott Chen
      2. 2026.1.patch
        3 kB
        Joydeep Sen Sarma

        Issue Links

          Activity

          Scott Chen created issue -
          Scott Chen made changes -
          Field Original Value New Value
          Attachment MAPREDUCE-2026.txt [ 12452699 ]
          Scott Chen made changes -
          Attachment MAPREDUCE-2026.txt [ 12452699 ]
          Scott Chen made changes -
          Attachment MAPREDUCE-2026.txt [ 12452700 ]
          Scott Chen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Scott Chen made changes -
          Assignee Scott Chen [ schen ] Joydeep Sen Sarma [ jsensarma ]
          Joydeep Sen Sarma made changes -
          Attachment 2026.1.patch [ 12456941 ]
          Joydeep Sen Sarma made changes -
          Link This issue duplicates MAPREDUCE-2114 [ MAPREDUCE-2114 ]
          Nigel Daley made changes -
          Fix Version/s 0.22.0 [ 12314184 ]
          Scott Chen made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.22.0 [ 12314184 ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Joydeep Sen Sarma
              Reporter:
              Scott Chen
            • Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development