Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1533

Reduce or remove usage of String.format() usage in CapacityTaskScheduler.updateQSIObjects and Counters.makeEscapedString()

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.22.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Incremental enhancements to the JobTracker to optimize heartbeat handling.

      Description

      When short jobs are executed in hadoop with OutOfBandHeardBeat=true, JT executes heartBeat() method heavily. This internally makes a call to CapacityTaskScheduler.updateQSIObjects().

      CapacityTaskScheduler.updateQSIObjects(), internally calls String.format() for setting the job scheduling information. Based on the datastructure size of "jobQueuesManager" and "queueInfoMap", the number of times String.format() gets executed becomes very high. String.format() internally does pattern matching which turns to be out very heavy (This was revealed while profiling JT. Almost 57% of time was spent in CapacityScheduler.assignTasks(), out of which String.format() took 46%.

      Would it be possible to do String.format() only at the time of invoking JobInProgress.getSchedulingInfo?. This might reduce the pressure on JT while processing heartbeats.

      1. mapreduce-1533--2010-05-24.patch
        39 kB
        Dick King
      2. mapreduce-1533--2010-05-21a.patch
        39 kB
        Dick King
      3. mapreduce-1533--2010-05-21.patch
        40 kB
        Dick King
      4. mapreduce-1533--2010-05-10a.patch
        36 kB
        Dick King
      5. mapreduce-1533-v1.8.patch
        27 kB
        Amar Kamat
      6. MAPREDUCE-1533-and-others-20100413.bugfix.txt
        2 kB
        Arun C Murthy
      7. ASF.LICENSE.NOT.GRANTED--MAPREDUCE-1533-and-others-20100413.1.txt
        30 kB
        Vinod Kumar Vavilapalli
      8. mapreduce-1533-v1.4.patch
        15 kB
        Amar Kamat

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Dick King
              Reporter:
              Rajesh Balamohan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development