Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1533

Reduce or remove usage of String.format() usage in CapacityTaskScheduler.updateQSIObjects and Counters.makeEscapedString()

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.20.1
    • 0.22.0
    • jobtracker
    • None
    • Reviewed
    • Incremental enhancements to the JobTracker to optimize heartbeat handling.

    Description

      When short jobs are executed in hadoop with OutOfBandHeardBeat=true, JT executes heartBeat() method heavily. This internally makes a call to CapacityTaskScheduler.updateQSIObjects().

      CapacityTaskScheduler.updateQSIObjects(), internally calls String.format() for setting the job scheduling information. Based on the datastructure size of "jobQueuesManager" and "queueInfoMap", the number of times String.format() gets executed becomes very high. String.format() internally does pattern matching which turns to be out very heavy (This was revealed while profiling JT. Almost 57% of time was spent in CapacityScheduler.assignTasks(), out of which String.format() took 46%.

      Would it be possible to do String.format() only at the time of invoking JobInProgress.getSchedulingInfo?. This might reduce the pressure on JT while processing heartbeats.

      Attachments

        1. mapreduce-1533-v1.8.patch
          27 kB
          Amar Kamat
        2. mapreduce-1533-v1.4.patch
          15 kB
          Amar Kamat
        3. MAPREDUCE-1533-and-others-20100413.bugfix.txt
          2 kB
          Arun Murthy
        4. mapreduce-1533--2010-05-24.patch
          39 kB
          Dick King
        5. mapreduce-1533--2010-05-21a.patch
          39 kB
          Dick King
        6. mapreduce-1533--2010-05-21.patch
          40 kB
          Dick King
        7. mapreduce-1533--2010-05-10a.patch
          36 kB
          Dick King
        8. ASF.LICENSE.NOT.GRANTED--MAPREDUCE-1533-and-others-20100413.1.txt
          30 kB
          Vinod Kumar Vavilapalli

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dking Dick King Assign to me
            rajesh.balamohan Rajesh Balamohan
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment