Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2790

TaskInProgress.hasSpeculativeTask is very inefficient

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.17.0
    • None
    • None

    Description

      Each call to JobInProgress.findNewTask can call TaskInProgress.hasSpeculativeTask once per a task. Each call to hasSpeculativeTask calls System.getCurrentTimeMillis, which can result in hundreds of thousands of calls to getCurrentTimeMillis. Additionally, it calls TaskInProgress.isOnlyCommitPending, which calls .values() on the map from task id to host name and iterates through them to see if any of the tasks are in commit pending. It would be better to have a commit pending boolean flag in the TaskInProgress. It also looks like there are other opportunities here, but those jumped out at me. We should also look at this method in the profiler.

      Attachments

        1. 2790.patch
          6 kB
          Owen O'Malley
        2. 2790-2.patch
          6 kB
          Owen O'Malley

        Issue Links

          Activity

            People

              omalley Owen O'Malley
              omalley Owen O'Malley
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: