Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1995

FairScheduler can wait for JobInProgress lock while holding JobTracker lock

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.22.0
    • Component/s: contrib/fair-share
    • Labels:
      None

      Description

      Here is the related code path.

      JobInProgress.getTaskInProgress() (Locks JobInProgress)
      LocalityLevel.fromTask()
      FairScheduler.updateLastMapLocalityLevel()
      FairScheduler.assignTasks()
      JobTracker.heartbeat() (Locks JobTracker)
      

        Activity

        Hide
        Scott Chen added a comment -

        After looking at it more carefully, I think this may not be a problem.
        Because when FairScheduler calls updateLastMapLocalityLevel(), the JobInProgress is already initialized.
        JobInProgess.initTasks() is what make things slow.

        Sorry for the spam.

        Show
        Scott Chen added a comment - After looking at it more carefully, I think this may not be a problem. Because when FairScheduler calls updateLastMapLocalityLevel(), the JobInProgress is already initialized. JobInProgess.initTasks() is what make things slow. Sorry for the spam.

          People

          • Assignee:
            Scott Chen
            Reporter:
            Scott Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development