Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1764

FairScheduler locality delay may put heavy pressure on Jobtracker

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.22.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      FairScheduler locality delay feature holds the scheduling of jobs until it gets good locality.
      This greatly improves the locality of the tasks. Reduce the cost of traffic.

      We have observed the following problem on FairScheduler locality delay:
      We have some machines have older data and some newly added machines do not have important data.
      When these machines send heartbeat, JT scans tasks to find jobs has the right locality.
      Often time, these machines will scan all of the tasks of all the jobs and do not get any tasks.
      Scanning all the tasks on the JT is very costly. This makes JT very slow.
      And these machines often time do not get scheduled. This hurts the cluster utilization.

      Any ideas?

        Attachments

          Activity

            People

            • Assignee:
              dms Dmytro Molkov
              Reporter:
              schen Scott Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: