Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2734

DistCp with FairScheduler assign all map tasks in one TT

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 2.3.0
    • Component/s: jobtracker
    • Labels:
      None

      Description

      1 Using FairScheduler.
      2 distcp a directory contains 5 files.

      The DistCp job will launch 5 map tasks, and FairScheduler assign them to one TaskTracker.
      It should be assigned different, like the original JobQueueScheduler.

        Activity

        Hide
        Bochun Bai added a comment -

        I will check how the HEAD version works.

        Show
        Bochun Bai added a comment - I will check how the HEAD version works.
        Hide
        Bochun Bai added a comment -

        Turning mapred.fairscheduler.assignmultiple=false solves the problem.
        There is no component like contrib/fair-scheduler. I move this issues to the jobtracker.

        Considering two possible improvement:
        1. document this parameter in mapreduce/src/docs/src/documentation/content/xdocs/fair_scheduler.xml
        2. change the default value to false.

        Show
        Bochun Bai added a comment - Turning mapred.fairscheduler.assignmultiple=false solves the problem. There is no component like contrib/fair-scheduler. I move this issues to the jobtracker. Considering two possible improvement: 1. document this parameter in mapreduce/src/docs/src/documentation/content/xdocs/fair_scheduler.xml 2. change the default value to false.
        Hide
        Bochun Bai added a comment -

        I can find this config is default to false in 2.3.0. It is fixed in YARN-302.

        Show
        Bochun Bai added a comment - I can find this config is default to false in 2.3.0. It is fixed in YARN-302 .

          People

          • Assignee:
            Bochun Bai
            Reporter:
            Bochun Bai
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development