Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1129

Assign multiple Map and Reduce tasks in Fairscheduler

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.20.1
    • Fix Version/s: None
    • Component/s: contrib/fair-share
    • Labels:
      None

      Description

      In Hadoop-0.20, the period of heartbeat becomes much longer.
      Fairscheduler assigns at most one Map and one Reduce task per heartbeat.
      This makes the cluster become very inefficient. Often time only half of the slots are used.

      One idea is that we make Fairscheduler detect this situation (cluster under used) and scheduler more tasks in a heartbeat.
      Thoughts?

        Activity

        Hide
        Matei Zaharia added a comment -

        Todd, are you guys also doing a backport for vanilla (Apache) Hadoop 0.20?

        Show
        Matei Zaharia added a comment - Todd, are you guys also doing a backport for vanilla (Apache) Hadoop 0.20?
        Hide
        Todd Lipcon added a comment -

        Resolving as duplicate since this is in MAPREDUCE-706. Regarding the backport for Cloudera's distro, we will definitely include this in 0.20 in CDH3, and it's looking like we may backport for CDH2 in the coming weeks as well.

        Show
        Todd Lipcon added a comment - Resolving as duplicate since this is in MAPREDUCE-706 . Regarding the backport for Cloudera's distro, we will definitely include this in 0.20 in CDH3, and it's looking like we may backport for CDH2 in the coming weeks as well.
        Hide
        Aaron Kimball added a comment -

        Matei: On our roadmap. It's a tricky backport, so it's taking a while to test and bake in.

        Show
        Aaron Kimball added a comment - Matei: On our roadmap. It's a tricky backport, so it's taking a while to test and bake in.
        Hide
        Matei Zaharia added a comment -

        This feature is implemented as part of MAPREDUCE-706. I think some Cloudera folks were porting this to Hadoop 0.20, but I'm not sure what the status is.

        Show
        Matei Zaharia added a comment - This feature is implemented as part of MAPREDUCE-706 . I think some Cloudera folks were porting this to Hadoop 0.20, but I'm not sure what the status is.
        Hide
        Zheng Shao added a comment -

        One simple idea is to change the interface of LoadManager:

        From:

        [code}
        boolean LoadManager.canAssignMap(...)
        boolean LoadManager.canAssignReduce(...)

        
        To:
        
        [code}
        int LoadManager.canAssignMapCount(...)
        int LoadManager.canAssignReduceCount(...)
        

        The LoadManager returns the number of map(or reduce) tasks that can be assigned to this TaskTracker, so we can assign multiple tasks in a single heartbeat.
        The default implementation of the new functions can directly call the old functions to return 0 or 1, to provide backward compatibility with user-defined LoadManagers.

        Show
        Zheng Shao added a comment - One simple idea is to change the interface of LoadManager: From: [code} boolean LoadManager.canAssignMap(...) boolean LoadManager.canAssignReduce(...) To: [code} int LoadManager.canAssignMapCount(...) int LoadManager.canAssignReduceCount(...) The LoadManager returns the number of map(or reduce) tasks that can be assigned to this TaskTracker, so we can assign multiple tasks in a single heartbeat. The default implementation of the new functions can directly call the old functions to return 0 or 1, to provide backward compatibility with user-defined LoadManagers.
        Hide
        Allen Wittenauer added a comment -

        We're seeing exactly the same issue.

        Show
        Allen Wittenauer added a comment - We're seeing exactly the same issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Scott Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development