Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1129

Assign multiple Map and Reduce tasks in Fairscheduler

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.20.1
    • Fix Version/s: None
    • Component/s: contrib/fair-share
    • Labels:
      None

      Description

      In Hadoop-0.20, the period of heartbeat becomes much longer.
      Fairscheduler assigns at most one Map and one Reduce task per heartbeat.
      This makes the cluster become very inefficient. Often time only half of the slots are used.

      One idea is that we make Fairscheduler detect this situation (cluster under used) and scheduler more tasks in a heartbeat.
      Thoughts?

        Activity

        Hide
        aw Allen Wittenauer added a comment -

        We're seeing exactly the same issue.

        Show
        aw Allen Wittenauer added a comment - We're seeing exactly the same issue.
        Hide
        zshao Zheng Shao added a comment -

        One simple idea is to change the interface of LoadManager:

        From:

        [code}
        boolean LoadManager.canAssignMap(...)
        boolean LoadManager.canAssignReduce(...)

        To:
        
        [code}
        int LoadManager.canAssignMapCount(...)
        int LoadManager.canAssignReduceCount(...)
        

        The LoadManager returns the number of map(or reduce) tasks that can be assigned to this TaskTracker, so we can assign multiple tasks in a single heartbeat.
        The default implementation of the new functions can directly call the old functions to return 0 or 1, to provide backward compatibility with user-defined LoadManagers.

        Show
        zshao Zheng Shao added a comment - One simple idea is to change the interface of LoadManager: From: [code} boolean LoadManager.canAssignMap(...) boolean LoadManager.canAssignReduce(...) To: [code} int LoadManager.canAssignMapCount(...) int LoadManager.canAssignReduceCount(...) The LoadManager returns the number of map(or reduce) tasks that can be assigned to this TaskTracker, so we can assign multiple tasks in a single heartbeat. The default implementation of the new functions can directly call the old functions to return 0 or 1, to provide backward compatibility with user-defined LoadManagers.
        Hide
        matei Matei Zaharia added a comment -

        This feature is implemented as part of MAPREDUCE-706. I think some Cloudera folks were porting this to Hadoop 0.20, but I'm not sure what the status is.

        Show
        matei Matei Zaharia added a comment - This feature is implemented as part of MAPREDUCE-706 . I think some Cloudera folks were porting this to Hadoop 0.20, but I'm not sure what the status is.
        Hide
        kimballa Aaron Kimball added a comment -

        Matei: On our roadmap. It's a tricky backport, so it's taking a while to test and bake in.

        Show
        kimballa Aaron Kimball added a comment - Matei: On our roadmap. It's a tricky backport, so it's taking a while to test and bake in.
        Hide
        tlipcon Todd Lipcon added a comment -

        Resolving as duplicate since this is in MAPREDUCE-706. Regarding the backport for Cloudera's distro, we will definitely include this in 0.20 in CDH3, and it's looking like we may backport for CDH2 in the coming weeks as well.

        Show
        tlipcon Todd Lipcon added a comment - Resolving as duplicate since this is in MAPREDUCE-706 . Regarding the backport for Cloudera's distro, we will definitely include this in 0.20 in CDH3, and it's looking like we may backport for CDH2 in the coming weeks as well.
        Hide
        matei Matei Zaharia added a comment -

        Todd, are you guys also doing a backport for vanilla (Apache) Hadoop 0.20?

        Show
        matei Matei Zaharia added a comment - Todd, are you guys also doing a backport for vanilla (Apache) Hadoop 0.20?

          People

          • Assignee:
            Unassigned
            Reporter:
            schen Scott Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development