Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2905

CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 1.1.0
    • Component/s: contrib/fair-share
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to spread the job across more nodes so that a relatively small job doesn't saturate any node in the cluster.

      Testing has shown that setting mapred.fairscheduler.assignmultiple to false gives the desired behavior for small jobs, but is unnecessary for large jobs. However, since this is a cluster-wide setting, we can't properly tune.

      It'd be nice if jobs can set a param similar to mapred.fairscheduler.assignmultiple on submission to better control the task distribution of a particular job.

        Attachments

        1. mr-2905.txt
          11 kB
          Todd Lipcon
        2. mr-2905.txt
          10 kB
          Todd Lipcon
        3. ASF.LICENSE.NOT.GRANTED--screenshot-1.jpg
          58 kB
          Jeff Bean
        4. MR-2905.10-13-2011
          12 kB
          Jeff Bean
        5. MR-2905.patch.2
          5 kB
          Jeff Bean
        6. MR-2905.patch
          4 kB
          Jeff Bean

          Activity

            People

            • Assignee:
              jwfbean Jeff Bean
              Reporter:
              jwfbean Jeff Bean
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: