Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2905

CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 1.1.0
    • Component/s: contrib/fair-share
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to spread the job across more nodes so that a relatively small job doesn't saturate any node in the cluster.

      Testing has shown that setting mapred.fairscheduler.assignmultiple to false gives the desired behavior for small jobs, but is unnecessary for large jobs. However, since this is a cluster-wide setting, we can't properly tune.

      It'd be nice if jobs can set a param similar to mapred.fairscheduler.assignmultiple on submission to better control the task distribution of a particular job.

      1. MR-2905.patch
        4 kB
        Jeff Bean
      2. MR-2905.patch.2
        5 kB
        Jeff Bean
      3. MR-2905.10-13-2011
        12 kB
        Jeff Bean
      4. ASF.LICENSE.NOT.GRANTED--screenshot-1.jpg
        58 kB
        Jeff Bean
      5. mr-2905.txt
        10 kB
        Todd Lipcon
      6. mr-2905.txt
        11 kB
        Todd Lipcon

        Activity

        Jeff Bean created issue -
        Jeff Bean made changes -
        Field Original Value New Value
        Description We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to round-robin spread the job across more nodes.

        It'd be nice developers can set a param similar to mapred.fairscheduler.assignmultiple on a per-job basis to better control the task allocation of a particular job.
        We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to spread the job across more nodes so that a relatively small job doesn't saturate any node in the cluster.

        It'd be nice developers can set a param similar to mapred.fairscheduler.assignmultiple on a per-job basis to better control the task allocation of a particular job.
        Jeff Bean made changes -
        Description We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to spread the job across more nodes so that a relatively small job doesn't saturate any node in the cluster.

        It'd be nice developers can set a param similar to mapred.fairscheduler.assignmultiple on a per-job basis to better control the task allocation of a particular job.
        We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to spread the job across more nodes so that a relatively small job doesn't saturate any node in the cluster.

        Testing has shown that setting mapred.fairscheduler.assignmultiple to false gives the desired behavior for small jobs, but is unnecessary for large jobs. However, since this is a cluster-wide setting, we can't properly tune.

        It'd be nice jobs can set a param similar to mapred.fairscheduler.assignmultiple on submission to better control the task distribution of a particular job.
        Jeff Bean made changes -
        Affects Version/s 0.20.2 [ 12314205 ]
        Jeff Bean made changes -
        Description We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to spread the job across more nodes so that a relatively small job doesn't saturate any node in the cluster.

        Testing has shown that setting mapred.fairscheduler.assignmultiple to false gives the desired behavior for small jobs, but is unnecessary for large jobs. However, since this is a cluster-wide setting, we can't properly tune.

        It'd be nice jobs can set a param similar to mapred.fairscheduler.assignmultiple on submission to better control the task distribution of a particular job.
        We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to spread the job across more nodes so that a relatively small job doesn't saturate any node in the cluster.

        Testing has shown that setting mapred.fairscheduler.assignmultiple to false gives the desired behavior for small jobs, but is unnecessary for large jobs. However, since this is a cluster-wide setting, we can't properly tune.

        It'd be nice if jobs can set a param similar to mapred.fairscheduler.assignmultiple on submission to better control the task distribution of a particular job.
        Jeff Bean made changes -
        Summary Allow mapred.fairscheduler.assignmultple to be set per job CapBasedLoadManager cannot access running tasks (was: assignmultiple per job)
        Jeff Bean made changes -
        Attachment MR-2905.patch [ 12497947 ]
        Jeff Bean made changes -
        Summary CapBasedLoadManager cannot access running tasks (was: assignmultiple per job) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
        Jeff Bean made changes -
        Attachment MR-2905.patch.2 [ 12498843 ]
        Jeff Bean made changes -
        Issue Type Improvement [ 4 ] Bug [ 1 ]
        Jeff Bean made changes -
        Attachment MR-2905.10-13-2011 [ 12498916 ]
        Jeff Bean made changes -
        Attachment screenshot-1.jpg [ 12498917 ]
        Todd Lipcon made changes -
        Attachment mr-2905.txt [ 12499497 ]
        Eli Collins made changes -
        Target Version/s 0.20.206.0 [ 12317960 ]
        Eli Collins made changes -
        Assignee Jeff Bean [ jwfbean ]
        Todd Lipcon made changes -
        Attachment mr-2905.txt [ 12503954 ]
        Todd Lipcon made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.20.206.0 [ 12317960 ]
        Resolution Fixed [ 1 ]
        Matt Foley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Jeff Bean
            Reporter:
            Jeff Bean
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development