Hadoop Common
  1. Hadoop Common
  2. HADOOP-4943

fair share scheduler does not utilize all slots if the task trackers are configured heterogeneously

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.19.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      HADOOP-4943: Fixed fair share scheduler to utilize all slots when the task trackers are configured heterogeneously.
      Show
      HADOOP-4943 : Fixed fair share scheduler to utilize all slots when the task trackers are configured heterogeneously.

      Description

      There is some code in the fairshare scheduler that tries to make the load across the whole cluster the same.
      That piece of code will break if the task trackers are configured differently. Basically, we will stop assigning more tasks to tasks trackers that have tasks above the cluster average, but we may still want to do that because other task trackers may have less slots.

      We should change the code to maintain a cluster-wide slot usage percentage (instead of absolute number of slot usage) to make sure the load is evenly distributed.

      1. HADOOP-4943-1.patch
        5 kB
        Zheng Shao
      2. hadoop-4943-2.patch
        6 kB
        Matei Zaharia

        Activity

        Zheng Shao created issue -
        Zheng Shao made changes -
        Field Original Value New Value
        Attachment HADOOP-4943-1.patch [ 12396758 ]
        Zheng Shao made changes -
        Affects Version/s 0.19.0 [ 12313211 ]
        Status Open [ 1 ] Patch Available [ 10002 ]
        dhruba borthakur made changes -
        Fix Version/s 0.19.1 [ 12313473 ]
        Affects Version/s 0.19.0 [ 12313211 ]
        Matei Zaharia made changes -
        Attachment hadoop-4943-2.patch [ 12397664 ]
        Zheng Shao made changes -
        Fix Version/s 0.21.0 [ 12313563 ]
        Release Note HADOOP-4943: Fixed fair share scheduler to utilize all slots when the task trackers are configured heterogeneously.
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.20.0 [ 12313438 ]
        Resolution Fixed [ 1 ]
        Nigel Daley made changes -
        Fix Version/s 0.20.0 [ 12313438 ]
        Fix Version/s 0.21.0 [ 12313563 ]
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s contrib/fair-share [ 12312456 ]

          People

          • Assignee:
            Zheng Shao
            Reporter:
            Zheng Shao
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development