Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-315

Bias the decision of task scheduling (both for not-running and running) on node metrics (load, processing rate etc).



    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None


      HADOOP-2014 deals with a similar issue but at the rack level. This issue deals with the nodes on the same rack. Consider the following case

      node/host Runnable && ~Running TIPs host status
      H1 T1 good
      H2 T2 good
      H3 T3 good
      H4 T4,T5,T6 bad
      H5 - good
      rack1 T1,T2,T3,T4,T5,T6 good in all terms (runnable/numSlots) etc

      Now if H5 asks for a new task to run, it will be given T1. Ideally it should be given T4 because the processing rate of H1 is better than H4. Giving T1 will kill the locality of a better node. It makes more sense to kill the locality of a bad node instead. If H4 is overloaded then its better to select T4 since that might increase the chances of hitting the locality. With HADOOP-2014 the chances of TIPs from rack1 getting selected decreases since there might be other racks which are overloaded and the hosts having T4,T5,T6 (locally) are in better shape than H5. The point is select splits from nodes (rack local) that either have many local splits or might not be able to process them faster.

      Even for speculation, its better to speculate a TIP that might take longer than expected amongst those which can be speculated. Consider the following

      node/host Running TIPs num TIPs processed host status
      H1 T1 2 good
      H2 T2 1 bad

      Here it makes sense to speculate T2 since there is a reason to believe that T2 on H2 might become a long tail.


        Issue Links



              Unassigned Unassigned
              amar_kamat Amar Kamat
              0 Vote for this issue
              2 Start watching this issue