Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4893

MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Say the MR AppMaster asks the RM for 3 containers on nodes n1, n2 and n3. There are 10 node n1-n10 in the same rack. The RM can give it allocated containers in the list order n5, n2, n1. The way AM map->container assignment happens, the AM will try to assign node local maps to n5, failing which it will assign rack local maps to n5. These rack local maps could be node local on n2 and n1 and would have been assigned to containers on n1 and n2 if the AM had not made an early rack local match for them on n5. This can lead to poor locality.

      1. MAPREDUCE-4893.3.patch
        19 kB
        Bikas Saha
      2. MAPREDUCE-4893.2.patch
        19 kB
        Bikas Saha
      3. MAPREDUCE-4893.1.patch
        19 kB
        Bikas Saha

        Issue Links

          Activity

          Jason Lowe made changes -
          Link This issue is related to MAPREDUCE-5622 [ MAPREDUCE-5622 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Jason Lowe made changes -
          Link This issue is related to MAPREDUCE-5002 [ MAPREDUCE-5002 ]
          Jason Lowe made changes -
          Link This issue is duplicated by MAPREDUCE-4982 [ MAPREDUCE-4982 ]
          Jason Lowe made changes -
          Fix Version/s 0.23.7 [ 12323954 ]
          Vinod Kumar Vavilapalli made changes -
          Component/s applicationmaster [ 12315625 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.0.3-alpha [ 12323275 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Resolution Fixed [ 1 ]
          Bikas Saha made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Bikas Saha made changes -
          Attachment MAPREDUCE-4893.3.patch [ 12567104 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Bikas Saha made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Bikas Saha made changes -
          Attachment MAPREDUCE-4893.2.patch [ 12564478 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Bikas Saha made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Bikas Saha made changes -
          Attachment MAPREDUCE-4893.1.patch [ 12561803 ]
          Bikas Saha made changes -
          Attachment MAPREDUCE-4893.1.patch [ 12561811 ]
          Bikas Saha made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Bikas Saha made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Bikas Saha made changes -
          Field Original Value New Value
          Attachment MAPREDUCE-4893.1.patch [ 12561803 ]
          Bikas Saha created issue -

            People

            • Assignee:
              Bikas Saha
              Reporter:
              Bikas Saha
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development