Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7017

Too many times of meaningless invocation in TaskAttemptImpl#resolveHosts

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha4
    • Fix Version/s: None
    • Component/s: mr-am
    • Labels:
      None

      Description

      MRAppMaster uses TaskAttemptImpl::resolveHosts to determine the dataLocalHosts for each task when the location of data split is IP, which will call a lot of times ( taskNum * dfsReplication) of function InetAddress::getByName and most of the funcition calls are redundant. When the job has a great number of tasks and the speed of DNS resolution is not fast enough, it will take a lot of time at this stage before the job running.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jiayuhan-it jiayuhan-it
                Reporter:
                MagicBunny jiayuhan-it
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: