Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7017

Too many times of meaningless invocation in TaskAttemptImpl#resolveHosts

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha4
    • Fix Version/s: None
    • Component/s: mr-am
    • Labels:
      None

      Description

      MRAppMaster uses TaskAttemptImpl::resolveHosts to determine the dataLocalHosts for each task when the location of data split is IP, which will call a lot of times ( taskNum * dfsReplication) of function InetAddress::getByName and most of the funcition calls are redundant. When the job has a great number of tasks and the speed of DNS resolution is not fast enough, it will take a lot of time at this stage before the job running.

        Attachments

        1. MAPREDUCE-7017.001.patch
          3 kB
          jiayuhan-it

          Activity

            People

            • Assignee:
              jiayuhan-it jiayuhan-it
              Reporter:
              MagicBunny jiayuhan-it
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: