Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1062

MRAppMaster take a long time to init taskAttempt

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.23.6
    • Fix Version/s: None
    • Component/s: applications
    • Labels:
      None

      Description

      In our cluster, MRAppMaster take a long time to init taskAttempt, the following log last one minute,

      2013-07-17 11:28:06,328 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved r01f11012.yh.aliyun.com to /r01f11
      2013-07-17 11:28:06,357 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved r01f11004.yh.aliyun.com to /r01f11
      2013-07-17 11:28:06,383 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved r03b05042.yh.aliyun.com to /r03b05
      2013-07-17 11:28:06,384 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1373523419753_4543_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
      2013-07-17 11:28:06,415 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved r03b02006.yh.aliyun.com to /r03b02
      2013-07-17 11:28:06,436 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved r02f02045.yh.aliyun.com to /r02f02
      2013-07-17 11:28:06,457 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved r02f02034.yh.aliyun.com to /r02f02
      2013-07-17 11:28:06,457 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1373523419753_4543_m_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED

      The reason is: resolved one host to rack almost take 25ms (We resolve the host to rack by a python script). Our hdfs cluster is more than 4000 datanodes, then a large input job will take a long time to init TaskAttempt.

      Is there any good idea to solve this problem.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                shenhong shenh062326
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: