Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6224

resolve the hosts in DNSToSwitchMapping before inter tracker server start to avoid IPC timeout in Task Tracker heartbeat

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: mrv1
    • Labels:

      Description

      Resolve the hosts to fill up the cache in CachedDNSToSwitchMapping before inter tracker server start to avoid IPC timeout in Task Tracker heartbeat.
      We saw IPC timeout happen in Task Tracker heartbeat for a large MR1 cluster which use topology script(ShellCommandExecutor) to resolve the Network Topology for Task Tracker host in ScriptBasedMapping.
      The reason is
      Right after inter tracker server start in Job Tracker, Job Tracker receive a lots HeartBeat from the Task Tracker.
      heartbeat function call resolveAndAddToTopology to resolve the Network Topology for Task Tracker host in ScriptBasedMapping which implement CachedDNSToSwitchMapping.
      ScriptBasedMapping#resolve will check whether the host is in the cache,
      If the host is not in the cache, it will run topology script to get the host's Network Topology using ShellCommandExecutor. Normally running topology script is time consuming, which may cause the IPC time if too many heartbeat happened at the same time for a large MR1 cluster.
      The solution is to resolve the Network Topology for all hosts in the hosts list from HostsFileReader before receive any heartbeat from Task Tracker, so the cache in ScriptBasedMapping will be filled up, and when heartbeat call resolveAndAddToTopology, it will get the result from the cache instead of running topology script.

        Attachments

          Activity

            People

            • Assignee:
              zxu zhihai xu
              Reporter:
              zxu zhihai xu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: