[TEZ-2738] ContainerLauncher tries to connect to unhealthy node for large number of times

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Target Version/s: 0.8.6
• Assignee: Unassigned
• Reporter: Rajesh Balamohan
• Created: 24/Aug/15 23:01
• Updated: 14/Mar/17 03:40

Description

Env: Ran a job with tez (built from master branch on aug-24).

One of the nodes went down in the middle of the run. And DAGAppMaster had a container launch in that node. After sometime, this node was declared as unhealthy. Even though the job lasted only for 7 minutes, DAGAppMaster was unresponsive after dag cleanup for > 1.5 hours. It kept on trying to connect to the unhealthy node. I will attach the logs in this JIRA.

ipc.client.connect.max.retries has been set to 50 in core-site.xml

 <property>
    <name>ipc.client.connect.max.retries</name>
    <value>50</value>
    <description>Defines the maximum number of retries for IPC connections.</description>
  </property>