[YARN-3944] Connection refused to nodemanagers are retried at multiple levels

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: 2.6.0
• Assignee: Siqi Li
• Reporter: Siqi Li
• Created: 20/Jul/15 21:42
• Updated: 28/Sep/15 17:37
• Resolved: 28/Sep/15 17:33

Description

This is related to YARN-3238. When NM is down, ipc client will get ConnectException.

Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)

However, retry happens at two layers(ipc retry 40 times and serverProxy retrying 91 times), this could end up with ~1 hour retry interval.

Issue Links

• relates to: YARN-3238 Connection timeouts to nodemanagers are retried at multiple levels