[YARN-3644] Node manager shuts down if unable to connect with RM

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: nodemanager
• Assignee: Raju Bairishetti
• Reporter: Srikanth Sundarrajan
• Created: 14/May/15 05:55
• Updated: 13/Jul/15 18:16

Description

When NM is unable to connect to RM, NM shuts itself down.

          } catch (ConnectException e) {
            //catch and throw the exception if tried MAX wait time to connect RM
            dispatcher.getEventHandler().handle(
                new NodeManagerEvent(NodeManagerEventType.SHUTDOWN));
            throw new YarnRuntimeException(e);

In large clusters, if RM is down for maintenance for longer period, all the NMs shuts themselves down, requiring additional work to bring up the NMs.

Setting the yarn.resourcemanager.connect.wait-ms to -1 has other side effects, where non connection failures are being retried infinitely by all YarnClients (via RMProxy).

Issue Links

• is related to: YARN-196 Nodemanager should be more robust in handling connection failure to ResourceManager when a cluster is started
• is related to: YARN-479 NM retry behavior for connection to RM should be similar for lost heartbeats
• is related to: YARN-3668 Long run service shouldn't be killed even if Yarn crashed
• relates to: YARN-3847 Refactor TestNodeStatusUpdater
• relates to: YARN-3646 Applications are getting stuck some times in case of retry policy forever