19/04/23 15:59:45 WARN ipc.Client: Address change detected. Old: yarnrm1.grappler.eu-west-1.sandbox.a ws.skyscanner.local/10.41.104.172:8031 New: yarnrm1.grappler.eu-west-1.sandbox.aws.skyscanner.local/1 0.41.104.77:8031 19/04/23 15:59:45 DEBUG ipc.Client: closing ipc connection to yarnrm1.grappler.eu-west-1.sandbox.aws.skyscanner.local/10.41.104.77:8031: No route to host java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:609) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:707) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1446) at org.apache.hadoop.ipc.Client.call(Client.java:1407) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.nodeHeartbeat(Unknown Source) at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.nodeHeartbeat(ResourceTrackerPBClientImpl.java:80) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.nodeHeartbeat(Unknown Source) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:596) at java.lang.Thread.run(Thread.java:745) 19/04/23 15:59:45 DEBUG ipc.Client: IPC Client (330513933) connection to yarnrm1.grappler.eu-west-1.sandbox.aws.skyscanner.local/10.41.104.172:8031 from ec2-user: closed 19/04/23 15:59:45 INFO retry.RetryInvocationHandler: Exception while invoking nodeHeartbeat of class ResourceTrackerPBClientImpl over rm1 after 3 fail over attempts. Trying to fail over after sleeping for 29109ms. java.net.NoRouteToHostException: No Route to Host from ip-10-41-97-250/10.41.97.250 to yarnrm1.grappler.eu-west-1.sandbox.aws.skyscanner.local:8031 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see: http://wiki.apache.org/hadoop/NoRouteToHost at sun.reflect.GeneratedConstructorAccessor14.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:758) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1407) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.nodeHeartbeat(Unknown Source) at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.nodeHeartbeat(ResourceTrackerPBClientImpl.java:80) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.nodeHeartbeat(Unknown Source) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:596) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:609) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:707) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1446) ... 12 more 19/04/23 15:59:46 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 15:59:48 DEBUG ipc.Server: IPC Server idle connection scanner for port 8040: task running 19/04/23 15:59:48 DEBUG ipc.Server: IPC Server idle connection scanner for port 45400: task running 19/04/23 15:59:49 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 15:59:52 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 15:59:55 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 15:59:58 DEBUG ipc.Server: IPC Server idle connection scanner for port 8040: task running 19/04/23 15:59:58 DEBUG ipc.Server: IPC Server idle connection scanner for port 45400: task running 19/04/23 15:59:58 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 16:00:01 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 16:00:04 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 16:00:07 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 16:00:08 DEBUG ipc.Server: IPC Server idle connection scanner for port 8040: task running 19/04/23 16:00:08 DEBUG ipc.Server: IPC Server idle connection scanner for port 45400: task running 19/04/23 16:00:10 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 16:00:13 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 16:00:15 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 19/04/23 16:00:15 DEBUG ipc.Client: The ping interval is 60000 ms. 19/04/23 16:00:15 DEBUG ipc.Client: Connecting to yarnrm2.grappler.eu-west-1.sandbox.aws.skyscanner.local/10.41.162.2:8031 19/04/23 16:00:16 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 16:00:18 DEBUG ipc.Server: IPC Server idle connection scanner for port 8040: task running 19/04/23 16:00:18 DEBUG ipc.Server: IPC Server idle connection scanner for port 45400: task running 19/04/23 16:00:19 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 16:00:22 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 16:00:25 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 16:00:28 DEBUG ipc.Server: IPC Server idle connection scanner for port 8040: task running 19/04/23 16:00:28 DEBUG ipc.Server: IPC Server idle connection scanner for port 45400: task running 19/04/23 16:00:28 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 16:00:31 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ] 19/04/23 16:00:34 DEBUG monitor.ContainersMonitorImpl: Current ProcessTree list : [ ]