[YARN-9646] DistributedShell tests failed to bind to a local host name

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.4
• Fix Version/s: 3.3.0
• Component/s: test
• Hadoop Flags: Reviewed
• Assignee: Ray Yang
• Reporter: Ray Yang
• Created: 24/Jun/19 18:04
• Updated: 17/Jul/19 21:35
• Resolved: 17/Jul/19 00:39

Description

When running the integration test 

org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell#testDSShellWithoutDomain

at home

The following error happened:

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.BindException: Problem binding to [ruyang-mn3.linkedin.biz:0] java.net.BindException: Can't assign requested address; For more details see:  http://wiki.apache.org/hadoop/BindException

 

at org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:327)

at org.apache.hadoop.yarn.server.MiniYARNCluster.access$400(MiniYARNCluster.java:99)

at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:447)

at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)

at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)

at org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:278)

at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)

at org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.setupInternal(TestDistributedShell.java:91)

at org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.setup(TestDistributedShell.java:71)

…

Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.BindException: Problem binding to [ruyang-mn3.linkedin.biz:0] java.net.BindException: Can't assign requested address; For more details see:  http://wiki.apache.org/hadoop/BindException

at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:139)

at org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)

at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)

at org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.serviceStart(ResourceTrackerService.java:163)

at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)

at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)

at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:588)

at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)

at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:976)

at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1017)

at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1013)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)

at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1013)

at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1053)

at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)

at org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:319)

... 31 more

Caused by: java.net.BindException: Problem binding to [ruyang-mn3.linkedin.biz:0]java.net.BindException: Can't assign requested address; For more details see:  http://wiki.apache.org/hadoop/BindException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)

at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:721)

at org.apache.hadoop.ipc.Server.bind(Server.java:494)

at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:715)

at org.apache.hadoop.ipc.Server.<init>(Server.java:2464)

at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:996)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:539)

at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:514)

at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:840)

at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.createServer(RpcServerFactoryPBImpl.java:173)

at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132)

... 48 more

Caused by: java.net.BindException: Can't assign requested address

at sun.nio.ch.Net.bind0(Native Method)

at sun.nio.ch.Net.bind(Net.java:433)

at sun.nio.ch.Net.bind(Net.java:425)

at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)

at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)

at org.apache.hadoop.ipc.Server.bind(Server.java:477)

... 56 more

Process finished with exit code 255

 

It's because the test uses  

InetAddress.getLocalHost().getHostName();

to get the host name and tries to bind to it.

The machine is issued at work. At home, the FQDN of the hostname is not resolvable. It makes WFH more difficult. Even on VPN, somehow the same error happened from time to time. The exact cause of that behavior on VPN is unknown yet.

 

Issue Links

• relates to: YARN-5728 TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization timeout
• links to: GitHub Pull Request #1020