[HDFS-11508] Fix bind failure in SimpleTCPServer & Portmap where bind fails because socket is in TIME_WAIT state

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-alpha4
• Component/s: nfs
• Hadoop Flags: Reviewed
• Assignee: Mukul Kumar Singh
• Reporter: Mukul Kumar Singh
• Created: 07/Mar/17 04:55
• Updated: 24/Apr/18 20:46
• Resolved: 07/Mar/17 19:42

Description

Bind can fail in SimpleTCPServer & Portmap because socket is in TIME_WAIT state.

Socket options should be changed here to use the setReuseAddress option.

2017-03-05 08:37:13,283 INFO  oncrpc.SimpleUdpServer (SimpleUdpServer.java:run(73)) - Started listening to UDP requests at port 4242 for Rpc program: mountd at localhost:4242 with workerCount 1
2017-03-05 08:37:13,298 FATAL mount.MountdBase (MountdBase.java:startTCPServer(85)) - Failed to start the TCP server.
org.jboss.netty.channel.ChannelException: Failed to bind to: 0.0.0.0/0.0.0.0:4242
	at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
	at org.apache.hadoop.oncrpc.SimpleTcpServer.run(SimpleTcpServer.java:86)
	at org.apache.hadoop.mount.MountdBase.startTCPServer(MountdBase.java:83)
	at org.apache.hadoop.mount.MountdBase.start(MountdBase.java:98)
	at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startServiceInternal(Nfs3.java:56)
	at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startService(Nfs3.java:69)
	at org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter.start(PrivilegedNfsGatewayStarter.java:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
Caused by: java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:433)
	at sun.nio.ch.Net.bind(Net.java:425)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2017-03-05 08:37:13,304 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
2017-03-05 08:37:13,308 INFO  nfs3.Nfs3Base (LogAdapter.java:info(45)) - SHUTDOWN_MSG: 
/************************************************************

Issue Links

• relates to: HDFS-11533 reuseAddress option should be used for child channels in Portmap and SimpleTcpServer