[SAMZA-476] Container fails due to port in use

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0
• Labels: newbie++
• Assignee: Unassigned
• Reporter: Yan Fang
• Created: 24/Nov/14 22:30
• Updated: 17/Nov/15 22:23

Description

When starting Samza, the first container sometimes fails because of the "Port already in use" error. After AM automatically restarts a new container, this error is gone.

To be more clear, I run a hello-samza job which starts two containers in local machine. When checking the log directory, I can find four containers, one is for AM (container-00001) , two (container-00003, container-00004) are for the running job, and another one (container-00002) has the following log:

Exception in thread "main" java.rmi.server.ExportException: Port already in use: 64356; nested exception is: 
	java.net.BindException: Address already in use
	at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:329)
	at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:237)
	at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411)
	at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
	at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:207)
	at java.rmi.server.UnicastRemoteObject.exportObject(UnicastRemoteObject.java:310)
	at java.rmi.server.UnicastRemoteObject.exportObject(UnicastRemoteObject.java:273)
	at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:118)
	at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:95)
	at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:403)
	at org.apache.samza.metrics.JmxServer.<init>(JmxServer.scala:90)
	at org.apache.samza.metrics.JmxServer.<init>(JmxServer.scala:44)
	at org.apache.samza.container.SamzaContainer$.safeMain$default$1(SamzaContainer.scala:72)
	at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:69)
	at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
Caused by: java.net.BindException: Address already in use
	at java.net.PlainSocketImpl.socketBind(Native Method)
	at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
	at java.net.ServerSocket.bind(ServerSocket.java:376)
	at java.net.ServerSocket.<init>(ServerSocket.java:237)
	at java.net.ServerSocket.<init>(ServerSocket.java:128)
	at sun.rmi.transport.proxy.RMIDirectSocketFactory.createServerSocket(RMIDirectSocketFactory.java:45)
	at sun.rmi.transport.proxy.RMIMasterSocketFactory.createServerSocket(RMIMasterSocketFactory.java:344)
	at sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666)
	at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:318)
	... 14 more

However, when I check the AM's log, I can not find this port is in use.