[FLUME-874] Thirft server doesn't check if port is free

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.9.4
• Fix Version/s: 0.9.5
• Component/s: Sinks+Sources

Environment

RHEL 5

• Assignee: Unassigned
• Reporter: Ossi L
• Created: 08/Dec/11 13:48
• Updated: 05/Nov/14 10:55
• Resolved: 05/Nov/14 10:55

Description

When starting a collector with multiple flows / logical nodes configured, there might be conflict with overlapping ports.

2011-12-08 13:12:00,470 INFO org.mortbay.log: Extract /usr/lib/flume/webapps/flumeagent.war to /tmp/Jetty_0_0_0_0_35862_flumeagent.war_flumeagent_4cgrz3/webapp
2011-12-08 13:12:00,680 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:35862
2011-12-08 13:12:00,680 INFO com.cloudera.util.InternalHttpServer: Server started on port 35862
2011-12-08 13:12:05,785 INFO com.cloudera.flume.agent.LogicalNodeManager: Loading node name with FlumeConfigData: {srcVer:'Thu Dec 08 12:55:34 CET 2011' snkVer:'Thu Dec 08 12:55:34 CET 2011' ts='Thu Dec 08 12:55:34 CET 2011' flowId:'dk-flow-https-fe' source:'rpcSource( 35862 )' sink:'collectorSink( "hdfs://hq-priv-01:8020/flume/fusedk/httpd-fe/%Y-%m-%d/", "%

{host}ssl-access" )' }
2011-12-08 13:12:05,787 INFO com.cloudera.flume.agent.LogicalNode: Node config successfully set to FlumeConfigData: {srcVer:'Thu Dec 08 12:55:34 CET 2011' snkVer:'Thu Dec 08 12:55:34 CET 2011' ts='Thu Dec 08 12:55:34 CET 2011' flowId:'dk-flow-https-fe' source:'rpcSource( 35862 )' sink:'collectorSink( "hdfs://hq-priv-01:8020/flume/fusedk/httpd-fe/%Y-%m-%d/", "%{host}

ssl-access" )' }
2011-12-08 13:12:05,798 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Starting blocking thread pool server on port 35862...
org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:35862.

So, first flume start jetty on port 35862 and then it tries to start thrift server on the same port and it naturally fails.
Same problem will be with master if that server is running multiple logical nodes / flows.
And of course with any other service which happens to be using ports near above range.

This can be avoided by configuring http port of nodes and master to be than 35853, but it doesn't fix problem with some other services.

So, thrift should check if the port is free or not and bind only after that, if port is free.