Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-874

Thirft server doesn't check if port is free

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.9.4
    • Fix Version/s: 0.9.5
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Environment:

      RHEL 5

      Description

      When starting a collector with multiple flows / logical nodes configured, there might be conflict with overlapping ports.

      2011-12-08 13:12:00,470 INFO org.mortbay.log: Extract /usr/lib/flume/webapps/flumeagent.war to /tmp/Jetty_0_0_0_0_35862_flumeagent.war_flumeagent_4cgrz3/webapp
      2011-12-08 13:12:00,680 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:35862
      2011-12-08 13:12:00,680 INFO com.cloudera.util.InternalHttpServer: Server started on port 35862
      2011-12-08 13:12:05,785 INFO com.cloudera.flume.agent.LogicalNodeManager: Loading node name with FlumeConfigData: {srcVer:'Thu Dec 08 12:55:34 CET 2011' snkVer:'Thu Dec 08 12:55:34 CET 2011' ts='Thu Dec 08 12:55:34 CET 2011' flowId:'dk-flow-https-fe' source:'rpcSource( 35862 )' sink:'collectorSink( "hdfs://hq-priv-01:8020/flume/fusedk/httpd-fe/%Y-%m-%d/", "%

      {host}ssl-access" )' }
      2011-12-08 13:12:05,787 INFO com.cloudera.flume.agent.LogicalNode: Node config successfully set to FlumeConfigData: {srcVer:'Thu Dec 08 12:55:34 CET 2011' snkVer:'Thu Dec 08 12:55:34 CET 2011' ts='Thu Dec 08 12:55:34 CET 2011' flowId:'dk-flow-https-fe' source:'rpcSource( 35862 )' sink:'collectorSink( "hdfs://hq-priv-01:8020/flume/fusedk/httpd-fe/%Y-%m-%d/", "%{host}

      ssl-access" )' }
      2011-12-08 13:12:05,798 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Starting blocking thread pool server on port 35862...
      org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:35862.

      So, first flume start jetty on port 35862 and then it tries to start thrift server on the same port and it naturally fails.
      Same problem will be with master if that server is running multiple logical nodes / flows.
      And of course with any other service which happens to be using ports near above range.

      This can be avoided by configuring http port of nodes and master to be than 35853, but it doesn't fix problem with some other services.

      So, thrift should check if the port is free or not and bind only after that, if port is free.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              lossil Ossi L
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: