ActiveMQ
  1. ActiveMQ
  2. AMQ-3591

bridge transport is stuck on socket write at both ends

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 5.5.0
    • Fix Version/s: None
    • Component/s: Transport
    • Labels:
      None
    • Environment:

      2 redhat linux

      Description

      We have embedded (into Jboss 7 web app) broker A on one linux connecting through tcp duplex=true bridge to another standalone broker B. Both brokers exchange messages through 4 common queues. Each broker has its own persistence storage in mssql db.

      After a few hours of flawless work the bridge stops delivering the messages in both directions.

      broker A offending thread stack trace (causing the rest to wait on oneway mutex):
      BrokerService[as2analyticsEmbedded] Task-15 [RUNNABLE, IN_NATIVE] CPU time: 0:00
      java.net.SocketOutputStream.socketWrite0(FileDescriptor, byte[], int, int)
      java.net.SocketOutputStream.socketWrite(byte[], int, int)
      java.net.SocketOutputStream.write(byte[], int, int)
      org.apache.activemq.transport.tcp.TcpBufferedOutputStream.write(byte[], int, int)
      java.io.DataOutputStream.write(byte[], int, int)
      org.apache.activemq.openwire.v7.BaseDataStreamMarshaller.tightMarshalByteSequence2(ByteSequence, DataOutput, BooleanStream)
      org.apache.activemq.openwire.v7.MessageMarshaller.tightMarshal2(OpenWireFormat, Object, DataOutput, BooleanStream)
      org.apache.activemq.openwire.v7.ActiveMQMessageMarshaller.tightMarshal2(OpenWireFormat, Object, DataOutput, BooleanStream)
      org.apache.activemq.openwire.v7.ActiveMQObjectMessageMarshaller.tightMarshal2(OpenWireFormat, Object, DataOutput, BooleanStream)
      org.apache.activemq.openwire.OpenWireFormat.marshal(Object, DataOutput)
      org.apache.activemq.transport.tcp.TcpTransport.oneway(Object)
      org.apache.activemq.transport.InactivityMonitor.oneway(Object)
      org.apache.activemq.transport.TransportFilter.oneway(Object)
      org.apache.activemq.transport.WireFormatNegotiator.oneway(Object)
      org.apache.activemq.transport.failover.FailoverTransport.oneway(Object)
      org.apache.activemq.transport.MutexTransport.oneway(Object)
      org.apache.activemq.transport.ResponseCorrelator.asyncRequest(Object, ResponseCallback)
      org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(Command)
      org.apache.activemq.network.DemandForwardingBridgeSupport$1.onCommand(Object)
      org.apache.activemq.transport.ResponseCorrelator.onCommand(Object)
      org.apache.activemq.transport.TransportFilter.onCommand(Object)
      org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport, TransportListener, Object)
      org.apache.activemq.transport.vm.VMTransport.oneway(Object)
      org.apache.activemq.transport.MutexTransport.oneway(Object)
      org.apache.activemq.transport.ResponseCorrelator.oneway(Object)
      org.apache.activemq.broker.TransportConnection.dispatch(Command)
      org.apache.activemq.broker.TransportConnection.processDispatch(Command)
      org.apache.activemq.broker.TransportConnection.iterate()
      org.apache.activemq.thread.PooledTaskRunner.runTask()
      org.apache.activemq.thread.PooledTaskRunner$1.run()
      java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Runnable)
      java.util.concurrent.ThreadPoolExecutor$Worker.run()
      java.lang.Thread.run()

      broker B offending thread stack trace (causing the rest to wait on oneway mutex):

      ActiveMQ Connection Dispatcher: vm://analyticsCentral#6 [RUNNABLE, IN_NATIVE] CPU time: 0:00
      java.net.SocketOutputStream.socketWrite0(FileDescriptor, byte[], int, int)
      java.net.SocketOutputStream.socketWrite(byte[], int, int)
      java.net.SocketOutputStream.write(byte[], int, int)
      org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush()
      java.io.DataOutputStream.flush()
      org.apache.activemq.transport.tcp.TcpTransport.oneway(Object)
      org.apache.activemq.transport.InactivityMonitor.oneway(Object)
      org.apache.activemq.transport.TransportFilter.oneway(Object)
      org.apache.activemq.transport.WireFormatNegotiator.oneway(Object)
      org.apache.activemq.transport.MutexTransport.oneway(Object)
      org.apache.activemq.transport.ResponseCorrelator.asyncRequest(Object, ResponseCallback)
      org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(Command)
      org.apache.activemq.network.DemandForwardingBridgeSupport$1.onCommand(Object)
      org.apache.activemq.transport.ResponseCorrelator.onCommand(Object)
      org.apache.activemq.transport.TransportFilter.onCommand(Object)
      org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport, TransportListener, Object)
      org.apache.activemq.transport.vm.VMTransport.oneway(Object)
      org.apache.activemq.transport.MutexTransport.oneway(Object)
      org.apache.activemq.transport.ResponseCorrelator.oneway(Object)
      org.apache.activemq.broker.TransportConnection.dispatch(Command)
      org.apache.activemq.broker.TransportConnection.processDispatch(Command)
      org.apache.activemq.broker.TransportConnection.iterate()
      org.apache.activemq.thread.DedicatedTaskRunner.runTask()
      org.apache.activemq.thread.DedicatedTaskRunner$1.run()

      Both brokers run on a local 1 Gbit network. This scenario happens always after few hours of work and is recreated consistently.

      Broker A relevant configuration:
      <beans
      xmlns="http://www.springframework.org/schema/beans"
      xmlns:amq="http://activemq.apache.org/schema/core"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core-5.5.0.xsd"
      >

      <broker xmlns="http://activemq.apache.org/schema/core" brokerName="as2analyticsEmbedded" persistent="true" dataDirectory="/home/as2/amq_storage" destroyApplicationContextOnStop="true">
      <destinationPolicy>
      <policyMap>
      <policyEntries>
      <policyEntry queue=">" producerFlowControl="false" memoryLimit="1gb">
      <pendingQueuePolicy>
      <fileQueueCursor/>
      </pendingQueuePolicy>
      </policyEntry>
      </policyEntries>
      </policyMap>
      </destinationPolicy>

      <managementContext>
      <managementContext createConnector="false"/>
      </managementContext>

      <networkConnectors>
      <networkConnector uri="static:(failover:(tcp://10.0.213.38:61616))"
      name="bridge"
      duplex="true"
      conduitSubscriptions="true"
      decreaseNetworkConsumerPriority="false">
      </networkConnector>
      </networkConnectors>

      <!-- both FS and DB must be provided. AMQ will use DB for long-term storage.
      FS contents that has not been consumed yet is stored to DB from time to time -->
      <persistenceAdapter>
      <jdbcPersistenceAdapter dataDirectory="/home/as2/amq_storage/as2analyticsEmbedded" dataSource="#mssql-ds" useDatabaseLock="false">
      <adapter>
      <imageBasedJDBCAdaptor/>
      </adapter>
      </jdbcPersistenceAdapter>
      </persistenceAdapter>

      <systemUsage>
      <systemUsage>
      <memoryUsage>
      <memoryUsage limit="2gb"/>
      </memoryUsage>
      <storeUsage>
      <storeUsage limit="80gb"/>
      </storeUsage>
      <tempUsage>
      <tempUsage limit="10gb"/>
      </tempUsage>
      </systemUsage>
      </systemUsage>

      <transportConnectors>
      <transportConnector name="as2analyticsEmbeddedConnector" uri="vm://as2analyticsEmbedded"/>
      </transportConnectors>

      </broker>

      <bean id="mssql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
      <property name="driverClassName" value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
      <property name="url" value="jdbc:sqlserver://10.0.213.159:1433;databaseName=obfuscated"/>
      <property name="username" value="obfuscated"/>
      <property name="password" value="obfuscated"/>
      <property name="initialSize" value="1"/>
      <property name="maxActive" value="100"/>
      <property name="poolPreparedStatements" value="true"/>
      </bean>

      </beans>

      Broker B relevant configuration:

      <beans
      xmlns="http://www.springframework.org/schema/beans"
      xmlns:amq="http://activemq.apache.org/schema/core"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
      http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

      <!-- Allows us to use system properties as variables in this configuration file -->
      <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
      <property name="locations">
      <value>file:$

      {activemq.base}/conf/credentials.properties</value>
      </property>
      </bean>

      <!--
      The <broker> element is used to configure the ActiveMQ broker.
      -->
      <broker xmlns="http://activemq.apache.org/schema/core" brokerName="analyticsCentral" dataDirectory="${activemq.base}

      /data" destroyApplicationContextOnStop="true">

      <!--
      For better performances use VM cursor and small memory limit.
      For more information, see:

      http://activemq.apache.org/message-cursors.html

      Also, if your producer is "hanging", it's probably due to producer flow control.
      For more information, see:
      http://activemq.apache.org/producer-flow-control.html
      -->

      <destinationPolicy>
      <policyMap>
      <policyEntries>
      <policyEntry queue=">" producerFlowControl="false" memoryLimit="1gb">
      <pendingQueuePolicy>
      <fileQueueCursor/>
      </pendingQueuePolicy>
      </policyEntry>
      <policyEntry queue=">">
      <deadLetterStrategy>
      <individualDeadLetterStrategy
      queuePrefix="DLQ." useQueueForQueueMessages="true" />
      </deadLetterStrategy>
      </policyEntry>
      </policyEntries>
      </policyMap>
      </destinationPolicy>

      <!--
      The managementContext is used to configure how ActiveMQ is exposed in
      JMX. By default, ActiveMQ uses the MBean server that is started by
      the JVM. For more information, see:

      http://activemq.apache.org/jmx.html
      -->
      <managementContext>
      <managementContext createConnector="false"/>
      </managementContext>

      <!--
      Configure message persistence for the broker. The default persistence
      mechanism is the KahaDB store (identified by the kahaDB tag).
      For more information, see:

      http://activemq.apache.org/persistence.html
      -->
      <persistenceAdapter>
      <jdbcPersistenceAdapter dataDirectory="$

      {activemq.base}

      /data" dataSource="#mssql-ds" useDatabaseLock="false">
      <adapter>
      <imageBasedJDBCAdaptor/>
      </adapter>
      </jdbcPersistenceAdapter>
      </persistenceAdapter>

      <systemUsage>
      <systemUsage>
      <memoryUsage>
      <memoryUsage limit="2gb"/>
      </memoryUsage>
      <storeUsage>
      <storeUsage limit="10gb"/>
      </storeUsage>
      <tempUsage>
      <tempUsage limit="5gb"/>
      </tempUsage>
      </systemUsage>
      </systemUsage>

      <!--
      The transport connectors expose ActiveMQ over a given protocol to
      clients and other brokers. For more information, see:

      http://activemq.apache.org/configuring-transports.html
      -->
      <transportConnectors>
      <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
      </transportConnectors>

      </broker>

      <amq:connectionFactory id="jmsFactory" brokerURL="tcp://localhost">
      <amq:redeliveryPolicy>
      <amq:redeliveryPolicy maximumRedeliveries="0"/>
      </amq:redeliveryPolicy>
      </amq:connectionFactory>

      <bean id="mssql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
      <property name="driverClassName" value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
      <property name="url" value="jdbc:sqlserver://10.0.213.159:1433;DatabaseName=obfuscated"/>
      <property name="username" value="obfuscated"/>
      <property name="password" value="obfuscated"/>
      <property name="initialSize" value="1"/>
      <property name="maxActive" value="100"/>
      <property name="poolPreparedStatements" value="true"/>
      </bean>

      <!--
      Enable web consoles, REST and Ajax APIs and demos
      It also includes Camel (with its web console), see $

      {ACTIVEMQ_HOME}/conf/camel.xml for more info

      Take a look at ${ACTIVEMQ_HOME}

      /conf/jetty.xml for more details
      -->
      <import resource="jetty.xml"/>

      </beans>

        Activity

        Hide
        Gary Tully added a comment -

        what does netstat say about both ends of the tcp connection? Is the connection half closed or aborted on one end and we are waiting on tcp write timeouts.
        Also, the use of static: and failover: together is not recommended as the failover transport hides transport errors from the network connector, sometimes leaving it in an inconsistent state.

        Show
        Gary Tully added a comment - what does netstat say about both ends of the tcp connection? Is the connection half closed or aborted on one end and we are waiting on tcp write timeouts. Also, the use of static: and failover: together is not recommended as the failover transport hides transport errors from the network connector, sometimes leaving it in an inconsistent state.
        Hide
        Pavel Bekkerman added a comment -

        -------------------- central AMQ broker --------------------

        tcp 0 0 :::43648 :::* LISTEN 501 1804625 1429/java
        tcp 0 0 :::8161 :::* LISTEN 501 1804661 1429/java
        tcp 0 0 :::11600 :::* LISTEN 501 1804666 1429/java
        tcp 0 0 :::61616 :::* LISTEN 501 1804649 1429/java
        tcp 117068 158768 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.153:46485 ESTABLISHED 501 1807983 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50832 ESTABLISHED 501 2084078 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50833 ESTABLISHED 501 2084079 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50836 ESTABLISHED 501 2084083 1429/java
        tcp 0 0 ::ffff:10.0.213.239:36098 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096871 1429/java
        tcp 0 130 ::ffff:10.0.213.239:36107 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096900 1429/java
        tcp 0 0 ::ffff:10.0.213.239:36106 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096899 1429/java
        tcp 0 0 ::ffff:10.0.213.239:36074 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096840 1429/java
        tcp 0 254 ::ffff:10.0.213.239:35976 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096674 1429/java
        tcp 0 0 ::ffff:10.0.213.239:35997 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096702 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50838 ESTABLISHED 501 2084088 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50841 ESTABLISHED 501 2084097 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50842 ESTABLISHED 501 2084101 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50816 ESTABLISHED 501 2084013 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50817 ESTABLISHED 501 2084016 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50818 ESTABLISHED 501 2084027 1429/java
        tcp 0 0 ::ffff:10.0.213.239:39765 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096386 1429/java
        tcp 0 0 ::ffff:10.0.213.239:39816 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096522 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50819 ESTABLISHED 501 2084047 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50820 ESTABLISHED 501 2084048 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50822 ESTABLISHED 501 2084050 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50823 ESTABLISHED 501 2084051 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50824 ESTABLISHED 501 2084053 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50825 ESTABLISHED 501 2084052 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50826 ESTABLISHED 501 2084054 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50827 ESTABLISHED 501 2084055 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50829 ESTABLISHED 501 2084075 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50830 ESTABLISHED 501 2084076 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50802 ESTABLISHED 501 2083971 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50804 ESTABLISHED 501 2083985 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50807 ESTABLISHED 501 2083988 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50809 ESTABLISHED 501 2083991 1429/java
        tcp 0 2792 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50811 ESTABLISHED 501 2083997 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50813 ESTABLISHED 501 2084005 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50785 ESTABLISHED 501 2083934 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50787 ESTABLISHED 501 2083939 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50791 ESTABLISHED 501 2083951 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50778 ESTABLISHED 501 2083918 1429/java
        tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50779 ESTABLISHED 501 2083919 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35648 ESTABLISHED 501 2084059 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35651 ESTABLISHED 501 2084077 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35654 ESTABLISHED 501 2084080 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35655 ESTABLISHED 501 2084081 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35657 ESTABLISHED 501 2084087 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35659 ESTABLISHED 501 2084092 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35660 ESTABLISHED 501 2084094 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35663 ESTABLISHED 501 2084103 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35632 ESTABLISHED 501 2084001 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35634 ESTABLISHED 501 2084009 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35635 ESTABLISHED 501 2084010 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35641 ESTABLISHED 501 2084049 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35616 ESTABLISHED 501 2083961 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35617 ESTABLISHED 501 2083963 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35618 ESTABLISHED 501 2083967 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35619 ESTABLISHED 501 2083966 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35621 ESTABLISHED 501 2083969 1429/java
        tcp 0 15 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35623 ESTABLISHED 501 2083984 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35625 ESTABLISHED 501 2083986 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35626 ESTABLISHED 501 2083987 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35628 ESTABLISHED 501 2083990 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35630 ESTABLISHED 501 2083995 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35600 ESTABLISHED 501 2083921 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35601 ESTABLISHED 501 2083923 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35602 ESTABLISHED 501 2083927 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35603 ESTABLISHED 501 2083929 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35604 ESTABLISHED 501 2083931 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35606 ESTABLISHED 501 2083937 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35608 ESTABLISHED 501 2083943 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35609 ESTABLISHED 501 2083945 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35610 ESTABLISHED 501 2083950 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35612 ESTABLISHED 501 2083954 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35613 ESTABLISHED 501 2083955 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35614 ESTABLISHED 501 2083957 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35615 ESTABLISHED 501 2083959 1429/java
        tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35597 ESTABLISHED 501 2083912 1429/java

        -------------------- embedded AMQ broker --------------------

        tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 1008 2875431 6965/java
        tcp6 0 0 ::ffff:10.0.213.15:1090 :::* LISTEN 1008 2875470 6965/java
        tcp6 0 0 ::ffff:10.0.213.15:1091 :::* LISTEN 1008 2875474 6965/java
        tcp6 0 0 ::ffff:10.0.213.15:5445 :::* LISTEN 1008 2875490 6965/java
        tcp6 0 0 ::ffff:127.0.0.1:9990 :::* LISTEN 1008 2875454 6965/java
        tcp6 0 0 ::ffff:10.0.213.15:3528 :::* LISTEN 1008 2875478 6965/java
        tcp6 0 0 ::ffff:10.0.213.15:5455 :::* LISTEN 1008 2875491 6965/java
        tcp6 0 0 ::ffff:127.0.0.1:9999 :::* LISTEN 1008 2875448 6965/java
        tcp6 0 0 ::ffff:127.0.0.1:61616 :::* LISTEN 1008 2882636 6965/java
        tcp6 0 0 ::ffff:10.0.213.15:8080 :::* LISTEN 1008 2875462 6965/java
        tcp6 0 0 :::10001 :::* LISTEN 1008 2882641 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:39771 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2882632 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:39773 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2882735 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:39774 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2882737 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48487 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947137 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48486 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947135 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48485 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947133 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48484 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947131 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48483 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947129 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48482 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947127 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48481 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947125 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48480 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947123 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48492 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947148 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48491 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947146 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48490 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947144 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48489 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947142 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48488 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947140 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48479 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947121 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:48478 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947119 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51539 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945750 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51538 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945748 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51537 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945746 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51536 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945744 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51523 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945717 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51522 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945715 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51521 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945713 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51520 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945711 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51527 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945725 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51526 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945723 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51525 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945721 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51524 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945719 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51531 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945734 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51530 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945732 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51529 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945729 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51528 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945727 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51535 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945742 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51534 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945740 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51533 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945738 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51532 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945736 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51475 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945620 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51474 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945618 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51473 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945616 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51472 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945614 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51479 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945628 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51478 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945626 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51477 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945624 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51476 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945622 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51483 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945637 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51482 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945635 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51481 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945633 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51480 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945631 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51487 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945645 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51486 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945643 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51485 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945641 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51484 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945639 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51459 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945588 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51458 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945586 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51457 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945584 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51456 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945582 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51463 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945596 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51462 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945594 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51461 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945592 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51460 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945590 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51467 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945604 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51466 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945602 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51465 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945600 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51464 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945598 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51471 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945612 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51470 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945610 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51469 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945608 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51468 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945606 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51507 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945685 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51506 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945683 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51505 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945681 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51504 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945679 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51511 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945693 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51510 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945691 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51509 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945689 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51508 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945687 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51515 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945701 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51514 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945699 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51513 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945697 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51512 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945695 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51519 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945709 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51518 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945707 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51517 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945705 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51516 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945703 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51491 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945653 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51490 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945651 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51489 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945649 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51488 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945647 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51495 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945661 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51494 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945659 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51493 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945657 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51492 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945655 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51499 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945669 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51498 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945667 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51497 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945665 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51496 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945663 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51503 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945677 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51502 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945675 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51501 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945673 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51500 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945671 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51442 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945554 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51443 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945556 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51440 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945550 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51441 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945552 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51446 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945562 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51447 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945564 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51444 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945558 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51445 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945560 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51450 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945570 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51451 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945572 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51448 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945566 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51449 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945568 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51454 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945578 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51455 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945580 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51452 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945574 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51453 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945576 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51435 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945540 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51438 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945546 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51439 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945548 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51436 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945542 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:51437 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945544 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50812 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946451 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50813 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946453 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50814 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946455 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50808 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946443 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50809 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946445 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50810 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946447 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50811 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946449 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50804 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946435 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50805 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946437 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50806 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946439 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50807 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946441 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50800 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946427 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50801 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946429 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50802 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946431 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50803 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946433 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50796 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946419 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50797 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946421 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50798 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946423 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50799 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946425 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50792 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946411 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50793 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946413 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50794 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946415 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50795 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946417 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50788 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946403 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50789 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946405 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50790 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946407 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50791 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946409 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50785 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946397 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50786 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946399 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:50787 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946401 6965/java
        tcp6 96779 90960 ::ffff:10.0.213.1:46485 ::ffff:10.0.213.2:61616 ESTABLISHED1008 2882640 6965/java
        tcp6 0 0 ::ffff:10.0.213.1:57965 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2883672 6965/java

        Show
        Pavel Bekkerman added a comment - -------------------- central AMQ broker -------------------- tcp 0 0 :::43648 :::* LISTEN 501 1804625 1429/java tcp 0 0 :::8161 :::* LISTEN 501 1804661 1429/java tcp 0 0 :::11600 :::* LISTEN 501 1804666 1429/java tcp 0 0 :::61616 :::* LISTEN 501 1804649 1429/java tcp 117068 158768 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.153:46485 ESTABLISHED 501 1807983 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50832 ESTABLISHED 501 2084078 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50833 ESTABLISHED 501 2084079 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50836 ESTABLISHED 501 2084083 1429/java tcp 0 0 ::ffff:10.0.213.239:36098 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096871 1429/java tcp 0 130 ::ffff:10.0.213.239:36107 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096900 1429/java tcp 0 0 ::ffff:10.0.213.239:36106 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096899 1429/java tcp 0 0 ::ffff:10.0.213.239:36074 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096840 1429/java tcp 0 254 ::ffff:10.0.213.239:35976 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096674 1429/java tcp 0 0 ::ffff:10.0.213.239:35997 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096702 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50838 ESTABLISHED 501 2084088 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50841 ESTABLISHED 501 2084097 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50842 ESTABLISHED 501 2084101 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50816 ESTABLISHED 501 2084013 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50817 ESTABLISHED 501 2084016 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50818 ESTABLISHED 501 2084027 1429/java tcp 0 0 ::ffff:10.0.213.239:39765 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096386 1429/java tcp 0 0 ::ffff:10.0.213.239:39816 ::ffff:10.0.213.159:1433 ESTABLISHED 501 2096522 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50819 ESTABLISHED 501 2084047 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50820 ESTABLISHED 501 2084048 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50822 ESTABLISHED 501 2084050 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50823 ESTABLISHED 501 2084051 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50824 ESTABLISHED 501 2084053 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50825 ESTABLISHED 501 2084052 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50826 ESTABLISHED 501 2084054 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50827 ESTABLISHED 501 2084055 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50829 ESTABLISHED 501 2084075 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50830 ESTABLISHED 501 2084076 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50802 ESTABLISHED 501 2083971 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50804 ESTABLISHED 501 2083985 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50807 ESTABLISHED 501 2083988 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50809 ESTABLISHED 501 2083991 1429/java tcp 0 2792 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50811 ESTABLISHED 501 2083997 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50813 ESTABLISHED 501 2084005 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50785 ESTABLISHED 501 2083934 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50787 ESTABLISHED 501 2083939 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50791 ESTABLISHED 501 2083951 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50778 ESTABLISHED 501 2083918 1429/java tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:50779 ESTABLISHED 501 2083919 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35648 ESTABLISHED 501 2084059 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35651 ESTABLISHED 501 2084077 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35654 ESTABLISHED 501 2084080 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35655 ESTABLISHED 501 2084081 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35657 ESTABLISHED 501 2084087 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35659 ESTABLISHED 501 2084092 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35660 ESTABLISHED 501 2084094 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35663 ESTABLISHED 501 2084103 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35632 ESTABLISHED 501 2084001 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35634 ESTABLISHED 501 2084009 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35635 ESTABLISHED 501 2084010 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35641 ESTABLISHED 501 2084049 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35616 ESTABLISHED 501 2083961 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35617 ESTABLISHED 501 2083963 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35618 ESTABLISHED 501 2083967 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35619 ESTABLISHED 501 2083966 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35621 ESTABLISHED 501 2083969 1429/java tcp 0 15 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35623 ESTABLISHED 501 2083984 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35625 ESTABLISHED 501 2083986 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35626 ESTABLISHED 501 2083987 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35628 ESTABLISHED 501 2083990 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35630 ESTABLISHED 501 2083995 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35600 ESTABLISHED 501 2083921 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35601 ESTABLISHED 501 2083923 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35602 ESTABLISHED 501 2083927 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35603 ESTABLISHED 501 2083929 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35604 ESTABLISHED 501 2083931 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35606 ESTABLISHED 501 2083937 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35608 ESTABLISHED 501 2083943 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35609 ESTABLISHED 501 2083945 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35610 ESTABLISHED 501 2083950 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35612 ESTABLISHED 501 2083954 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35613 ESTABLISHED 501 2083955 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35614 ESTABLISHED 501 2083957 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35615 ESTABLISHED 501 2083959 1429/java tcp 0 0 ::ffff:10.0.213.239:61616 ::ffff:10.0.213.239:35597 ESTABLISHED 501 2083912 1429/java -------------------- embedded AMQ broker -------------------- tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 1008 2875431 6965/java tcp6 0 0 ::ffff:10.0.213.15:1090 :::* LISTEN 1008 2875470 6965/java tcp6 0 0 ::ffff:10.0.213.15:1091 :::* LISTEN 1008 2875474 6965/java tcp6 0 0 ::ffff:10.0.213.15:5445 :::* LISTEN 1008 2875490 6965/java tcp6 0 0 ::ffff:127.0.0.1:9990 :::* LISTEN 1008 2875454 6965/java tcp6 0 0 ::ffff:10.0.213.15:3528 :::* LISTEN 1008 2875478 6965/java tcp6 0 0 ::ffff:10.0.213.15:5455 :::* LISTEN 1008 2875491 6965/java tcp6 0 0 ::ffff:127.0.0.1:9999 :::* LISTEN 1008 2875448 6965/java tcp6 0 0 ::ffff:127.0.0.1:61616 :::* LISTEN 1008 2882636 6965/java tcp6 0 0 ::ffff:10.0.213.15:8080 :::* LISTEN 1008 2875462 6965/java tcp6 0 0 :::10001 :::* LISTEN 1008 2882641 6965/java tcp6 0 0 ::ffff:10.0.213.1:39771 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2882632 6965/java tcp6 0 0 ::ffff:10.0.213.1:39773 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2882735 6965/java tcp6 0 0 ::ffff:10.0.213.1:39774 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2882737 6965/java tcp6 0 0 ::ffff:10.0.213.1:48487 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947137 6965/java tcp6 0 0 ::ffff:10.0.213.1:48486 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947135 6965/java tcp6 0 0 ::ffff:10.0.213.1:48485 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947133 6965/java tcp6 0 0 ::ffff:10.0.213.1:48484 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947131 6965/java tcp6 0 0 ::ffff:10.0.213.1:48483 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947129 6965/java tcp6 0 0 ::ffff:10.0.213.1:48482 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947127 6965/java tcp6 0 0 ::ffff:10.0.213.1:48481 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947125 6965/java tcp6 0 0 ::ffff:10.0.213.1:48480 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947123 6965/java tcp6 0 0 ::ffff:10.0.213.1:48492 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947148 6965/java tcp6 0 0 ::ffff:10.0.213.1:48491 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947146 6965/java tcp6 0 0 ::ffff:10.0.213.1:48490 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947144 6965/java tcp6 0 0 ::ffff:10.0.213.1:48489 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947142 6965/java tcp6 0 0 ::ffff:10.0.213.1:48488 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947140 6965/java tcp6 0 0 ::ffff:10.0.213.1:48479 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947121 6965/java tcp6 0 0 ::ffff:10.0.213.1:48478 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2947119 6965/java tcp6 0 0 ::ffff:10.0.213.1:51539 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945750 6965/java tcp6 0 0 ::ffff:10.0.213.1:51538 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945748 6965/java tcp6 0 0 ::ffff:10.0.213.1:51537 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945746 6965/java tcp6 0 0 ::ffff:10.0.213.1:51536 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945744 6965/java tcp6 0 0 ::ffff:10.0.213.1:51523 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945717 6965/java tcp6 0 0 ::ffff:10.0.213.1:51522 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945715 6965/java tcp6 0 0 ::ffff:10.0.213.1:51521 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945713 6965/java tcp6 0 0 ::ffff:10.0.213.1:51520 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945711 6965/java tcp6 0 0 ::ffff:10.0.213.1:51527 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945725 6965/java tcp6 0 0 ::ffff:10.0.213.1:51526 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945723 6965/java tcp6 0 0 ::ffff:10.0.213.1:51525 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945721 6965/java tcp6 0 0 ::ffff:10.0.213.1:51524 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945719 6965/java tcp6 0 0 ::ffff:10.0.213.1:51531 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945734 6965/java tcp6 0 0 ::ffff:10.0.213.1:51530 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945732 6965/java tcp6 0 0 ::ffff:10.0.213.1:51529 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945729 6965/java tcp6 0 0 ::ffff:10.0.213.1:51528 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945727 6965/java tcp6 0 0 ::ffff:10.0.213.1:51535 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945742 6965/java tcp6 0 0 ::ffff:10.0.213.1:51534 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945740 6965/java tcp6 0 0 ::ffff:10.0.213.1:51533 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945738 6965/java tcp6 0 0 ::ffff:10.0.213.1:51532 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945736 6965/java tcp6 0 0 ::ffff:10.0.213.1:51475 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945620 6965/java tcp6 0 0 ::ffff:10.0.213.1:51474 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945618 6965/java tcp6 0 0 ::ffff:10.0.213.1:51473 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945616 6965/java tcp6 0 0 ::ffff:10.0.213.1:51472 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945614 6965/java tcp6 0 0 ::ffff:10.0.213.1:51479 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945628 6965/java tcp6 0 0 ::ffff:10.0.213.1:51478 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945626 6965/java tcp6 0 0 ::ffff:10.0.213.1:51477 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945624 6965/java tcp6 0 0 ::ffff:10.0.213.1:51476 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945622 6965/java tcp6 0 0 ::ffff:10.0.213.1:51483 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945637 6965/java tcp6 0 0 ::ffff:10.0.213.1:51482 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945635 6965/java tcp6 0 0 ::ffff:10.0.213.1:51481 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945633 6965/java tcp6 0 0 ::ffff:10.0.213.1:51480 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945631 6965/java tcp6 0 0 ::ffff:10.0.213.1:51487 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945645 6965/java tcp6 0 0 ::ffff:10.0.213.1:51486 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945643 6965/java tcp6 0 0 ::ffff:10.0.213.1:51485 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945641 6965/java tcp6 0 0 ::ffff:10.0.213.1:51484 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945639 6965/java tcp6 0 0 ::ffff:10.0.213.1:51459 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945588 6965/java tcp6 0 0 ::ffff:10.0.213.1:51458 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945586 6965/java tcp6 0 0 ::ffff:10.0.213.1:51457 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945584 6965/java tcp6 0 0 ::ffff:10.0.213.1:51456 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945582 6965/java tcp6 0 0 ::ffff:10.0.213.1:51463 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945596 6965/java tcp6 0 0 ::ffff:10.0.213.1:51462 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945594 6965/java tcp6 0 0 ::ffff:10.0.213.1:51461 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945592 6965/java tcp6 0 0 ::ffff:10.0.213.1:51460 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945590 6965/java tcp6 0 0 ::ffff:10.0.213.1:51467 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945604 6965/java tcp6 0 0 ::ffff:10.0.213.1:51466 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945602 6965/java tcp6 0 0 ::ffff:10.0.213.1:51465 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945600 6965/java tcp6 0 0 ::ffff:10.0.213.1:51464 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945598 6965/java tcp6 0 0 ::ffff:10.0.213.1:51471 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945612 6965/java tcp6 0 0 ::ffff:10.0.213.1:51470 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945610 6965/java tcp6 0 0 ::ffff:10.0.213.1:51469 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945608 6965/java tcp6 0 0 ::ffff:10.0.213.1:51468 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945606 6965/java tcp6 0 0 ::ffff:10.0.213.1:51507 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945685 6965/java tcp6 0 0 ::ffff:10.0.213.1:51506 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945683 6965/java tcp6 0 0 ::ffff:10.0.213.1:51505 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945681 6965/java tcp6 0 0 ::ffff:10.0.213.1:51504 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945679 6965/java tcp6 0 0 ::ffff:10.0.213.1:51511 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945693 6965/java tcp6 0 0 ::ffff:10.0.213.1:51510 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945691 6965/java tcp6 0 0 ::ffff:10.0.213.1:51509 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945689 6965/java tcp6 0 0 ::ffff:10.0.213.1:51508 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945687 6965/java tcp6 0 0 ::ffff:10.0.213.1:51515 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945701 6965/java tcp6 0 0 ::ffff:10.0.213.1:51514 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945699 6965/java tcp6 0 0 ::ffff:10.0.213.1:51513 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945697 6965/java tcp6 0 0 ::ffff:10.0.213.1:51512 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945695 6965/java tcp6 0 0 ::ffff:10.0.213.1:51519 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945709 6965/java tcp6 0 0 ::ffff:10.0.213.1:51518 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945707 6965/java tcp6 0 0 ::ffff:10.0.213.1:51517 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945705 6965/java tcp6 0 0 ::ffff:10.0.213.1:51516 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945703 6965/java tcp6 0 0 ::ffff:10.0.213.1:51491 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945653 6965/java tcp6 0 0 ::ffff:10.0.213.1:51490 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945651 6965/java tcp6 0 0 ::ffff:10.0.213.1:51489 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945649 6965/java tcp6 0 0 ::ffff:10.0.213.1:51488 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945647 6965/java tcp6 0 0 ::ffff:10.0.213.1:51495 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945661 6965/java tcp6 0 0 ::ffff:10.0.213.1:51494 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945659 6965/java tcp6 0 0 ::ffff:10.0.213.1:51493 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945657 6965/java tcp6 0 0 ::ffff:10.0.213.1:51492 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945655 6965/java tcp6 0 0 ::ffff:10.0.213.1:51499 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945669 6965/java tcp6 0 0 ::ffff:10.0.213.1:51498 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945667 6965/java tcp6 0 0 ::ffff:10.0.213.1:51497 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945665 6965/java tcp6 0 0 ::ffff:10.0.213.1:51496 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945663 6965/java tcp6 0 0 ::ffff:10.0.213.1:51503 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945677 6965/java tcp6 0 0 ::ffff:10.0.213.1:51502 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945675 6965/java tcp6 0 0 ::ffff:10.0.213.1:51501 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945673 6965/java tcp6 0 0 ::ffff:10.0.213.1:51500 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945671 6965/java tcp6 0 0 ::ffff:10.0.213.1:51442 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945554 6965/java tcp6 0 0 ::ffff:10.0.213.1:51443 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945556 6965/java tcp6 0 0 ::ffff:10.0.213.1:51440 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945550 6965/java tcp6 0 0 ::ffff:10.0.213.1:51441 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945552 6965/java tcp6 0 0 ::ffff:10.0.213.1:51446 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945562 6965/java tcp6 0 0 ::ffff:10.0.213.1:51447 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945564 6965/java tcp6 0 0 ::ffff:10.0.213.1:51444 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945558 6965/java tcp6 0 0 ::ffff:10.0.213.1:51445 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945560 6965/java tcp6 0 0 ::ffff:10.0.213.1:51450 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945570 6965/java tcp6 0 0 ::ffff:10.0.213.1:51451 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945572 6965/java tcp6 0 0 ::ffff:10.0.213.1:51448 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945566 6965/java tcp6 0 0 ::ffff:10.0.213.1:51449 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945568 6965/java tcp6 0 0 ::ffff:10.0.213.1:51454 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945578 6965/java tcp6 0 0 ::ffff:10.0.213.1:51455 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945580 6965/java tcp6 0 0 ::ffff:10.0.213.1:51452 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945574 6965/java tcp6 0 0 ::ffff:10.0.213.1:51453 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945576 6965/java tcp6 0 0 ::ffff:10.0.213.1:51435 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945540 6965/java tcp6 0 0 ::ffff:10.0.213.1:51438 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945546 6965/java tcp6 0 0 ::ffff:10.0.213.1:51439 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945548 6965/java tcp6 0 0 ::ffff:10.0.213.1:51436 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945542 6965/java tcp6 0 0 ::ffff:10.0.213.1:51437 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2945544 6965/java tcp6 0 0 ::ffff:10.0.213.1:50812 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946451 6965/java tcp6 0 0 ::ffff:10.0.213.1:50813 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946453 6965/java tcp6 0 0 ::ffff:10.0.213.1:50814 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946455 6965/java tcp6 0 0 ::ffff:10.0.213.1:50808 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946443 6965/java tcp6 0 0 ::ffff:10.0.213.1:50809 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946445 6965/java tcp6 0 0 ::ffff:10.0.213.1:50810 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946447 6965/java tcp6 0 0 ::ffff:10.0.213.1:50811 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946449 6965/java tcp6 0 0 ::ffff:10.0.213.1:50804 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946435 6965/java tcp6 0 0 ::ffff:10.0.213.1:50805 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946437 6965/java tcp6 0 0 ::ffff:10.0.213.1:50806 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946439 6965/java tcp6 0 0 ::ffff:10.0.213.1:50807 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946441 6965/java tcp6 0 0 ::ffff:10.0.213.1:50800 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946427 6965/java tcp6 0 0 ::ffff:10.0.213.1:50801 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946429 6965/java tcp6 0 0 ::ffff:10.0.213.1:50802 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946431 6965/java tcp6 0 0 ::ffff:10.0.213.1:50803 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946433 6965/java tcp6 0 0 ::ffff:10.0.213.1:50796 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946419 6965/java tcp6 0 0 ::ffff:10.0.213.1:50797 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946421 6965/java tcp6 0 0 ::ffff:10.0.213.1:50798 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946423 6965/java tcp6 0 0 ::ffff:10.0.213.1:50799 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946425 6965/java tcp6 0 0 ::ffff:10.0.213.1:50792 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946411 6965/java tcp6 0 0 ::ffff:10.0.213.1:50793 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946413 6965/java tcp6 0 0 ::ffff:10.0.213.1:50794 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946415 6965/java tcp6 0 0 ::ffff:10.0.213.1:50795 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946417 6965/java tcp6 0 0 ::ffff:10.0.213.1:50788 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946403 6965/java tcp6 0 0 ::ffff:10.0.213.1:50789 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946405 6965/java tcp6 0 0 ::ffff:10.0.213.1:50790 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946407 6965/java tcp6 0 0 ::ffff:10.0.213.1:50791 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946409 6965/java tcp6 0 0 ::ffff:10.0.213.1:50785 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946397 6965/java tcp6 0 0 ::ffff:10.0.213.1:50786 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946399 6965/java tcp6 0 0 ::ffff:10.0.213.1:50787 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2946401 6965/java tcp6 96779 90960 ::ffff:10.0.213.1:46485 ::ffff:10.0.213.2:61616 ESTABLISHED1008 2882640 6965/java tcp6 0 0 ::ffff:10.0.213.1:57965 ::ffff:10.0.213.15:1433 ESTABLISHED1008 2883672 6965/java
        Hide
        Pavel Bekkerman added a comment -

        I'm yet to try to remove failover: (and leave static: only), but above is our netstat, as you suggeted.
        I don't see any unusual socket states - LISTEN or ESTABLISHED.
        Any ideas on how to proceed with the solution?

        Show
        Pavel Bekkerman added a comment - I'm yet to try to remove failover: (and leave static: only), but above is our netstat, as you suggeted. I don't see any unusual socket states - LISTEN or ESTABLISHED. Any ideas on how to proceed with the solution?
        Hide
        Gary Tully added a comment -

        remove failover, also check for flow control messages in the logs, the send blocking on a memory usage limit.
        Attach a full thread dump of both brokers, you show the blocked write, but what is the corresponding reader thread doing on the other end of the transport connection?
        From netstat it looks like the tcp connection is alive so the only reason for a blocked write is push back from the other end where a send must be blocked.

        Show
        Gary Tully added a comment - remove failover, also check for flow control messages in the logs, the send blocking on a memory usage limit. Attach a full thread dump of both brokers, you show the blocked write, but what is the corresponding reader thread doing on the other end of the transport connection? From netstat it looks like the tcp connection is alive so the only reason for a blocked write is push back from the other end where a send must be blocked.
        Hide
        Pavel Bekkerman added a comment -

        I've removed "failover:" and running the test that replicates the block.

        In the meantime, something that might be related to our case:

        We have a producer P1 for a queue Q1 connected to the central broker CB.
        We have a consumer C1 for a queue Q1 connected to the embedded broke EB.
        CB and EB are connected using duplex network (as described above).
        We send 1000 messages using P1 to Q1. No other activity is performed.
        While processing of each message from Q1 by C1 may take minutes, we see in the CB web console, that all 1000 messages were dequeued in seconds after being sent.

        Is this behavior normal?

        Show
        Pavel Bekkerman added a comment - I've removed "failover:" and running the test that replicates the block. In the meantime, something that might be related to our case: We have a producer P1 for a queue Q1 connected to the central broker CB. We have a consumer C1 for a queue Q1 connected to the embedded broke EB. CB and EB are connected using duplex network (as described above). We send 1000 messages using P1 to Q1. No other activity is performed. While processing of each message from Q1 by C1 may take minutes, we see in the CB web console, that all 1000 messages were dequeued in seconds after being sent. Is this behavior normal?
        Hide
        Pavel Bekkerman added a comment -

        Also, we're sure to create 5 consumers on Q1 to consumer from EB.
        But we see in the CB web console only 1 consumer on Q1.

        Is this behavior normal?

        (P.S. We checked without EB, and it shows the correct number of consumers - 5.)

        Show
        Pavel Bekkerman added a comment - Also, we're sure to create 5 consumers on Q1 to consumer from EB. But we see in the CB web console only 1 consumer on Q1. Is this behavior normal? (P.S. We checked without EB, and it shows the correct number of consumers - 5.)

          People

          • Assignee:
            Unassigned
            Reporter:
            Leon Fleysher
          • Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development