[AMQ-6558] Constant reconnecting after ActiveMQ restart with HTTP Layer

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 5.14.1
• Component/s: Network of Brokers

Environment

all OSs

• Assignee: Unassigned
• Reporter: Jiri Formanek
• Created: 11/Jan/17 17:04
• Updated: 13/Jan/17 19:28
• Resolved: 13/Jan/17 19:28

Description

ActiveMQ broker reconnects constantly after ActiveMQ restart, when HTTP is used as transport layer.

Let's consider following simple broker network, where machine A connects to machine B:

Machine A config:

<networkConnector duplex="true" uri="static:(http://machineB:61617)" name="bridgeToB" />

Machine B config:

<transportConnector name="http" uri="http://0.0.0.0:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>

ActiveMQ on machine A is restarted. After the restart the connection to machine B is successfully re-established. But then after 30 seconds the keep-alive mechanism sends a HTTP GET request to machine B and this fails. Keep-alive failure causes the stopping of connection and establishing of new connection, which will be again stopped after 30 seconds. And so on over and over...see the log:

2017-01-11 14:32:19,005 | INFO  | Establishing network connection from vm://localhost?async=false&create=false to http://machineB:61617 | org.apache.activemq.network.DiscoveryNetworkConnector
| ActiveMQ Task-1017
2017-01-11 14:32:20,726 | INFO  | Broker Servlet supports GZip compression. | org.apache.activemq.transport.http.HttpClientTransport | ActiveMQ Task-1017
2017-01-11 14:32:20,923 | INFO  | Network connection between vm://localhost#9512 and HTTP Reader http://machineB:61617 (localhost) has been established. | org.apache.activemq.network.DemandFor
wardingBridgeSupport | triggerStartAsyncNetworkBridgeCreation: remoteBroker=HTTP Reader http://machineB:61617, localBroker= vm://localhost#9512
2017-01-11 14:32:54,912 | WARN  | Network connection between vm://localhost#9512 and HTTP Reader http://machineB:61617 shutdown due to a remote error: java.io.IOException: Failed to perform GE
T on: http://machineB:61617 Reason: Read timed out | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: HTTP Reader http://machineB:61617
2017-01-11 14:32:54,917 | INFO  | localhost bridge to localhost stopped | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[localhost] Task-4377
2017-01-11 14:32:55,913 | INFO  | Establishing network connection from vm://localhost?async=false&create=false to http://machineB:61617 | org.apache.activemq.network.DiscoveryNetworkConnector
| ActiveMQ Task-1018
2017-01-11 14:32:57,642 | INFO  | Broker Servlet supports GZip compression. | org.apache.activemq.transport.http.HttpClientTransport | ActiveMQ Task-1018
2017-01-11 14:32:57,841 | INFO  | Network connection between vm://localhost#9516 and HTTP Reader http://machineB:61617 (localhost) has been established. | org.apache.activemq.network.DemandFor
wardingBridgeSupport | triggerStartAsyncNetworkBridgeCreation: remoteBroker=HTTP Reader http://machineB:61617, localBroker= vm://localhost#9516
2017-01-11 14:33:31,429 | WARN  | Network connection between vm://localhost#9516 and HTTP Reader http://machineB:61617 shutdown due to a remote error: java.io.IOException: Failed to perform GE
T on: http://machineB:61617 Reason: Read timed out | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: HTTP Reader http://machineB:61617
2017-01-11 14:33:31,433 | INFO  | localhost bridge to localhost stopped | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[localhost] Task-4390
2017-01-11 14:33:32,430 | INFO  | Establishing network connection from vm://localhost?async=false&create=false to http://machineB:61617 | org.apache.activemq.network.DiscoveryNetworkConnector
| ActiveMQ Task-1019
2017-01-11 14:33:34,157 | INFO  | Broker Servlet supports GZip compression. | org.apache.activemq.transport.http.HttpClientTransport | ActiveMQ Task-1019
2017-01-11 14:33:34,355 | INFO  | Network connection between vm://localhost#9520 and HTTP Reader http://machineB:61617 (localhost) has been established. | org.apache.activemq.network.DemandFor
wardingBridgeSupport | triggerStartAsyncNetworkBridgeCreation: remoteBroker=HTTP Reader http://machineB:61617, localBroker= vm://localhost#9520
2017-01-11 14:34:07,169 | WARN  | Network connection between vm://localhost#9520 and HTTP Reader http://machineB:61617 shutdown due to a remote error: java.io.IOException: Failed to perform GE
T on: http://machineB:61617 Reason: Read timed out | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: HTTP Reader http://machineB:61617
2017-01-11 14:34:07,175 | INFO  | localhost bridge to localhost stopped | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[localhost] Task-4400
2017-01-11 14:34:08,170 | INFO  | Establishing network connection from vm://localhost?async=false&create=false to http://machineB:61617 | org.apache.activemq.network.DiscoveryNetworkConnector
| ActiveMQ Task-1020
2017-01-11 14:34:09,898 | INFO  | Broker Servlet supports GZip compression. | org.apache.activemq.transport.http.HttpClientTransport | ActiveMQ Task-1020
2017-01-11 14:34:10,095 | INFO  | Network connection between vm://localhost#9524 and HTTP Reader http://machineB:61617 (localhost) has been established. | org.apache.activemq.network.DemandFor
wardingBridgeSupport | triggerStartAsyncNetworkBridgeCreation: remoteBroker=HTTP Reader http://machineB:61617, localBroker= vm://localhost#9524
2017-01-11 14:34:42,505 | WARN  | Network connection between vm://localhost#9524 and HTTP Reader http://machineB:61617 shutdown due to a remote error: java.io.IOException: Failed to perform GE
T on: http://machineB:61617 Reason: Read timed out | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: HTTP Reader http://machineB:61617
2017-01-11 14:34:42,508 | INFO  | localhost bridge to localhost stopped | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[localhost] Task-4407
2017-01-11 14:34:43,506 | INFO  | Establishing network connection from vm://localhost?async=false&create=false to http://machineB:61617 | org.apache.activemq.network.DiscoveryNetworkConnector
| ActiveMQ Task-1021
2017-01-11 14:34:45,221 | INFO  | Broker Servlet supports GZip compression. | org.apache.activemq.transport.http.HttpClientTransport | ActiveMQ Task-1021
2017-01-11 14:34:45,419 | INFO  | Network connection between vm://localhost#9528 and HTTP Reader http://machineB:61617 (localhost) has been established. | org.apache.activemq.network.DemandFor
wardingBridgeSupport | triggerStartAsyncNetworkBridgeCreation: remoteBroker=HTTP Reader http://machineB:61617, localBroker= vm://localhost#9528
2017-01-11 14:35:19,002 | WARN  | Network connection between vm://localhost#9528 and HTTP Reader http://machineB:61617 shutdown due to a remote error: java.io.IOException: Failed to perform GE
T on: http://machineB:61617 Reason: Read timed out | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: HTTP Reader http://machineB:61617
2017-01-11 14:35:19,006 | INFO  | localhost bridge to localhost stopped | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[localhost] Task-4415
2017-01-11 14:35:20,003 | INFO  | Establishing network connection from vm://localhost?async=false&create=false to http://machineB:61617 | org.apache.activemq.network.DiscoveryNetworkConnector
| ActiveMQ Task-1022
2017-01-11 14:35:21,751 | INFO  | Broker Servlet supports GZip compression. | org.apache.activemq.transport.http.HttpClientTransport | ActiveMQ Task-1022
2017-01-11 14:35:21,949 | INFO  | Network connection between vm://localhost#9532 and HTTP Reader http://machineB:61617 (localhost) has been established. | org.apache.activemq.network.DemandFor
wardingBridgeSupport | triggerStartAsyncNetworkBridgeCreation: remoteBroker=HTTP Reader http://machineB:61617, localBroker= vm://localhost#9532
2017-01-11 14:35:55,973 | WARN  | Network connection between vm://localhost#9532 and HTTP Reader http://machineB:61617 shutdown due to a remote error: java.io.IOException: Failed to perform GE

Issue Links

• duplicates: AMQ-6557 Constant reconnecting after ActiveMQ restart with HTTP Layer