[AMQ-6609] A lot async error and null pointer exception

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 5.5.0
• Component/s: JMX, Network of Brokers, Transport
• Labels: windows

Environment

2 virtual machines, in same network.
Having 2 AMQ server instance, with a network connection. HA - Active-Active mode.

CONFIG#1

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:amq="http://activemq.apache.org/schema/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd   http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <value>file:${activemq.base}/conf/credentials.properties</value>
        </property>
    </bean>
    <broker xmlns="http://activemq.apache.org/schema/core" brokerName="xx1" dataDirectory="${activemq.base}/data" destroyApplicationContextOnStop="true" deleteAllMessagesOnStartup="true" networkConnectorStartAsync="true" enableStatistics="false" schedulerSupport="false" useJmx="true">
        <destinationPolicy>
            <policyMap>
                <policyEntries>
                    <policyEntry topic="&gt;" producerFlowControl="true" memoryLimit="5mb">
                        <pendingSubscriberPolicy>
                            <fileCursor />
                        </pendingSubscriberPolicy>
                    </policyEntry>
                    <policyEntry queue="&gt;" producerFlowControl="true" memoryLimit="5mb" optimizedDispatch="true">
                        <deadLetterStrategy>
                            <sharedDeadLetterStrategy processExpired="false" />
                        </deadLetterStrategy>
                        <pendingQueuePolicy>
                            <fileQueueCursor />
                        </pendingQueuePolicy>
                    </policyEntry>
                </policyEntries>
            </policyMap>
        </destinationPolicy>
        <managementContext>
            <managementContext createConnector="false" />
        </managementContext>
        <persistenceAdapter>
            <kahaDB directory="${activemq.base}/data/kahadb" enableIndexWriteAsync="true" />
        </persistenceAdapter>
        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="500 mb" />
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="1 gb" />
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="1 gb" />
                </tempUsage>
            </systemUsage>
        </systemUsage>
        <transportConnectors>
            <transportConnector name="openwire" uri="nio://0.0.0.0:5672?keepAlive=true&amp;soTimeout=60000&amp;wireFormat.maxInactivityDurationInitalDelay=8000&amp;wireFormat.maxInactivityDuration=19000" />
        </transportConnectors>
    </broker>
    <import resource="jetty.xml" />
</beans>

CONFIG#2:

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:amq="http://activemq.apache.org/schema/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd   http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <value>file:${activemq.base}/conf/credentials.properties</value>
        </property>
    </bean>
    <broker xmlns="http://activemq.apache.org/schema/core" brokerName="xx2" dataDirectory="${activemq.base}/data" destroyApplicationContextOnStop="true" deleteAllMessagesOnStartup="true" networkConnectorStartAsync="true" enableStatistics="false" schedulerSupport="false" useJmx="true">
        <destinationPolicy>
            <policyMap>
                <policyEntries>
                    <policyEntry topic="&gt;" producerFlowControl="true" memoryLimit="5mb">
                        <pendingSubscriberPolicy>
                            <fileCursor />
                        </pendingSubscriberPolicy>
                    </policyEntry>
                    <policyEntry queue="&gt;" producerFlowControl="true" memoryLimit="5mb" optimizedDispatch="true">
                        <deadLetterStrategy>
                            <sharedDeadLetterStrategy processExpired="false" />
                        </deadLetterStrategy>
                        <pendingQueuePolicy>
                            <fileQueueCursor />
                        </pendingQueuePolicy>
                    </policyEntry>
                </policyEntries>
            </policyMap>
        </destinationPolicy>
        <managementContext>
            <managementContext createConnector="false" />
        </managementContext>
        <networkConnectors>
            <networkConnector uri="static:(tcp://xx1:5672)" duplex="true" conduitSubscriptions="true" decreaseNetworkConsumerPriority="true" />
        </networkConnectors>
        <persistenceAdapter>
            <kahaDB directory="${activemq.base}/data/kahadb" enableIndexWriteAsync="true" />
        </persistenceAdapter>
        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="500 mb" />
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="1 gb" />
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="1 gb" />
                </tempUsage>
            </systemUsage>
        </systemUsage>
        <transportConnectors>
            <transportConnector name="openwire" uri="nio://0.0.0.0:5672?keepAlive=true&amp;soTimeout=60000&amp;wireFormat.maxInactivityDurationInitalDelay=8000&amp;wireFormat.maxInactivityDuration=19000" />
        </transportConnectors>
    </broker>
    <import resource="jetty.xml" />
</beans>

• Flags: Important
• Assignee: Unassigned
• Reporter: Tamas Szabo
• Created: 23/Feb/17 16:31
• Updated: 23/Feb/17 16:54
• Resolved: 23/Feb/17 16:41

Description

We have a production system installed, and on peaks we get a lot error because of undelivered messages.
We have a java application which usually does the logic, and a c# application which makes requests to the java app.
The error is customer sees a lot slowness in their system because of these errors.

Errors are following:
Error-1

XX-1 LOG

2017-02-21 08:07:06,075 | WARN | Network connection between vm://xx-2#68 and tcp:///xx-2:5672 shutdown due to a remote error: java.net.SocketException: Software caused connection abort: recv failed | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///xx-2:5672
2017-02-21 08:07:06,075 | INFO | Network connection between vm://xx-2#68 and tcp:///xx-2:5672 shutdown due to a local error: java.net.SocketException: Software caused connection abort: socket write error | org.apache.activemq.network.DemandForwardingBridgeSupport | BrokerService[xx-2] Task-472152

XX-2 LOG

2017-02-21 08:07:05,476 | INFO | Network connection between vm://xx-1#68 and tcp:///xx-1:50234 shutdown due to a local error: javax.jms.JMSException: The destination temp-queue://ID:XYZA1234-65130-636232610113011707-1:0:2 does not exist. | org.apache.activemq.network.DemandForwardingBridgeSupport | BrokerService[xx-1] Task-592541
2017-02-21 08:07:10,500 | INFO | The connection to 'null' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Task-15133
2017-02-21 08:07:15,508 | INFO | The connection to 'null' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Task-15133
2017-02-21 08:07:20,520 | INFO | The connection to 'null' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Task-15133
2017-02-21 08:07:25,528 | INFO | The connection to 'null' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Task-15133
2017-02-21 08:07:30,536 | INFO | The connection to 'null' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Task-15133
2017-02-21 08:07:30,941 | INFO | Connector vm://xx-1 Stopped | org.apache.activemq.broker.TransportConnector | ActiveMQ Task-15134
2017-02-21 08:07:30,941 | INFO | xx-1 bridge to xx-2 stopped | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Task-15134
2017-02-21 08:08:11,051 | INFO | Connector vm://xx-1 Started | org.apache.activemq.broker.TransportConnector | ActiveMQ NIO Worker
2017-02-21 08:08:11,083 | INFO | Started responder end of duplex bridge NC@ID:XX-2-55636-1486253750115-0:1 | org.apache.activemq.broker.TransportConnection | ActiveMQ NIO Worker
2017-02-21 08:08:11,083 | INFO | Network connection between vm://xx-1#70 and tcp:///xx-1:50249(xx-2) has been established. | org.apache.activemq.network.DemandForwardingBridgeSupport | StartLocalBridge: localBroker=vm://xx-1#70
2017-02-21 08:32:23,098 | INFO | Transport failed: java.io.IOException: An existing connection was forcibly closed by the remote host | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 08:39:44,635 | INFO | Transport failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>19000) long: /11.111.111.11:64200 | org.apache.activemq.broker.TransportConnection.Transport | InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@37d226bd
2017-02-21 08:47:31,816 | INFO | Transport failed: java.io.IOException: An existing connection was forcibly closed by the remote host | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 08:51:17,393 | INFO | Transport failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>19000) long: /11.111.111.12:53436 | org.apache.activemq.broker.TransportConnection.Transport | InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@511249ed
2017-02-21 08:51:47,940 | INFO | Transport failed: java.io.IOException: An existing connection was forcibly closed by the remote host | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 08:59:18,723 | INFO | Transport failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>19000) long: /11.111.111.14:51599 | org.apache.activemq.broker.TransportConnection.Transport | InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@3d79ba00
2017-02-21 09:21:23,154 | INFO | Transport failed: java.io.IOException: An existing connection was forcibly closed by the remote host | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 09:28:38,819 | INFO | Transport failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>19000) long: /11.111.111.14:49185 | org.apache.activemq.broker.TransportConnection.Transport | InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@6a3566bf
2017-02-21 09:33:57,336 | INFO | Transport failed: java.io.IOException: An existing connection was forcibly closed by the remote host | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 09:49:52,090 | INFO | Transport failed: java.io.IOException: An existing connection was forcibly closed by the remote host | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 09:54:22,246 | INFO | Transport failed: java.io.IOException: An existing connection was forcibly closed by the remote host | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 10:12:02,324 | INFO | Transport failed: java.io.IOException: An existing connection was forcibly closed by the remote host | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker
2017-02-21 11:14:57,879 | INFO | ActiveMQ Message Broker (xx-1, ID:XX-1-49435-1486253561864-0:1) is shutting down | org.apache.activemq.broker.BrokerService | Thread-24
2017-02-21 11:14:57,879 | INFO | ActiveMQ Message Broker (xx-1, ID:XX-1-49435-1486253561864-0:1) is shutting down | org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook

Error-2
We have a lot of following errors (daily hundreds or more.)

2017-02-21 08:07:05,476 | WARN | Async error occurred: javax.jms.JMSException: The destination temp-queue://ID:XYZA1234-65130-123123123123123123-1:0:2 does not exist. | org.apache.activemq.broker.TransportConnection.Service | ActiveMQ NIO Worker
javax.jms.JMSException: The destination temp-queue://ID:XYZA1234-65130-123123123123123123-1:0:2 does not exist.
at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:135)