Critical Description
Apache ActiveMQ suddenly stopped working, with the followig traces:
2018-02-28 08:20:07,729 | ERROR | Looking for key 3035 but not found in fileMap:
2018-02-28 08:20:07,736 | ERROR | Failed to start Apache ActiveMQ ([localhost, null], java.io.IOException: Could not
locate data file /home/fc_1t40_pro_data/ActiveMQ/kahadb/db-3035.log) | org.apache.activemq.broker.BrokerService | mai
n
2018-02-28 08:20:07,778 | INFO | Apache ActiveMQ 5.13.3 (localhost, null) is shutting down | org.apache.activemq.bro
ker.BrokerService | main
2018-02-28 08:20:07,779 | INFO | Connector openwire stopped | org.apache.activemq.broker.TransportConnector | main
2018-02-28 08:20:07,781 | INFO | Connector amqp stopped | org.apache.activemq.broker.TransportConnector | main
2018-02-28 08:20:07,782 | INFO | Connector stomp stopped | org.apache.activemq.broker.TransportConnector | main
2018-02-28 08:20:07,783 | INFO | Connector mqtt stopped | org.apache.activemq.broker.TransportConnector | main
2018-02-28 08:20:07,785 | INFO | Connector ws stopped | org.apache.activemq.broker.TransportConnector | main
2018-02-28 08:20:07,813 | INFO | PListStore:[/home/fc_1t40_pro_data/ActiveMQ/localhost/tmp_storage] stopped | org.ap
ache.activemq.store.kahadb.plist.PListStoreImpl | main
2018-02-28 08:20:07,814 | INFO | Stopping async queue tasks | org.apache.activemq.store.kahadb.KahaDBStore | main
2018-02-28 08:20:07,815 | INFO | Stopping async topic tasks | org.apache.activemq.store.kahadb.KahaDBStore | main
2018-02-28 08:20:07,816 | INFO | Stopped KahaDB | org.apache.activemq.store.kahadb.KahaDBStore | main
2018-02-28 08:20:14,957 | INFO | Apache ActiveMQ 5.13.3 (localhost, null) uptime 160 days 16 hours | org.apache.acti
vemq.broker.BrokerService | main
2018-02-28 08:20:14,960 | INFO | Apache ActiveMQ 5.13.3 (localhost, null) is shutdown | org.apache.activemq.broker.B
rokerService | main
This has only happened in production server once, and after several document searching at apache Software foundation, I haven't found a workaround.
Currently, ActiveMQ broker configuration is:
activemq.xml
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor
license agreements. See the NOTICE file distributed with this work for additional
information regarding copyright ownership. The ASF licenses this file to
You under the Apache License, Version 2.0 (the "License"); you may not use
this file except in compliance with the License. You may obtain a copy of
the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required
by applicable law or agreed to in writing, software distributed under the
License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
OF ANY KIND, either express or implied. See the License for the specific
language governing permissions and limitations under the License. -->
<!-- START SNIPPET: example -->
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this configuration
file -->
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<!-- Allows accessing the server log -->
<bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton" init-method="start"
destroy-method="stop">
</bean>
<!-- The <broker> element is used to configure the ActiveMQ broker. -->
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.data}">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" expireMessagesPeriod="0"
optimizedDispatch="true">
<!-- The constantPendingMessageLimitStrategy is used to prevent slow
topic consumers to block producers and affect other consumers by limiting
the number of messages that are retained For more information, see: http://activemq.apache.org/slow-consumer-handling.html -->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy
limit="1000000" />
</pendingMessageLimitStrategy>
</policyEntry>
<policyEntry queue="ADNS_TASKS" prioritizedMessages="true"
useCache="false" expireMessagesPeriod="0" queuePrefetch="1">
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy
limit="1000000" />
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!-- The managementContext is used to configure how ActiveMQ is exposed
in JMX. By default, ActiveMQ uses the MBean server that is started by the
JVM. For more information, see: http://activemq.apache.org/jmx.html -->
<managementContext>
<managementContext createConnector="false" />
</managementContext>
<!-- Configure message persistence for the broker. The default persistence
mechanism is the KahaDB store (identified by the kahaDB tag). For more information,
see: http://activemq.apache.org/persistence.html -->
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"
ignoreMissingJournalfiles="true"
checkForCorruptJournalFiles="true"
checksumJournalFiles="true">
<locker>
<shared-file-locker lockAcquireSleepInterval="1000" />
</locker>
</kahaDB>
</persistenceAdapter>
<!-- The systemUsage controls the maximum amount of space the broker will
use before disabling caching and/or slowing down producers. For more information,
see: http://activemq.apache.org/producer-flow-control.html -->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb" />
</tempUsage>
</systemUsage>
</systemUsage>
<!-- The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see: http://activemq.apache.org/configuring-transports.html -->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size
to 100MB -->
<transportConnector name="openwire"
uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600" />
<transportConnector name="amqp"
uri="amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600" />
<transportConnector name="stomp"
uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600" />
<transportConnector name="mqtt"
uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600" />
<transportConnector name="ws"
uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600" />
</transportConnectors>
<!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
</broker>
<!-- Enable web consoles, REST and Ajax APIs and demos The web consoles
requires by default login, you can disable this in the jetty.xml file Take
a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details -->
<import resource="jetty.xml" />
</beans>
<!-- END SNIPPET: example -->
Attachments
Attachments
Issue Links
- is related to
-
AMQ-6938 KahaDB's ignoreMissingJournalFiles argument doesn't actually ignore missing journal files
-
- Open
-