Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Won't Fix
-
5.12.0
-
None
-
None
-
Centos 6.5, Zookeeper 3.4.6, VMWare
Description
Hi,
We are seeing a failure on the integrity of our LevelDB datastore which is causing the broker to stop processing messages.
We've set up ActiveMQ as a cluster of three nodes (master/slave/slave), each with a local Zookeeper node, each AMQ/ZK pair sitting on a VMWare host. All based on this guide: http://activemq.apache.org/replicated-leveldb-store.html
We have 10 Tomcat instances sending messages to the active broker, and 12 Tomcat instances reading from the VirtualQueue.
After several hours runtime, we see an exception in the log:
2015-12-15 04:03:17,173 | INFO | Ignoring IO exception, java.io.IOException | org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException handler.
java.io.IOException
at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)[activemq-client-5.12.0.jar:5.12.0]
at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:1044)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at org.apache.activemq.leveldb.LevelDBClient.store(LevelDBClient.scala:1390)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at org.apache.activemq.leveldb.DBManager$$anonfun$drainFlushes$1.apply$mcV$sp(DBManager.scala:627)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)[hawtdispatch-scala-2.11-1.21.jar:1.21]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_51]
at java.lang.Thread.run(Thread.java:744)[:1.7.0_51]
2015-12-15 04:03:17,194 | INFO | Suppressing reply to: ActiveMQTextMessage
on: org.apache.activemq.broker.SuppressReplyException: Store has been stopped, cause: null | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: tcp:///10.241.163.60:40607@61616
2015-12-15 04:03:17,195 | WARN | Async error occurred: org.apache.activemq.broker.SuppressReplyException: Store has been stopped | org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport: tcp:///10.241.163.60:40607@61616
2015-12-15 04:03:17,198 | INFO | Ignoring IO exception, java.io.IOException: Not running | org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ Transport: tcp:///10.241.163.60:40607@61616
java.io.IOException: Not running
The broker then stops and does not allow any messages to pass through until the ActiveMQ node has been failed across manually.
I have attached my activemq.xml configuration.
Many thanks