Major Description
Dear Support,
We are having KHADB on a shared DFS storage connected to a two nodes as master slave AMQ approach. The system was running for about two weeks and suddenly it stopped as shown below in the logs, please give me your recommendations.
NODE1:
INFO | jvm 1 | 2015/03/04 12:54:11 | INFO | Database \\qatch-fs\AMQ_Data_PRO\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '\\qatch-fs\AMQ_Data_PRO\data\kahadb\lock' could not be locked.
INFO | jvm 1 | 2015/03/04 12:54:36 | WARN | Index corrupted. Recovering the index through journal replay. Cause:java.io.IOException: The requested resource is in use
INFO | jvm 1 | 2015/03/04 12:54:36 | ERROR | Failed to start Apache ActiveMQ ([CenBroker, null], java.io.IOException: Could not delete: \\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e\db.data)
INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Apache ActiveMQ 5.10.0 (CenBroker, null) is shutting down
INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Connector openwire stopped
INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | PListStore:\\HQ-QTCM1\data\CenBroker\tmp_storage stopped
INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopping async queue tasks
INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopping async topic tasks
INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopped KahaDB
INFO | jvm 1 | 2015/03/04 12:54:36 | ERROR | Could not stop service: KahaDB:\\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e. Reason: java.lang.IllegalStateException: Cannot unload the page file when it is not loaded
INFO | jvm 1 | 2015/03/04 12:54:36 | java.lang.IllegalStateException: Cannot unload the page file when it is not loaded
INFO | jvm 1 | 2015/03/04 12:54:36 |
at org.apache.activemq.store.kahadb.disk.page.PageFile.unload(PageFile.java:484)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:36 |
at org.apache.activemq.store.kahadb.MessageDatabase.close(MessageDatabase.java:432)[activemq-kahadb-store-5.10.0.jar:5.10.0]
Node2:
INFO | jvm 1 | 2015/03/04 12:54:07 | ERROR | Checkpoint failed
INFO | jvm 1 | 2015/03/04 12:54:07 | java.io.IOException: The requested resource is in use
INFO | jvm 1 | 2015/03/04 12:54:07 |
at sun.nio.ch.FileDispatcherImpl.force0(Native Method)[:1.7.0_71]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at sun.nio.ch.FileDispatcherImpl.force(Unknown Source)[:1.7.0_71]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at sun.nio.ch.FileChannelImpl.force(Unknown Source)[:1.7.0_71]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at org.apache.activemq.util.RecoverableRandomAccessFile.sync(RecoverableRandomAccessFile.java:402)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at org.apache.activemq.store.kahadb.disk.page.PageFile.writeBatch(PageFile.java:1094)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at org.apache.activemq.store.kahadb.disk.page.PageFile.flush(PageFile.java:516)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1512)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at org.apache.activemq.store.kahadb.MessageDatabase$17.execute(MessageDatabase.java:1484)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1481)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:929)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:357)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 | INFO | Stopping BrokerService[CenBroker] due to exception, java.io.IOException: The requested resource is in use
Best Regadrs,
Botros Kfoury