[AMQ-3051] Failed to read journal - Could not locate data file

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 5.4.1
• Component/s: Broker

Environment

Solaris 10 running on Sparc

• Assignee: Unassigned
• Reporter: Keith Brown
• Created: 24/Nov/10 14:26
• Updated: 07/Dec/12 22:21
• Resolved: 07/Dec/12 22:21

Description

We are consistently seeing the following exception in our Production environment. At the current time we are seeing over 200K messages on the queue and no consumers can consume any messages. Due to issues with the kahadb not deleting files, we are using the AMQStore for persistence. We are using a NFS drive for the output of our persistent store.

The process cannot then be restarted as it will fail trying to load back in the file store. The messages stored in the queue are critical, a way to recover these messages is important also.

2010-11-24 08:20:44,015 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:Consumer.CaseManagement.VirtualTopic.intralinks.eventService
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Failed to read to journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could not locate data file data-17376
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1681)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1845)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1432)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.lang.RuntimeException: java.io.IOException: Failed to read to journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could not locate data file data-17376
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:261)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
... 6 more
Caused by: java.io.IOException: Failed to read to journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could not locate data file data-17376
at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
at org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:680)
at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:564)
at org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:444)
at org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:59)
at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:89)
at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:127)
at org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:537)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:258)
... 7 more
Caused by: java.io.IOException: Could not locate data file data-17376
at org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:315)
at org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:637)
at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:561)
... 14 more