[AMQ-3310] IOException PListStore "Could not locate data file" from FilePendingMessageCursor

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.5.0
• Fix Version/s: 5.6.0
• Component/s: Message Store
• Labels: PListStore cursors temp
• Assignee: Gary Tully
• Reporter: Gary Tully
• Created: 09/May/11 15:16
• Updated: 20/Sep/11 10:42
• Resolved: 09/May/11 16:12

Description

Corruption of the temp store used by file pending message cursors due to removal of data files in error. Contention over reference counts and async writes. Reproducible under heavy load with many concurrent consumers.

java.lang.RuntimeException: java.io.IOException: Could not locate data file ./tmp_storage/db-332.log
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.<init>(FilePendingMessageCursor.java:486)
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.reset(FilePendingMessageCursor.java:129)
        at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
Caused by: java.io.IOException: Could not locate data file ./tmp_storage/db-332.log
        at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
        at org.apache.kahadb.journal.Journal.read(Journal.java:597)
        at org.apache.activemq.store.kahadb.plist.PListStore.getPayload(PListStore.java:337)
        at org.apache.activemq.store.kahadb.plist.PList.getFirst(PList.java:280)
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor$DiskIterator.<init>(FilePendingMessageCursor.java:482)