[AMQ-6376] IOException on pageIn should involve IOExeceptionHandler

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.13.0
• Fix Version/s: 5.14.0
• Component/s: Broker, Message Store
• Assignee: Gary Tully
• Reporter: Gary Tully
• Created: 27/Jul/16 10:52
• Updated: 27/Jul/16 14:59
• Resolved: 27/Jul/16 11:00

Description

This relates to AMQ-6372 where the broker was running out of file descriptors.
However if the IOException occurred for dispatch when the cache is disabled the broker carries on in error. The IOExceptionHandler is invoked on every store write failure, but should also be on the read error path.
This ensures a timely response to disk IO errors.

2016-07-27 11:50:26,292 [calhost] Task-1] - ERROR AbstractStoreCursor            - org.apache.activemq.broker.region.cursors.QueueStorePrefetch@7cd532ee:Test,batchResetNeeded=false,size=50,cacheEnabled=false,maxBatchSize:50,hasSpace:true,pendingCachedIds.size:0,lastSyncCachedId:null,lastSyncCachedId-seq:null,lastAsyncCachedId:null,lastAsyncCachedId-seq:null,store=permits:10000,sd=nextSeq:50,lastRet:MessageOrderCursor:[def:0, low:0, high:0],pending:0 - Failed to fill batch
java.io.EOFException
	at java.io.RandomAccessFile.readFully(RandomAccessFile.java:446)
	at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424)
	at org.apache.activemq.util.RecoverableRandomAccessFile.readFully(RecoverableRandomAccessFile.java:75)
	at org.apache.activemq.store.kahadb.disk.journal.DataFileAccessor.readRecord(DataFileAccessor.java:88)
	at org.apache.activemq.store.kahadb.disk.journal.Journal.read(Journal.java:867)
	at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:1118)
	at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1177)
	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:584)
	at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)
	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:573)
	at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:110)
	at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:120)
	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:406)
	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:149)
	at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:169)
	at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1935)
	at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2160)

Issue Links

• relates to: AMQ-6372 KahaDB reader pool not releasing unused open files till deletion - too many open files