[AMQ-5703] kahadb - index recovery - corrupt journal records cannot be skipped

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.11.0
• Fix Version/s: 5.12.0
• Component/s: KahaDB, Message Store
• Labels: kahadb recovery
• Assignee: Gary Tully
• Reporter: Gary Tully
• Created: 01/Apr/15 13:45
• Updated: 01/Apr/15 15:20
• Resolved: 01/Apr/15 15:20

Description

Working with some corrupt data stores - if corruption occurs in the middle of a journal and the index needs to be rebuilt we get:

java.io.EOFException
	at java.io.RandomAccessFile.readFully(RandomAccessFile.java:446)
	at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424)
	at org.apache.activemq.util.RecoverableRandomAccessFile.readFully(RecoverableRandomAccessFile.java:75)
	at org.apache.activemq.store.kahadb.disk.journal.DataFileAccessor.readRecord(DataFileAccessor.java:87)
	at org.apache.activemq.store.kahadb.disk.journal.Journal.read(Journal.java:641)
	at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:1014)
	at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:606)
	at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:400)
	at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:418)
	at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:262)
	at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:206)
	at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
	at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:223)
	at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
	at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:652)
	at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:641)
	at org.apache.activemq.broker.BrokerService.start(BrokerService.java:606)

attempting to read an invalid location. This stops further recovery and the entire journal needs to be removed to progress.
We have already identified the corrupt record, we just need to skip it when we replay.