[AMQ-5542] KahaDB data files containing acknowledgements are deleted during cleanup

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.10.0, 5.10.1
• Fix Version/s: 5.11.1, 5.12.0
• Component/s: Message Store
• Assignee: Gary Tully
• Reporter: Sergiy Barlabanov
• Created: 27/Jan/15 20:40
• Updated: 18/Jun/15 17:50
• Resolved: 02/Feb/15 13:07

Description

AMQ-2832 was not fixed cleanly.
The commit dd68c61e65f24b7dc498b36e34960a4bc46ded4b by Gary from 8.10.2010 introduced a problem by deleting too many files.

Scenarios we are facing currently in production:
Data file #1 contains unconsumed messages sitting in a DLQ. So this file is not a cleanup candidate.
The next file #2 contains acks of some messages from file #1. This file is not a cleanup candidate (because of ackMessageFileMap logic).
The next file #3 contains acks of some messages from file #2. And this file is deleted during the cleanup procedure. So on Broker restart all messages from #2, whose acks were from the deleted file #3, are replayed!
The reason is gcCandidates variable, which is a copy of gcCandidateSet (see MessageDatabase#checkpointUpdate at the end of the method - org/apache/activemq/store/kahadb/MessageDatabase.java:1659 on 5.10.0 tag). So when a candidate is deleted from gcCandidateSet (org/apache/activemq/store/kahadb/MessageDatabase.java:1668 on 5.10.0 tag), gcCandidates still contains that candidate and the comparison on org/apache/activemq/store/kahadb/MessageDatabase.java:1666 works wrong!
I will try to adjust AMQ2832Test.

Issue Links

• is broken by: AMQ-2736 KahaDB doesn't clean up old files
• is related to: AMQ-5695 KahaDB not cleaning up log files