[AMQ-4262] Pending send transaction can block on store usage and not resume once backlog is cleared

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.7.0
• Fix Version/s: 5.8.0
• Component/s: Message Store
• Labels: blocking gc kahadb pfc store transactions usage
• Assignee: Gary Tully
• Reporter: Gary Tully
• Created: 18/Jan/13 22:27
• Updated: 21/Jan/13 11:44
• Resolved: 21/Jan/13 11:44

Description

A transacted send can block pending store usage, an active transaction will be visible to the store. This will block kahadb from gc of data files that occur after the inflight transaction.
Consumers acking the pending messages can create additional data files that can't be gced. This is a problem because the queue can drain and not release the producer.
With message database trace is is possible to see:

14:20:20,515 | INFO  | .0.1:51273@61616 | Queue                            | 125 - org.apache.activemq.activemq-core - 5.7.0.fuse-71-047 | Usage(default:store:queue://flood_out:store) percentUsage=99%, usage=30447772, limit=20971520, percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=145%, usage=30447772, limit=20971520, percentUsageMinDelta=1%: Persistent store is Full, 100% of 20971520. Stopping producer (ID:..) to prevent flooding queue://flood_out. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 5650s)
.....

14:20:45,016 | TRACE | heckpoint Worker | MessageDatabase                  | 125 - org.apache.activemq.activemq-core - 5.7.0.fuse-71-047 | Last update: 30:453228, full gc candidates set: [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30]
14:20:45,016 | TRACE | heckpoint Worker | MessageDatabase                  | 125 - org.apache.activemq.activemq-core - 5.7.0.fuse-71-047 | gc candidates after first tx:11:860775, []
14:20:45,016 | TRACE | heckpoint Worker | MessageDatabase                  | 125 - org.apache.activemq.activemq-core - 5.7.0.fuse-71-047 | gc candidates: []

Here there is a build up of data files, after the inflight transaction, containing acks and messages that could be cleaned up.

The inflight transactions should protect a range of data files that they reference, not all subsequent data files.