[AMQ-6134] ActiveMQ corrupted state when local disk fills up and Queues begin to buffer to KahaDB

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Resolved
• Affects Version/s: 5.11.1
• Component/s: KahaDB
• Assignee: Unassigned
• Reporter: Paul Smith
• Created: 21/Jan/16 10:59
• Updated: 14/Mar/16 17:03
• Resolved: 14/Mar/16 17:03

Description

There are already issues reporting this in AMQ-3098 and AMQ-5786 but they have been closed as Incomplete, I've left a comment on 5786 but I fear that it may get lost.

Fundamentally KahaDB & ActiveMQ do not survive if Queues start filling up and buffering to KahaDB disk when the disk then becomes full because of an external issue (in our case rsyslog).

Because KahaDB does not pre-allocate space, it seems vunerable to this case.

I will attach broker logs and Kahadb store during the corrupted state.

Issue Links

• is related to: AMQ-6084 Have an option to error out if a limit is exceeded
• is related to: AMQ-3098 Failed to page in more queue messages - Cannot consume messages
• is related to: AMQ-5786 ActiveMQ failed to start with KahaDB reporting corrupt journal records and throwing NegativeArraySizeException