[AMQ-7010] No space left on device leading to broker shutdown even if ignored

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 5.15.0, 5.15.1, 5.15.2, 5.15.3
• Component/s: KahaDB
• Assignee: Unassigned
• Reporter: Laurent Chiarello
• Created: 12/Jul/18 08:38
• Updated: 27/Jul/18 12:54
• Resolved: 27/Jul/18 12:34

Description

Recently, we upgraded our ActiveMQ instance from 5.13.2 to 5.15.3. We basically use an almost default broker setup with KahaDB. Due to our setup, we occasionally run into java.io.IOException: No space left on device, which until now was fine since the broker would properly ignore them using the DefaultIOExceptionHandler, and resume its work once space is available again.

After upgrading to 5.15.3, however, we noticed that the broker would actually stop itself following a no space left exception. Digging a little bit, it seems that the change in behavior can be linked to a fix introduced in the 5.15.0 release, see AMQ-6625. With this fix, KahaDB would sometimes throw the following exception: java.io.IOException: Async Writer Thread Shutdown from its DataFileAppender, which, unfortunately, is not handled by the DefaultIOExceptionHandler unless ignoreAllErrors is set to true.

This would basically mean that the DefaultIOExceptionHandler has become useless in catching only no space left exceptions, which I do not think is the intended behavior. 

How to reproduce

1. Start a broker with default config on a small partition
2. Send a persistent message with web console to any queue
3. Fill the partition by any mean
4. Send more messages with web console, broker is eventually shut down

See the attached broker logs.

Issue Links

• is caused by: AMQ-6625 IOException in kahaDB needs to pause pending IOExceptionHandler intervention