[AMQ-2757] AMQ startup fails during Startup due to NullPointerException in KahaDB

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.2.0, 5.3.0, 5.3.1
• Fix Version/s: 5.4.0
• Component/s: Message Store

Environment

Red Hat Enterprise Linux, Kernel V2.6.18

• Assignee: Gary Tully
• Reporter: Thomas Schöftner
• Created: 02/Jun/10 11:29
• Updated: 12/Jul/10 11:20
• Resolved: 12/Jul/10 11:20

Description

During a long-time test run several message consumers listening to peristent queues crashed.
The queues were filling up rapidly, so that KahaDB allocated 3GB of disk space (max. limit 10GB) after 5hours (approx 800k messages).
At that point the broker was extremely slow and non-responsive to new messages.
JConsole showed that 1GB of RAM was used and most effort was necessary for garbage collection.
Although the max memory usage limit is configured to 1GB, AMQ should rather allocate disc space for messages to queues without containers.

AMQ shutdown took extremely long (several limits).
The subsequent AMQ restart keeps failing with the following exception:
java.lang.NullPointerException
at org.apache.kahadb.index.BTreeNode$BTreeIterator.findNextPage(BTreeNode.java:109)
at org.apache.kahadb.index.BTreeNode$BTreeIterator.hasNext(BTreeNode.java:120)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:223)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recover(KahaDBStore.java:220)
at org.apache.activemq.broker.region.Queue.initialize(Queue.java:252)
at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:464)
at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:120)
at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:268)
at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:142)
at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:142)
at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:149)
at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:94)
at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:178)
at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:101)
at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:154)
at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:112)
at org.apache.activemq.broker.BrokerService$3.start(BrokerService.java:1713)
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:473)
at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
...

The problem is reproducible also reproducible with recent nightly builds of AMQ/Fuse 5.4.0.

-Tom