ActiveMQ
  1. ActiveMQ
  2. AMQ-2757

AMQ startup fails during Startup due to NullPointerException in KahaDB

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.2.0, 5.3.0, 5.3.1
    • Fix Version/s: 5.4.0
    • Component/s: Message Store
    • Labels:
      None
    • Environment:

      Red Hat Enterprise Linux, Kernel V2.6.18

      Description

      During a long-time test run several message consumers listening to peristent queues crashed.
      The queues were filling up rapidly, so that KahaDB allocated 3GB of disk space (max. limit 10GB) after 5hours (approx 800k messages).
      At that point the broker was extremely slow and non-responsive to new messages.
      JConsole showed that 1GB of RAM was used and most effort was necessary for garbage collection.
      Although the max memory usage limit is configured to 1GB, AMQ should rather allocate disc space for messages to queues without containers.

      AMQ shutdown took extremely long (several limits).
      The subsequent AMQ restart keeps failing with the following exception:
      java.lang.NullPointerException
      at org.apache.kahadb.index.BTreeNode$BTreeIterator.findNextPage(BTreeNode.java:109)
      at org.apache.kahadb.index.BTreeNode$BTreeIterator.hasNext(BTreeNode.java:120)
      at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:223)
      at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
      at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recover(KahaDBStore.java:220)
      at org.apache.activemq.broker.region.Queue.initialize(Queue.java:252)
      at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
      at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:464)
      at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
      at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:120)
      at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:268)
      at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:142)
      at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:142)
      at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:149)
      at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:94)
      at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:178)
      at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:101)
      at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:154)
      at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:112)
      at org.apache.activemq.broker.BrokerService$3.start(BrokerService.java:1713)
      at org.apache.activemq.broker.BrokerService.start(BrokerService.java:473)
      at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:85)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      ...

      The problem is reproducible also reproducible with recent nightly builds of AMQ/Fuse 5.4.0.

      -Tom

      1. wrapper.conf
        6 kB
        Thomas Schöftner
      2. activemq.xml
        7 kB
        Thomas Schöftner
      3. wrapper.log
        31 kB
        Thomas Schöftner
      4. activemq.log
        8 kB
        Thomas Schöftner

        Activity

        Jeff Turner made changes -
        Project Import Fri Nov 26 22:32:02 EST 2010 [ 1290828722158 ]
        Gary Tully made changes -
        Resolution Fixed [ 1 ]
        Status Open [ 1 ] Resolved [ 5 ]
        Hide
        Hiram Chirino added a comment -

        Just committed a fix that should resolved the issue reported here.

        Fixed in rev 956741 on trunk. Try it out in a subsequent snapshot.

        Show
        Hiram Chirino added a comment - Just committed a fix that should resolved the issue reported here. Fixed in rev 956741 on trunk. Try it out in a subsequent snapshot.
        Hiram R. Chirino <chirino@apache.org> committed 9ae41192a9d2c39fcc82d8dfd101392be8698456 (1 file)
        Reviews: none

        Fix for AMQ-2757 - leaf nodes were not properly getting linked together in some cases.
        git-svn-id: https://svn.apache.org/repos/asf/activemq/trunk@956741 13f79535-47bb-0310-9956-ffa450edef68

        Hide
        Thomas Schöftner added a comment -

        The problem is related to the VMQueueCursor being active.
        If the queues are switched back to the default queue cursor, AMQ starts up smoothly.
        However, the behaviour is strange as AMQ is nowhere near it's memory limits when it crashes.

        Show
        Thomas Schöftner added a comment - The problem is related to the VMQueueCursor being active. If the queues are switched back to the default queue cursor, AMQ starts up smoothly. However, the behaviour is strange as AMQ is nowhere near it's memory limits when it crashes.
        Gary Tully made changes -
        Assignee Gary Tully [ gtully ]
        Fix Version/s 5.4.0 [ 12110 ]
        Thomas Schöftner made changes -
        Attachment wrapper.conf [ 19348 ]
        Hide
        Thomas Schöftner added a comment -

        attached wrapper config

        Show
        Thomas Schöftner added a comment - attached wrapper config
        Thomas Schöftner made changes -
        Field Original Value New Value
        Attachment wrapper.log [ 19346 ]
        Attachment activemq.log [ 19345 ]
        Attachment activemq.xml [ 19347 ]
        Thomas Schöftner created issue -

          People

          • Assignee:
            Gary Tully
            Reporter:
            Thomas Schöftner
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development