ActiveMQ
  1. ActiveMQ
  2. AMQ-2757

AMQ startup fails during Startup due to NullPointerException in KahaDB

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.2.0, 5.3.0, 5.3.1
    • Fix Version/s: 5.4.0
    • Component/s: Message Store
    • Labels:
      None
    • Environment:

      Red Hat Enterprise Linux, Kernel V2.6.18

      Description

      During a long-time test run several message consumers listening to peristent queues crashed.
      The queues were filling up rapidly, so that KahaDB allocated 3GB of disk space (max. limit 10GB) after 5hours (approx 800k messages).
      At that point the broker was extremely slow and non-responsive to new messages.
      JConsole showed that 1GB of RAM was used and most effort was necessary for garbage collection.
      Although the max memory usage limit is configured to 1GB, AMQ should rather allocate disc space for messages to queues without containers.

      AMQ shutdown took extremely long (several limits).
      The subsequent AMQ restart keeps failing with the following exception:
      java.lang.NullPointerException
      at org.apache.kahadb.index.BTreeNode$BTreeIterator.findNextPage(BTreeNode.java:109)
      at org.apache.kahadb.index.BTreeNode$BTreeIterator.hasNext(BTreeNode.java:120)
      at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:223)
      at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
      at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recover(KahaDBStore.java:220)
      at org.apache.activemq.broker.region.Queue.initialize(Queue.java:252)
      at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
      at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:464)
      at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
      at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:120)
      at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:268)
      at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:142)
      at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:142)
      at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:149)
      at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:94)
      at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:178)
      at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:101)
      at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:154)
      at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:112)
      at org.apache.activemq.broker.BrokerService$3.start(BrokerService.java:1713)
      at org.apache.activemq.broker.BrokerService.start(BrokerService.java:473)
      at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:85)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      ...

      The problem is reproducible also reproducible with recent nightly builds of AMQ/Fuse 5.4.0.

      -Tom

      1. wrapper.log
        31 kB
        Thomas Schöftner
      2. wrapper.conf
        6 kB
        Thomas Schöftner
      3. activemq.xml
        7 kB
        Thomas Schöftner
      4. activemq.log
        8 kB
        Thomas Schöftner

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        39d 23h 50m 1 Gary Tully 12/Jul/10 12:20
        Jeff Turner made changes -
        Project Import Fri Nov 26 22:32:02 EST 2010 [ 1290828722158 ]
        Gary Tully made changes -
        Resolution Fixed [ 1 ]
        Status Open [ 1 ] Resolved [ 5 ]
        Hide
        Hiram Chirino added a comment -

        Just committed a fix that should resolved the issue reported here.

        Fixed in rev 956741 on trunk. Try it out in a subsequent snapshot.

        Show
        Hiram Chirino added a comment - Just committed a fix that should resolved the issue reported here. Fixed in rev 956741 on trunk. Try it out in a subsequent snapshot.
        Hide
        Thomas Schöftner added a comment -

        The problem is related to the VMQueueCursor being active.
        If the queues are switched back to the default queue cursor, AMQ starts up smoothly.
        However, the behaviour is strange as AMQ is nowhere near it's memory limits when it crashes.

        Show
        Thomas Schöftner added a comment - The problem is related to the VMQueueCursor being active. If the queues are switched back to the default queue cursor, AMQ starts up smoothly. However, the behaviour is strange as AMQ is nowhere near it's memory limits when it crashes.
        Gary Tully made changes -
        Assignee Gary Tully [ gtully ]
        Fix Version/s 5.4.0 [ 12110 ]
        Thomas Schöftner made changes -
        Attachment wrapper.conf [ 19348 ]
        Hide
        Thomas Schöftner added a comment -

        attached wrapper config

        Show
        Thomas Schöftner added a comment - attached wrapper config
        Thomas Schöftner made changes -
        Field Original Value New Value
        Attachment wrapper.log [ 19346 ]
        Attachment activemq.log [ 19345 ]
        Attachment activemq.xml [ 19347 ]
        Thomas Schöftner created issue -

          People

          • Assignee:
            Gary Tully
            Reporter:
            Thomas Schöftner
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development