Uploaded image for project: 'ActiveMQ'
  1. ActiveMQ
  2. AMQ-5225

broker will not start when using leveldb

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 5.10.0
    • Fix Version/s: 5.14.0
    • Component/s: activemq-leveldb-store
    • Labels:
      None
    • Environment:

      Centos 6.3

      Description

      I've configured a 3 node activemq cluster using this as my guide: http://activemq.apache.org/replicated-leveldb-store.html

      1) Startup activemq on 1st node.
      2) Startup activemq on 2nd node

      At this point I see a log message on the 1st node stating it was promoted to master. Next I see this Exception and am unable to connect to the broker. Each time an activemq instance is promoted to master, this exception occurs and broker is left unusable.

      java.io.IOException: com.google.common.base.Objects.firstNonNull(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
      at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)[activemq-client-5.10.0.jar:5.10.0]
      at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)[activemq-leveldb-store-5.10.0.jar:5.10.0]
      at org.apache.activemq.leveldb.LevelDBClient.replay_init(LevelDBClient.scala:657)[activemq-leveldb-store-5.10.0.jar:5.10.0]
      at org.apache.activemq.leveldb.LevelDBClient.start(LevelDBClient.scala:558)[activemq-leveldb-store-5.10.0.jar:5.10.0]
      at org.apache.activemq.leveldb.DBManager.start(DBManager.scala:648)[activemq-leveldb-store-5.10.0.jar:5.10.0]
      at org.apache.activemq.leveldb.LevelDBStore.doStart(LevelDBStore.scala:235)[activemq-leveldb-store-5.10.0.jar:5.10.0]
      at org.apache.activemq.leveldb.replicated.MasterLevelDBStore.doStart(MasterLevelDBStore.scala:110)[activemq-leveldb-store-5.10.0.jar:5.10.0]
      at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.10.0.jar:5.10.0]
      at org.apache.activemq.leveldb.replicated.ElectingLevelDBStore$$anonfun$start_master$1.apply$mcV$sp(ElectingLevelDBStore.scala:226)[activemq-leveldb-store-5.10.0.jar:5.10.0]
      at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)[hawtdispatch-scala-2.11-1.21.jar:1.21]
      at java.lang.Thread.run(Thread.java:744)[:1.7.0_51]
      2014-06-12 19:02:32,350 | INFO | Stopped LevelDB[/opt/apache-activemq-5.10.0/bin/linux-x86-64/../../data/leveldb] | org.apache.activemq.leveldb.LevelDBStore | LevelDB IOException handler.

        Activity

        Hide
        johnleach John Leach added a comment -

        I'm getting the same with just a straight single broker. The config has worked for many months under 5.9.0, but upgrading to 5.10.0 gives this on startup and never finishes starting up:

        2014-06-13 11:11:10,931 | INFO | Using Persistence Adapter: LevelDB[/home/activemq/apache-activemq/data/leveldb] | org.apache.activemq.broker.BrokerService | main
        2014-06-13 11:11:10,999 | INFO | Using the pure java LevelDB implementation. | org.apache.activemq.leveldb.LevelDBClient | main
        2014-06-13 11:11:11,074 | INFO | No IOExceptionHandler registered, ignoring IO exception | org.apache.activemq.broker.BrokerService | LevelDB IOException handler.
        java.io.IOException: com.google.common.base.Objects.firstNonNull(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
        at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)[activemq-client-5.10.0.jar:5.10.0]
        at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)[activemq-leveldb-store-5.10.0.jar:5.10.0]
        at org.apache.activemq.leveldb.LevelDBClient.replay_init(LevelDBClient.scala:657)[activemq-leveldb-store-5.10.0.jar:5.10.0]
        at org.apache.activemq.leveldb.LevelDBClient.start(LevelDBClient.scala:558)[activemq-leveldb-store-5.10.0.jar:5.10.0]
        at org.apache.activemq.leveldb.DBManager.start(DBManager.scala:648)[activemq-leveldb-store-5.10.0.jar:5.10.0]
        at org.apache.activemq.leveldb.LevelDBStore.doStart(LevelDBStore.scala:235)[activemq-leveldb-store-5.10.0.jar:5.10.0]
        at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:640)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:629)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.BrokerService.start(BrokerService.java:594)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)[activemq-spring-5.10.0.jar:5.10.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.6.0_31]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.6.0_31]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.6.0_31]
        at java.lang.reflect.Method.invoke(Method.java:622)[:1.6.0_31]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1638)[spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1579)[spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509)[spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)[spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)[spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296)[spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)[spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)[spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)[spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628)[spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)[spring-context-3.2.8.RELEASE.jar:3.2.8.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)[spring-context-3.2.8.RELEASE.jar:3.2.8.RELEASE]
        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-3.16.jar:3.16]
        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-3.16.jar:3.16]
        at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.10.0.jar:5.10.0]
        at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.10.0.jar:5.10.0]
        at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.10.0.jar:5.10.0]
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)[activemq-console-5.10.0.jar:5.10.0]
        at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150)[activemq-console-5.10.0.jar:5.10.0]
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)[activemq-console-5.10.0.jar:5.10.0]
        at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.10.0.jar:5.10.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.6.0_31]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.6.0_31]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.6.0_31]
        at java.lang.reflect.Method.invoke(Method.java:622)[:1.6.0_31]
        at org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.10.0]
        at org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.10.0]
        2014-06-13 11:11:11,090 | INFO | Stopped LevelDB[/home/activemq/apache-activemq/data/leveldb] | org.apache.activemq.leveldb.LevelDBStore | LevelDB IOException handler.

        Show
        johnleach John Leach added a comment - I'm getting the same with just a straight single broker. The config has worked for many months under 5.9.0, but upgrading to 5.10.0 gives this on startup and never finishes starting up: 2014-06-13 11:11:10,931 | INFO | Using Persistence Adapter: LevelDB [/home/activemq/apache-activemq/data/leveldb] | org.apache.activemq.broker.BrokerService | main 2014-06-13 11:11:10,999 | INFO | Using the pure java LevelDB implementation. | org.apache.activemq.leveldb.LevelDBClient | main 2014-06-13 11:11:11,074 | INFO | No IOExceptionHandler registered, ignoring IO exception | org.apache.activemq.broker.BrokerService | LevelDB IOException handler. java.io.IOException: com.google.common.base.Objects.firstNonNull(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39) [activemq-client-5.10.0.jar:5.10.0] at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552) [activemq-leveldb-store-5.10.0.jar:5.10.0] at org.apache.activemq.leveldb.LevelDBClient.replay_init(LevelDBClient.scala:657) [activemq-leveldb-store-5.10.0.jar:5.10.0] at org.apache.activemq.leveldb.LevelDBClient.start(LevelDBClient.scala:558) [activemq-leveldb-store-5.10.0.jar:5.10.0] at org.apache.activemq.leveldb.DBManager.start(DBManager.scala:648) [activemq-leveldb-store-5.10.0.jar:5.10.0] at org.apache.activemq.leveldb.LevelDBStore.doStart(LevelDBStore.scala:235) [activemq-leveldb-store-5.10.0.jar:5.10.0] at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55) [activemq-client-5.10.0.jar:5.10.0] at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:640) [activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:629) [activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.BrokerService.start(BrokerService.java:594) [activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73) [activemq-spring-5.10.0.jar:5.10.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_31] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [:1.6.0_31] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [:1.6.0_31] at java.lang.reflect.Method.invoke(Method.java:622) [:1.6.0_31] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1638) [spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1579) [spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509) [spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521) [spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458) [spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296) [spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223) [spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293) [spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) [spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628) [spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE] at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932) [spring-context-3.2.8.RELEASE.jar:3.2.8.RELEASE] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479) [spring-context-3.2.8.RELEASE.jar:3.2.8.RELEASE] at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64) [xbean-spring-3.16.jar:3.16] at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52) [xbean-spring-3.16.jar:3.16] at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104) [activemq-spring-5.10.0.jar:5.10.0] at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104) [activemq-spring-5.10.0.jar:5.10.0] at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67) [activemq-spring-5.10.0.jar:5.10.0] at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71) [activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54) [activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87) [activemq-console-5.10.0.jar:5.10.0] at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57) [activemq-console-5.10.0.jar:5.10.0] at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150) [activemq-console-5.10.0.jar:5.10.0] at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57) [activemq-console-5.10.0.jar:5.10.0] at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104) [activemq-console-5.10.0.jar:5.10.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_31] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [:1.6.0_31] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [:1.6.0_31] at java.lang.reflect.Method.invoke(Method.java:622) [:1.6.0_31] at org.apache.activemq.console.Main.runTaskClass(Main.java:262) [activemq.jar:5.10.0] at org.apache.activemq.console.Main.main(Main.java:115) [activemq.jar:5.10.0] 2014-06-13 11:11:11,090 | INFO | Stopped LevelDB [/home/activemq/apache-activemq/data/leveldb] | org.apache.activemq.leveldb.LevelDBStore | LevelDB IOException handler.
        Hide
        johnleach John Leach added a comment -

        I've tried wiping out the data dir too - it's not a corrupted db problem.

        Show
        johnleach John Leach added a comment - I've tried wiping out the data dir too - it's not a corrupted db problem.
        Hide
        netlancermobi Netlancer added a comment - - edited

        Hi,

        the problem is because two different versions of guava cache related classes present in classpath. AMQ-5105 is for this issue.

        following workaround allows activemq to be run with leveldb configured

        1. remove pax-url-aether-1.5.2.jar from lib directory
        2. comment out the log query section

        <bean id="logQuery" class="org.fusesource.insight.log.log4j.Log4jLogQuery"
        lazy-init="false" scope="singleton"
        init-method="start" destroy-method="stop">
        </bean>

        Thanks,

        Show
        netlancermobi Netlancer added a comment - - edited Hi, the problem is because two different versions of guava cache related classes present in classpath. AMQ-5105 is for this issue. following workaround allows activemq to be run with leveldb configured 1. remove pax-url-aether-1.5.2.jar from lib directory 2. comment out the log query section <bean id="logQuery" class="org.fusesource.insight.log.log4j.Log4jLogQuery" lazy-init="false" scope="singleton" init-method="start" destroy-method="stop"> </bean> Thanks,
        Hide
        davicdsalves Davi Alves added a comment -

        Removing the com folder from the pax-url-aether-1.5.2.jar solved it too.

        Show
        davicdsalves Davi Alves added a comment - Removing the com folder from the pax-url-aether-1.5.2.jar solved it too.
        Hide
        enietzel Earle Nietzel added a comment -

        FYI Also had this same issue but it wasn't because pax-url-aether was on the classpath but rather an old version of google-collections-1.0...

        Just in case if someone else runs into something similar this option " -verbose:class" will help detect which class is being loaded.

        Show
        enietzel Earle Nietzel added a comment - FYI Also had this same issue but it wasn't because pax-url-aether was on the classpath but rather an old version of google-collections-1.0... Just in case if someone else runs into something similar this option " -verbose:class" will help detect which class is being loaded.
        Hide
        scoopex Marc Schöchlin added a comment -

        This should be fixed

        I do:

        zip --delete /opt/activemq/lib/pax-url-aether-1.5.2.jar "com/*"
        
        Show
        scoopex Marc Schöchlin added a comment - This should be fixed I do: zip --delete /opt/activemq/lib/pax-url-aether-1.5.2.jar "com/*"
        Hide
        johnleach John Leach added a comment -

        This is fixed in 5.14.0.

        Show
        johnleach John Leach added a comment - This is fixed in 5.14.0.

          People

          • Assignee:
            Unassigned
            Reporter:
            jrushford John Rushford
          • Votes:
            6 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development