Uploaded image for project: 'ActiveMQ Classic'
  1. ActiveMQ Classic
  2. AMQ-6282

LevelDB store can't recover from no disk space

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 5.12.2
    • None
    • LevelDB
    • None
    • CentOS 6.5

    Description

      If there is no space left on the drive that leveldb is using for storage, it will wind up truncating the store-version.txt file, which will prevent activemq from starting up once the drive space issue is resolved.

      To reproduce:
      1. Start activemq, configured to use leveldb as a persistence store.
      2. use fallocate to create a file(s) that use up all available disk space on the drive that leveldb is using to persist to.
      3. restart activemq. It will fail to start with errors in the log about no available disk space (expected).
      4. delete the files created in step 2, so that the drive is not full
      5. start activemq again. This time it will fail with the below exception:

      2016-05-05 13:34:39,986 [erSimpleAppMain] ERROR XBeanBrokerFactory - Failed to load: class path resource [activemq.xml], reason: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.lang.Exception: Cannot open the store. It's schema version is not supported.
      org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.lang.Exception: Cannot open the store. It's schema version is not supported.
      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1514)
      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
      at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
      at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
      at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
      at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
      at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:636)
      at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:934)
      at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
      at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
      at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
      at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)
      at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)
      at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)
      at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
      at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
      at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)
      at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:62)
      at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)
      at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:62)
      at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
      at org.apache.activemq.console.Main.main(Main.java:115)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.Exception: Cannot open the store. It's schema version is not supported.
      at org.apache.activemq.leveldb.LevelDBClient.init(LevelDBClient.scala:596)
      at org.apache.activemq.leveldb.LevelDBClient.start(LevelDBClient.scala:557)
      at org.apache.activemq.leveldb.DBManager.start(DBManager.scala:648)
      at org.apache.activemq.leveldb.LevelDBStore.doStart(LevelDBStore.scala:312)
      at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
      at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:651)
      at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:640)
      at org.apache.activemq.broker.BrokerService.start(BrokerService.java:604)
      at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1640)
      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1581)
      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1511)
      ... 33 more

      6. If you look at store-version.txt file (ie. /usr/share/activemq/data/leveldb/store-version.txt), the file is empty.
      7. If you edit the file and add the text "activemq_leveldb_store:1" to it, activemq will start up successfully the next time you try to start it.

      Attachments

        Activity

          People

            Unassigned Unassigned
            ryan.stedman@gmail.com Ryan Stedman
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: