Uploaded image for project: 'ActiveMQ Classic'
  1. ActiveMQ Classic
  2. AMQ-5026

Replicated LevelDB Store getting EOF exception

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Won't Fix
    • 5.10.0
    • None
    • LevelDB
    • None

    Description

      3 brokers setup for ReplicatedLevelDB master/slave conf.
      running producer and slow consumer and doing failover by stopping master broker.

      After few failovers, the leveldb throws EOF exception while reading record from the store.
      using the latest source code from activemq 5.10 branch.

      The file that has the issue:
      [root@vmx0568 data]# ls -l /solidstate/ldbkp/000000001f4022e4.log
      rw-rr-. 1 root root 91500224 Jan 17 14:44 /solidstate/ldbkp/000000001f4022e4.log
      [root@vmx0568 data]#

      Client | main
      2014-01-17 14:44:19,118 | INFO | locator read: DataLocator(24b41aac, 2283) | org.apache.activemq.leveldb.LevelDBClient | main
      2014-01-17 14:44:19,118 | INFO | Seq read: 260217 | org.apache.activemq.leveldb.LevelDBClient | main
      2014-01-17 14:44:19,118 | INFO | locator read: DataLocator(24b423ee, 2283) | org.apache.activemq.leveldb.LevelDBClient | main
      2014-01-17 14:44:19,118 | INFO | Seq read: 260218 | org.apache.activemq.leveldb.LevelDBClient | main
      2014-01-17 14:44:19,118 | INFO | locator read: DataLocator(24b42d02, 2283) | org.apache.activemq.leveldb.LevelDBClient | main
      2014-01-17 14:44:19,118 | INFO | Seq read: 260219 | org.apache.activemq.leveldb.LevelDBClient | main
      2014-01-17 14:44:19,118 | INFO | locator read: DataLocator(24b43628, 2283) | org.apache.activemq.leveldb.LevelDBClient | main
      2014-01-17 14:44:19,118 | INFO | Seq read: 260220 | org.apache.activemq.leveldb.LevelDBClient | main
      2014-01-17 14:44:19,118 | INFO | locator read: DataLocator(24b43f6a, 2283) | org.apache.activemq.leveldb.LevelDBClient | main
      2014-01-17 14:44:19,118 | INFO | Seq read: 260221 | org.apache.activemq.leveldb.LevelDBClient | main
      2014-01-17 14:44:19,118 | INFO | locator read: DataLocator(24b4487e, 2283) | org.apache.activemq.leveldb.LevelDBClient | main
      2014-01-17 14:44:19,118 | INFO | Seq read: 260222 | org.apache.activemq.leveldb.LevelDBClient | main
      2014-01-17 14:44:19,118 | INFO | locator read: DataLocator(24b451a4, 2283) | org.apache.activemq.leveldb.LevelDBClient | main
      2014-01-17 14:44:19,124 | INFO | No IOExceptionHandler registered, ignoring IO exception | org.apache.activemq.broker.BrokerService | LevelDB IOException handler.
      java.io.EOFException: File '/solidstate/ldbkp/000000001f4022e4.log' offset: 91500234
      at org.apache.activemq.leveldb.RecordLog$LogReader.read(RecordLog.scala:299)
      at org.apache.activemq.leveldb.RecordLog$$anonfun$read$2.apply(RecordLog.scala:564)
      at org.apache.activemq.leveldb.RecordLog$$anonfun$read$2.apply(RecordLog.scala:564)
      at org.apache.activemq.leveldb.RecordLog.get_reader(RecordLog.scala:554)
      at org.apache.activemq.leveldb.RecordLog.read(RecordLog.scala:564)
      at org.apache.activemq.leveldb.LevelDBClient.getMessage(LevelDBClient.scala:1318)
      at org.apache.activemq.leveldb.LevelDBClient$$anonfun$queueCursor$1.apply(LevelDBClient.scala:1257)
      at org.apache.activemq.leveldb.LevelDBClient$$anonfun$queueCursor$1.apply(LevelDBClient.scala:1252)
      at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1$$anonfun$apply$mcV$sp$12.apply(LevelDBClient.scala:1342)
      at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1$$anonfun$apply$mcV$sp$12.apply(LevelDBClient.scala:1341)
      at org.apache.activemq.leveldb.LevelDBClient$RichDB.check$4(LevelDBClient.scala:323)
      at org.apache.activemq.leveldb.LevelDBClient$RichDB.cursorRange(LevelDBClient.scala:325)
      at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1.apply$mcV$sp(LevelDBClient.scala:1341)
      at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1.apply(LevelDBClient.scala:1341)
      at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1.apply(LevelDBClient.scala:1341)
      at org.apache.activemq.leveldb.LevelDBClient.usingIndex(LevelDBClient.scala:1019)
      at org.apache.activemq.leveldb.LevelDBClient$$anonfun$might_fail_using_index$1.apply(LevelDBClient.scala:1025)
      at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:549)
      at org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:1025)
      at org.apache.activemq.leveldb.LevelDBClient.collectionCursor(LevelDBClient.scala:1340)
      at org.apache.activemq.leveldb.LevelDBClient.queueCursor(LevelDBClient.scala:1252)
      at org.apache.activemq.leveldb.DBManager.cursorMessages(DBManager.scala:713)
      at org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.recoverNextMessages(LevelDBStore.scala:758)
      at org.apache.activemq.broker.region.Queue.initialize(Queue.java:369)
      at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:87)
      at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:516)
      at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
      at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:134)
      at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:321)
      at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:167)
      at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:169)
      at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:167)
      at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:167)

      at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:172)
      at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:98)
      at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:190)
      at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:120)
      at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:182)
      at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:182)
      at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:120)
      at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2150)
      at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:648)
      at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:632)
      at org.apache.activemq.broker.BrokerService.start(BrokerService.java:568)
      at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1608)
      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1549)
      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1479)
      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
      at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
      at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
      at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
      at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
      at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
      at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628)
      at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
      at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
      at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
      at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
      at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)
      at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)
      at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)
      at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
      at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
      at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)
      at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
      at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150)
      at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
      at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
      at org.apache.activemq.console.Main.main(Main.java:115)
      2014-01-17 14:44:19,156 | INFO | Stopped LevelDB[/solidstate/ldbkp] | org.apache.activemq.leveldb.LevelDBStore | LevelDB IOException handler.

      Attachments

        Activity

          People

            Unassigned Unassigned
            kal123 kal patel
            Votes:
            8 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: