Uploaded image for project: 'ActiveMQ Classic'
  1. ActiveMQ Classic
  2. AMQ-1786 Journal files don't get cleaned up
  3. AMQ-1795

in use data files removed from data store under load

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 5.1.0
    • 5.2.0
    • None
    • None

    • all

    Description

      Under load, I see the error below.

      the problem is that org.apache.activemq.store.amq.AMQPersistenceAdapter.cleanup() does not capture all of the in use files.
      I have made some changes that improves the situation but there is still a window, that seems to be related to spooling messages when a memory limit is reached.

      2008-06-12 14:55:16,689 [main ] INFO BrokerService - Using Persistence Adapter: AMQPersistenceAdapter(activemq-data\localhost)
      2008-06-12 14:55:16,704 [main ] INFO AMQPersistenceAdapter - AMQStore starting using directory: activemq-data\localhost
      2008-06-12 14:55:17,361 [main ] INFO AMQPersistenceAdapter - Journal deleted:
      2008-06-12 14:55:17,361 [main ] INFO KahaStore - Kaha Store successfully deleted data directory activemq-data\localhost\kr-store\data
      2008-06-12 14:55:17,392 [main ] INFO KahaStore - Kaha Store successfully deleted data directory activemq-data\localhost\kr-store\state
      2008-06-12 14:55:17,392 [main ] INFO KahaStore - Kaha Store using data directory activemq-data\localhost\kr-store\state
      2008-06-12 14:55:17,470 [main ] INFO AMQPersistenceAdapter - Active data files: []
      2008-06-12 14:55:17,705 [main ] INFO BrokerService - ActiveMQ null JMS Message Broker (localhost) is starting
      2008-06-12 14:55:17,705 [main ] INFO BrokerService - For help or more information please see: http://activemq.apache.org/
      2008-06-12 14:55:17,814 [JMX connector ] INFO ManagementContext - JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
      2008-06-12 14:55:17,924 [main ] INFO KahaStore - Kaha Store using data directory activemq-data\localhost\kr-store\data
      2008-06-12 14:55:17,939 [main ] INFO TransportServerThreadSupport - Listening for connections at: tcp://salthill:61616
      2008-06-12 14:55:17,939 [main ] INFO TransportConnector - Connector Default Started
      2008-06-12 14:55:17,939 [main ] INFO BrokerService - ActiveMQ JMS Message Broker (localhost, ID:salthill-2790-1213278917752-0:0) started
      2008-06-12 14:55:17,939 [main ] INFO MissingDataFileTest - Starting broker..
      2008-06-12 14:55:58,516 [age Thread Pool] ERROR AsyncDataManager - Looking for key 551 but not found in fileMap:

      {1014=data-1014 number = 1014 , length = 1960 refCount = 6, 607=data-607 number = 607 , length = 1266 refCount = 1, 928=data-928 number = 928 , .. .. 622 , length = 1352 refCount = 2, 328=data-328 number = 328 , length = 1266 refCount = 1, 256=data-256 number = 256 , length = 1264 refCount = 1, 92=data-92 number = 92 , length = 1264 refCount = 1, 404=data-404 number = 404 , length = 1266 refCount = 1}

      2008-06-12 14:55:58,563 [age Thread Pool] ERROR AbstractStoreCursor - Failed to fill batch
      java.io.IOException: Failed to read to journal for: offset = 0, file = 551, size = 1, type = 0. Reason: java.io.IOException: Could not locate data file data-551
      at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
      at org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:635)
      at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:521)
      at org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:432)
      at org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:54)
      at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:82)
      at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:120)
      at org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:530)
      at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:75)
      at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:188)
      at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.onUsageChanged(AbstractStoreCursor.java:157)
      at org.apache.activemq.usage.Usage$1.run(Usage.java:266)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: java.io.IOException: Could not locate data file data--551
      at org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:303)
      at org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:613)
      at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:518)
      ... 12 more
      2008-06-12 14:55:58,563 [age Thread Pool] ERROR AbstractStoreCursor - Failed to fill batch
      java.lang.RuntimeException: java.io.IOException: Failed to read to journal for: offset = 0, file = 551, size = 1, type = 0. Reason: java.io.IOException: Could not locate data file data-551
      at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:191)
      at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.onUsageChanged(AbstractStoreCursor.java:157)
      at org.apache.activemq.usage.Usage$1.run(Usage.java:266)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: java.io.IOException: Failed to read to journal for: offset = 0, file = 551, size = 1, type = 0. Reason: java.io.IOException: Could not locate data file data-551
      at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
      at org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:635)
      at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:521)
      at org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:432)
      at org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:54)
      at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:82)
      at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:120)
      at org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:530)
      at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:75)
      at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:188)
      ... 5 more
      Caused by: java.io.IOException: Could not locate data file data--551
      at org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:303)
      at org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:613)
      at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:518)
      ... 12 more
      2008-06-12 15:00:18,531 [main ] INFO BrokerService - ActiveMQ Message Broker (localhost, ID:salthill-2790-1213278917752-0:0) is shutting down
      2008-06-12 15:00:20,438 [main ] INFO TransportConnector - Connector Default Stopped
      2008-06-12 15:00:20,641 [main ] INFO BrokerService - ActiveMQ JMS Message Broker (localhost, ID:salthill-2790-1213278917752-0:0) stopped

      Attachments

        1. AMQ-1795.test_and_partial_fix
          24 kB
          Gary Tully
        2. AMQ-1795.further_improvement
          2 kB
          Gary Tully

        Activity

          People

            rajdavies Robert Davies
            gtully Gary Tully
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: