ActiveMQ
  1. ActiveMQ
  2. AMQ-3051

Failed to read journal - Could not locate data file

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Won't Fix
    • Affects Version/s: 5.4.1
    • Fix Version/s: None
    • Component/s: Broker
    • Labels:
      None
    • Environment:

      Solaris 10 running on Sparc

      Description

      We are consistently seeing the following exception in our Production environment. At the current time we are seeing over 200K messages on the queue and no consumers can consume any messages. Due to issues with the kahadb not deleting files, we are using the AMQStore for persistence. We are using a NFS drive for the output of our persistent store.

      The process cannot then be restarted as it will fail trying to load back in the file store. The messages stored in the queue are critical, a way to recover these messages is important also.

      2010-11-24 08:20:44,015 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:Consumer.CaseManagement.VirtualTopic.intralinks.eventService
      java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Failed to read to journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could not locate data file data-17376
      at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
      at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
      at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1681)
      at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1845)
      at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1432)
      at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
      at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
      Caused by: java.lang.RuntimeException: java.io.IOException: Failed to read to journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could not locate data file data-17376
      at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:261)
      at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
      ... 6 more
      Caused by: java.io.IOException: Failed to read to journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could not locate data file data-17376
      at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
      at org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:680)
      at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:564)
      at org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:444)
      at org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:59)
      at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:89)
      at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:127)
      at org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:537)
      at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
      at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:258)
      ... 7 more
      Caused by: java.io.IOException: Could not locate data file data-17376
      at org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:315)
      at org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:637)
      at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:561)
      ... 14 more

        Activity

        Keith Brown created issue -
        Keith Brown made changes -
        Field Original Value New Value
        Attachment activemq-master-a.xml [ 19891 ]
        Keith Brown made changes -
        Description We are consistently seeing the following exception in our Production environment. At the current time we are seeing over 200K messages on the queue and no consumers can consume any messages. Due to issues with the kahadb not deleting files, we are using the AMQStore for persistence. We are using a NFS drive for the output of our persistent store.

        2010-11-24 08:20:44,015 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:Consumer.CaseManagement.VirtualTopic.intralinks.eventService
        java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Failed to read to journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could not locate data file data-17376
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
        at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1681)
        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1845)
        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1432)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
        Caused by: java.lang.RuntimeException: java.io.IOException: Failed to read to journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could not locate data file data-17376
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:261)
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
        ... 6 more
        Caused by: java.io.IOException: Failed to read to journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could not locate data file data-17376
        at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
        at org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:680)
        at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:564)
        at org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:444)
        at org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:59)
        at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:89)
        at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:127)
        at org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:537)
        at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:258)
        ... 7 more
        Caused by: java.io.IOException: Could not locate data file data-17376
        at org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:315)
        at org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:637)
        at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:561)
        ... 14 more
        We are consistently seeing the following exception in our Production environment. At the current time we are seeing over 200K messages on the queue and no consumers can consume any messages. Due to issues with the kahadb not deleting files, we are using the AMQStore for persistence. We are using a NFS drive for the output of our persistent store.

        The process cannot then be restarted as it will fail trying to load back in the file store. The messages stored in the queue are critical, a way to recover these messages is important also.

        2010-11-24 08:20:44,015 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:Consumer.CaseManagement.VirtualTopic.intralinks.eventService
        java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Failed to read to journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could not locate data file data-17376
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
        at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1681)
        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1845)
        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1432)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
        Caused by: java.lang.RuntimeException: java.io.IOException: Failed to read to journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could not locate data file data-17376
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:261)
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
        ... 6 more
        Caused by: java.io.IOException: Failed to read to journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could not locate data file data-17376
        at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
        at org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:680)
        at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:564)
        at org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:444)
        at org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:59)
        at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:89)
        at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:127)
        at org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:537)
        at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:258)
        ... 7 more
        Caused by: java.io.IOException: Could not locate data file data-17376
        at org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:315)
        at org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:637)
        at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:561)
        ... 14 more
        Keith Brown made changes -
        Attachment activemq-master-a.xml [ 19891 ]
        Jeff Turner made changes -
        Project Import Fri Nov 26 22:32:02 EST 2010 [ 1290828722158 ]
        Rob Davies made changes -
        Priority Blocker [ 1 ] Minor [ 4 ]
        Timothy Bish made changes -
        Status Open [ 1 ] Closed [ 6 ]
        Resolution Won't Fix [ 2 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Keith Brown
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development