[AMQ-9186] How to recover mkahadb

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 5.17.0
• Component/s: Broker, mKahaDB, Performance Test
• Assignee: Unassigned
• Reporter: Ashish
• Created: 19/Dec/22 09:57
• Updated: 19/Dec/22 10:02

Description

Hello,

I have migrated my active mq setup to cloud(eks). I am using efs as my persistence layer. During testing, I have been using ignoreMissingJournalfiles="false" and from the past few days i have been seeing the below error. (restart policy is always)

Caused by: java.io.IOException: Detected corrupt journal files. [60:29718101 >= key < 60:33554433]

In order to fix this I first tried to delete the db.data file and restarted the broker. That ended up in the below error and also ended up in a for loop restart (restart policy is always)

Caused by: java.lang.IllegalStateException: PageFile is not loaded 

Later i set ignoreMissingJournalfiles="true", which allowed the broker to ignore the corrupt journal file and start the broker.

What would be your recommendation/steps to address this problem. Are their already any documentation for it?

Note : The broker ends up in this state due to OOM, I have already increased the brokers memory substantially (4gb), I just dont want to increase the memory further without understanding if there is a way to recover the broker 1st. I am also using EFS with Performance mode : Max I/O and Throughput mode : Bursting

Regards
Ashish